Debugging techniques for Web development 2 – Configuring your environment

In the previous issue we discussed using the right tools to more easily debug your Web development scripts.

Now these tools can be made even more useful if they are configured in a way that makes the developer’s life easier.

2. Configuring your environment for better productivity

If you don’t make yourself at home in your own code base, who will?

Configuring your environment can mean many things, but the points I would like to cover in this post are:

  • Setting up your editor / subversion / server so that you can get through the day in the most painless way
  • Creating a development environment that works alongside your production
  • Writing your code so that if something goes wrong you can easily pinpoint the cause and quickly fix it

Setting up editor / server & co

When you start working on a project, you often want to start small and avoid wasting massive investments to organise your code. Now, once you’ve done it a few times, you’ll realize that settings up a clean and comfy environment doesn’t take that much effort, and really pays off in the long run.

By that I mean the following:

On the editor

  • Making sure your editor is the right one for the language you use, and is extendible. This means it allows you to define keyboard shortcuts for the kind of things you often do. I like creating shortcuts for the following stuff:
    • firing up a svn diff or svn commit command. Most of the time it’s just to figure out if I’ve changed the file at all, sometimes it’s to do an actual commit
    • opening the browser at a given location
    • bookmarks for often-used config files (e.g. the apache ini)

On the server

A common practice with beginners using a webserver is to put all their projects under a single root, so they can then get pages at

http://localhost/project1 (which will point to c:\bla\bla\root\project1)
http://localhost/project2 (which will point to c:\bla\bla\root\project2)

Now, by tweaking some config, you can achieve a much more symmetric setup, where your 2 projects will be hosted on the root of some (virtual) server. In order to do so, do the 2 following things:

  • Add as many entries as you like in your hosts files (see Wikipedia), of the following form
# all these names point to your localhost
# but Apache will see the difference
# due to the VirtualHosts directive

127.0.0.1    project1
127.0.0.1    project2
  • Add virtual hosts in your Apache config, like this

    DocumentRoot "c:\bla\bla\root\project1"
    ServerName project1

The result of that is the ability to get the website (from your computer that is!) at an address like

     http://project1/

I know – in theory it shouldn’t matter whether your pages are at the root of your server or not! However in practice it is a source of so much pain and wasted time when your local setup runs on a sub-folder of the root and your production setup runs on the root, and some things do not work as expected on your production server. Life’s too short to try and make it work (and some of the frameworks are just rubbish at working outside of the root of a server and you do not plan to fix them).

In the database

Not that I am doing it routinely, but some cool database features can be leveraged to make your life easier:

  • Views: these are queries that you can use as if they were a table. So for instance, if you’ve got a query like this
SELECT user_id, user_name FROM users WHERE user_type = 'admin'

Then you would create a view by doing this

CREATE VIEW admins AS
SELECT user_id, user_name FROM users WHERE user_type = 'admin'

Subsequently, you can then do

SELECT user_name FROM admins

This can be useful when you are debugging and don’t want to endlessly retype the same WHERE clauses.

  • Triggers: never used these, but they look pretty powerful. They allow you to have cascading effects in your tables, or prevent you from doing operations which would violate relationships (e.g. preventing you from deleting a user in the users table if the posts table still has entries linked to that user)

In your system

Create scripts to perform standard tasks.  I personally think that people who write batch / unix scripts are masochists, and so I do it in python much more easily. That includes

  • Issuing ssh commands to a remote server (e.g. svn update)
  • Tranfering files through SCP
  • Creating a dump of your mysql database using mysqldump
  • Going to some of your favorites directories

Creating a development environment

For some people, a development environment is something they are not even aware of and which consists of a series of commented print (PHP) or console.log (Javascript) statements scattered randomly in their code, and which will be uncommented when needed – and sometimes accidentally pushed to production with unpredictable consequences…

Any serious framework will consider the development environment an integral part of the project, and provide an API that will allow the developper to leave trails of what is happening. At the same time it is important that the user does not see any of this, be it for better customer experience or sometimes as a security precaution (i.e. avoiding to give the user insights in the server-side code). This can happen by having the notion of environment at the API level. This means that

  • At the root of your code (the HTML page for Javascript, your controller for PHP), there needs to be an assesment of what the current environment is, i.e. ‘development’, ‘testing’, or ‘production’. This can be done by some environment variable, some local file, can be based on the hostname (or the virtual host – even better!), or perhaps by some cookie on the user’s browser.
if ( /*... */ )
    $ENVIRONMENT = 'production'
if ( /* ... */ )
    $ENVIRONMENT = 'development'
  • Any setting that has reasons to vary depending on the environment should be predicated on the environment. So, for instance, the MySQL database could be a function of the environment, for instance like this
$db['production']['database'] = 'projectdb_prod';
$db['dev']['database']        = 'projectdb_dev';

And the subsequent calls will be like this

$server = $db[ $ENVIRONMENT ]['database']

A golden rule – A SINGLE CODEBASE

 The ‘single codebase’ rule means that you do not want to have different versions of the same file depending on the server. This would be difficult to maintain / deploy (not least because it would have to be outside of your control version system), and soon or later you are going to accidentally overwrite the config of one server onto another and scratch your head to understand the source of a bug. Instead, you want all the settings to be in the common codebase, and ensure that the choices are done on-the-fly depending on the environment / host / OS.

Writing your code so it is debuggable

Separation of concerns

Modularity, for example as in MVC frameworks can dramatically improve the debuggability of your code. This is because the business logic will be testable independently of the GUI (which tends to be tough to test).

Hard to give precise guidelines, but let’s say that the rule of thumb is to make it so that you can test key features of your application with one-liners and without having to use the user interface.

Using configuration settings or even bits of code that can be changed at runtime

A technique I have found useful, which is more related to fine-tuning the Javascript part of your application than debugging it, is that of using configuration settings.

For instance, in an app I am developping, I need to parametrize many settings so that some graphics looks as good as possible.

Previously my code was looking a bit like this

some_setting       = 0.5
some_other_setting = 25
timestep           = 0.25
some_curve         = timestep * some_setting + x * some_other_setting + /* .. */

This meant that, in order to test changes to any of these settings (say, move some_setting to 0.6), I needed to reload my page. Given that I am now convinced that one-page (javascript) apps are the future of the web, I end up with a page whose initialization can take up to 10 seconds. Given the features, this is acceptable for the end-user, but for debugging purposes, it is a bit of a pain.

The idea I came up with (well this must be a design pattern – I just don’t know how it’s called) was to use a global function to get and set settings, and attach a callback that gets triggered every time the settings are changed.

This broadly worked as follows.

In a library, I would have

// ideally this first line should be in a project specific library whilst
// the rest is totally generic
function DefaultSettings() { return { 'SOME_SETTING' : 0.5 /* ... */ } }
current  = DefaultSettings()
onchange = function() {}
function Settings(key) { return current[key] }
function SettingsSet(key,value)
{
     current[key] = value;
     onchange(); // trigger callback
}
function SettingsReset() { current = DefaultSettings(); onchange(); }
function SettingsAttachCB( callback ) { onchange = callback }

Then my code above would become

some_setting       = Settings('SOME_SETTING');
some_other_setting = Settings('SOME_OTHER_SETTING');
timestep           = Settings('TIMESTEP');
some_curve         = timestep * some_setting + x * some_other_setting + /* .. */

And in my initialization function, I would have a line like this

SettingsAttachCB( SomeRefreshFunction );

This would mean that, once the page has loaded, I can go to the Chrome debugger console, type SettingsSet('SOME_SETTING', 0.6), and that will automatically trigger SomeRefreshFunction.

Settings

As a matter of fact, I even wrote a little HTML widget that turns current in JSON into a popup window, and allows some users to modify it.

Continue reading

Debugging techniques for Web development 1 – Using the right tools

Over the years I’ve become a bit of a debugging freak.

I wish I was able to write code that works straightaway, but for some reason, I just love programming by trial and error. So I go through many cycles before it’s just right, and without some ways to shorten these cycles, programming would be a nightmare for me.

So I would like to share a few techniques gathered over time, which hopefully will help you as is, or will inspire you to develop a debugger’s mindset. Be it in Web programming or any scripting environment.

The techniques will be explained in several blog posts, which will be about

  1. Using the right tools – to do complex things in a simple way rather than the other way round…
  2. Configuring your environment for better productivity – if you don’t make yourself at home in your own codebase, who will?
  3. On-the-fly coding instead of edit/reload/edit/reload – coz’ life is to short to spend it waiting for your script to reinitialize.

Let’s then start with the first piece of advice, which I would assume is the easiest to apply

1. Using the right tools

To do complex things in a simple way rather than the other way round…

In the past, I’ve sometimes been doing stuff in a convoluted fashion because I was not aware that certain tools existed, which could have made things so much easier. This is a general principle, and so it’s mostly a question of discussion with the community to figure out which tools are going to help you in your particular project.

Let me however give a few tips specific for Web development.

Installing a [L/W]AMP stack

Unless you’re just writing 2-3 web pages, please do not do Web development by loading the files as is. Put yourself in an environment similar to the final deployment stage, by running all the pages through a proper webserver.

One reason for that is that Ajax call will likely not work from local pages due to the X-domain policy. You may also face issues of links working in your development environment but not on the server, or vice versa. And of course, PHP scripts and .htaccess stuff (i.e. Apache configs per directory) will definitely not run if you call them directly rather than through a webserver…

So – on Linux systems, the webserver  is usually provided out of the box by LAMP (Linux/Apache/MySQL/PHP), and on windows, you can install EasyPHP which provides these with minimal hassle.

Javascript

I really fell in love with JS the day I discovered the Chrome debugger. Obviously it requires using Chrome, which I fail to see a reason to do without – see table on the side (source: Wikipedia).

ScreenClip3

The debugger is invoked by pressing CTRL-SH-J, and will appear at the bottom of the page. It can be detached as a separate window, which can be desirable if your HTML layout is involved.

The debugger features a set of terrific features – see the Chrome Developer Tools page for more details.

One of them is the console (press ESC to open/close it, whilst the debugger is on). The console allows you to evaluate expressions on the fly after the page has loaded. This is extremely useful if you want to invoke functions defined in your script, or modify the state of global variables . It is also very cool for CSS experiments (see CSS debugging paragraph below).

The console is also terrific for leaving trails of your code’s execution using console.log, without impacting the HTML layout, like this

ScreenClip

As for modifying variables, this goes beyond just  numbers / strings. In a future issue I intend to discuss that more in detail, especially the use of the console to modify and test callbacks / event functions without having to reload the page. As a preview, let me mention the use of the console to modify DOM objects.

Say, for instance that you are trying to make your website look better by changing the css. Reloading your page might be fine if it’s a lightweight one, but as soon as your project starts growing, your main page is bound to load tons of stuff at startup. This might make it a pain to fine tune styles / colors / dimensions.

Assuming you are using JQuery, you can easily change attributes using the Chrome debugger. There are 2 methods

  • go on the part of the page you want to modify, right-click and choose ‘Inspect element’. This will open up the debugger, and will open a window on the right with all css attributes. You can modify any of these manually.
  • my preferred method is to go to the console, and do something like this
$("#someid").css( { width : '10px', color : 'white' } )
$("#somepopup").show().css( /* ... */ )

This will modify the object on-the-fly. Once you’ve sorted out your settings, you can integrate it in your script for the next time you reload your page.

PHP

The more advanced you are in PHP, the more likely you are to use frameworks which tend to provide extensive debugging information. One of these is Symfony, which I find very impressive, but personally way over-the-top for my usage. I therefore limit myself to CodeIgniter + Datamapper, which are fine, but do not have embedded debugging features.

However – debugging is made extremely simple using Xdebug. It is an apache extension that is part of the Zend framework, but can be installed on its own. In my case, it came bundled with EasyPHP, and it turned out that my editor Notepad++ has a simple plugin DBGp which communicates with Xdebug and allows debugging PHP scripts as easily as you would do it in Javascript in the Chrome debugger.

The setup consists of downloading Notepad++ and the DBGp plugin, and moving the plugin to the right Notepad++ folder. The Notepad++ settings require you to specify the server (typically this would be localhost), and turn the debugger on by going to the ‘DBGp / debugger’ menu, which will make the debugger appear in the bottom part of Notepad++.

On the server side you will need to enable Xdebug, which is done in the php.ini with something that in my case was this

zend_extension = "${path}\php\php546x130302144954\php_xdebug-2.2.1-5.4-vc9.dll"
xdebug.default_enable=0
xdebug.remote_enable=1
xdebug.remote_host=127.0.0.1
xdebug.remote_port=9000
xdebug.remote_handler=dbgp
xdebug.remote_mode=req
xdebug.remote_autostart = false

What then happens is that every time you load your page (i.e. by going to http://localhost/yourpage), it will invoke the debugger on Notepad++, which will start blinking. You will need to put breakpoints by pressing CTRL-F9 on a line of your scripts where you want to start stepping, and you can step the usual way (using the little buttons at the bottom left of the Notepad++ screen). It should look like this

ScreenClip2

MySQL

Working with databases can be a total nightmare without the proper tools. You need to be able to test your queries independently of your code, and I would recommend installing PHPMyAdmin, which comes bundled with EasyPHP or is easily installed on a LAMP stack (it’s all PHP based so requires no shell access to the server). Given that it is accessible through a web browser, this allows you to do basic tests or SQL operations from anywhere. It has a visual interface for many operations, including data modification, table creation, and even db synchronization.

A more powerful tool is MySQL workbench. The drawback is that it requires local installation on Windows, but the interface is less sluggish than what you would except from a web-based interface such as PHPMyAdmin.

Control versioning

Control versioning software allows you to track changes in software you are writing, by having the developer ‘committing’ changes when he deems they are worthy of being used in production. This then allows him to see differences in different versions of his scripts, which often helps to pinpoint the location of freshly introduced bugs.

Working on even small size projects without control versioning is a bit like driving with your eyes closed. If everything runs fine and you’re the only one on the road, it’s not an issue, but if there is any unexpected event, you’re heading for a crash.

Anyhow – installing control versioning is so simple that there is no excuse to do without.

I know git’s the most fashionable tool nowadays, but I happen to be using subversion which I find just fine.

The setup goes as follows

Server side – 2 possibilities (ok this might be confusing if the server is on your own machine)

  • Creating a local repository on your computer or any computer on your network. This is as simple as typing svnadmin create 'your_repository', then launching the server by doing svnserve -d -r 'c:\path\to\your_repository'.
  • Creating an account at CloudForge, for instance, which will host your code for free (but it is limited to a single user).

Client side

Once the server is running, you will need a client to access it, and modify the files. I like TortoiseSVN, which allows you to perform all kinds of operations from the Windows explorer.

The benefit of using a CVS system in a live environment such as a website is that you can do without FTP or SCP, and apply/rollback changes at a whim. In my case, I setup a batch script to ‘svn update’ my production server (through ssh). This means I can upgrade my production server without having to figure out what needs update or what not.

This does not prevent me from keeping external files (images / libraries) in separate directories which are ftp’ed once and for all, and can even be shared across different versions of my website by using (Linux filesystem) links.

Get a proper server

OK so unlike the other points, this one is not that easy to apply. People may not have the budget for an expensive setup, but I highly recommend getting server space with a shell access. I discovered Rackspace, which costs me 14£/month, for which I have a 512Mb ubuntu server with 20Gb RAM and root access. It took me less than an hour to install AMP / subversion, check out my repository, sync my db and have my website online as on my laptop. Just installing Ubuntu on a computer can sometimes take 45 minutes, whilst Rackspace provide preconfigured images which get installed in a minute.

Of course, you can do without, but having no shell access may mean that you can not use subversion to update your website, and you are at the mercy of any discrepancy between your home setup and the server (which at a typical provider will have an older version of PHP rather than a newer, resulting in some language idoms working at home and not on the server).

Given that you have root access, nothing refrains you from setting up several accounts and sharing the server with friends. Because of the root access, you can do the following

  • setup a subversion server for free, which would probably cost you 5-10£/month if you went  for the premium solutions at Cloudforge, for instance.
  • setup an unlimited number of domains hosted on your server (Rackspace provide you with a control panel to edit DNS entries)
  • install a nodejs server, or Django, or upgrade to whichever version you want / need

Conclusion

Depending on your experience and your luck googling for some of the tools above, you may have things working out-of-the-box, or you may struggle and give up and miss out on some terrific productivity boosters.

Don’t hesitate to ask for help if you are facing issues!

Getting to NodeJS

So recently I thought I should really give a go at Nodejs. Like most web developpers, I am tired of having to develop two parallel codebases on client and server side. I recently found out that Javascript is actually quite cool and powerful, so I started experimenting.

If you want to go straight to the solution, see below at the ‘Quick workaround’ paragraph.

Now, with proper use of Ajax / JSON and JQuery, you can get to a stage where 90% of the work happens on the client, and the server is just there to serve db requests (hardly producing html). That’s cool, but there are a few catches:

    • A lot of what I do is geared towards Chrome. I don’t even dare trying my website on Firefox, let alone on IE. On my smartphone the page looks sort of okayish, but the HTML5 canvas which I am heavily relying on looks like it was drawn by a 3y old

. You have little control on the kind of software your user is using, whilst if you pick a nodejs server your code will run as you expect it.

  • The more happens on the client, the more you leave the door open to abuses (i.e. users firing up the Chrome Javascript console and running instructions to the server). Obviously if your server filters the requests that shouldn’t be too much of an issue but still, it is one thing more to worry about
  • Most importantly, leaving all your code freely accessible to download by every user might not be great if you have some code you do not want to share. Minify partially solves that issue, but still, if it’s on your servers only there’s way less chance people steal your code

Anyhow, I started playing with node, it all looks cool until I thought, ‘ok, how do I integrate my existing code into this?’, which is the absolute requirement and potential show-stopper. After lots of reading / Googling I was going to give up. I tried looking at requireJS which looked like some sort of quiz for people with a 170 IQ. I then came across a post which change my life. Or my day shall we say…

In Sharing a javascript base class between node… the author gives a very simple trick to have scripts work both on client and server. Interestingly enough it is two years old…

To make it complete, let me run you through the complete solution I got out of it. It’s super down-to-earth and perhaps Javascript experts will think it’s retarded, but this all works. I’ve got around 10 JS files in my projects which are interrelated, and this solved my issue. It took me 15 minutes to modify my files and I can now run command-line scripts with the same code, and I am confident I will be able to make server-side scripts out of them as well.

Quick workaround

I will assume you’ve got nodejs installed. It’s a quite straightforward download anyway.

Step 1

Create a ‘node_utils.js’ with the following code

exports.import_global = function(module)
{
    console.log('Importing ', module, '...');
    var mod = require(module)
    for ( key in mod )
        global[key] = mod[key] // this copies the function to the global scope
}

The benefit of import_global vs require is that this will put all the imported functions in the global scope. This might be considered super bad practice, but in the meantime this is exactly how things happen on client-side Javascript

Step 2

Add the proper imports in each of your legacy scripts. This may be a bit trial and error, and pray for not having circular references (not sure how node deals with it…)

// NODE.JS Compatibility
NODEJS_MODE = typeof window == 'undefined'
if ( NODEJS_MODE )   // this will be false on client-side JS
{
	var import_global = require("./node_utils.js").import_global;  // boilerplate code

        // now do your imports
	import_global('../lib/js/complex.js')
	import_global('./step.js')
	import_global('./utils.js')
}

/*
**
** this is your legacy client-side code
**
** e.g.
** function somefunction()
** {
**    ...
** }
**
*/

// NODE.JS Compatibility
if ( NODEJS_MODE )
{
	exports.somefunction = somefunction;
}

Step 3

A cool thing to know is that node-inspector allows you to debug node pretty much the same way as client-side Javascript. That is if you’re using Chrome or a webkit-compatible browser.

Well I hope this will be helpful for you as much as it was for me. Could be that in one weeks time I will think this was stupid, but I couldn’t help sharing this.

Remarks

  • If you have JQuery code, you may have trouble integrating it as is. It looks like this is something requireJS is dealing with. In my case, my scripts were written in such a way that there is a fair amount of (rather tricky) code that is completely DOM-agnostic (it’s mostly graphical libraries that calculate coordinates of various shapes), so I am fine. I would think that DOM code should not be invoked in nodeJS scripts, but it might be easier said than done if your code is monolithic.
  • At some point I got shivers when I realized that I had stuff like this in my legacy code
    Array.prototype.Table = function() 
    {
         /* ... */
    }
    

    But turns out this is not an issue – I just wrote the usual

    exports.Table = Array.prototype.Table
    

    and my modified scripts work both on client and nodeJS, where I can write stuff like

    a = [1,2,3].Table();
    

Debugging PHP in a floating javascript window

Zvonko wrote an interesting article on codeforest: Debugging PHP in browser’s Javascript console.

Debugging tools are a natural feature of frameworks like Zend, Symfony and others, but there are cases where it is not practical to use such a framework. This is where simple debugging tools can be handy.

Zvonko’s idea is that the PHP developper can output debugging information without altering the layout of his page. This is achieved by printing to the javascript console, which is accessible in Chrome by doing CTR-Sh-J, then ESC. Not clear how that works in other browsers.

I offer an alternative: printing to a javascript dialog box. This means you will not need to go in the guts of your browser: the debugging info will appear on the side of your HTML page, and can be hidden if needed. I use jquery but it’s possible that writing it in javascript from scratch is not that hard.

My class is called JSDebugger. Usage is as follows:


$debug = new JSDebugger();
// here you want to display stuff to the debugging window:
$debug->debug( "Value of x is $x" );

The code below is not meant to work out of the box but let me know if I need to make it more generic. The purpose is just to show the idea.

<?php

require( "HTML.php");

// you may want to replace this with wherever you put your jquery libraries
function script($name)
{
	$Map = Array(
					'jquery' 	=> 'jquery.js',
					'jquery-ui' => 'jquery-ui-1.8.9.custom.js'
				);
	
	$script = $Map[$name];	
	?>
	
<script type="text/javascript" src="javascripts/<?php print $script ?>"></script>

<?php 
}

class JSDebugger {

	function __construct() 
	{
            // include jquery libraries
	    script('jquery');
	    script('jquery-ui');
	    ?>
	    
	<link rel="stylesheet" href="stylesheets/themes/start/jquery.ui.all.css"> 
	    
	<style>
	.ui-dialog 	{ background: #eeeeee; }
	#debuglog 	{ font-size: 0.4em; }
	.ui-dialog-title { font-size: 0.6em; 	}
	</style>

	    <script type="text/javascript">
	
		myvar = "";
		function init()
		{
		
		    $("#debuglog")
		    	.dialog( {
					position: [ "right", "top" ],
					title: "Debug console",
		    	})
		}
	
		$(document).ready(init);
		
	    </script>
	    <div id='debuglog'></div>
	    
	    <?php 
	}
	
	function debug($message, $type = LOG) 
	{
		$bt = debug_backtrace();
		$bt = $bt[0];
		$file = basename( $bt['file']);
		$log = sprintf( "<H3 class=debugger>%15s:%4s (%10s) </H3>\n", $file, $bt['line'], $bt['function'] );
		$log .= str_replace("'", '"', $message );
		
		$lines = split("\n", $log);
		?>
		
	    <script type="text/javascript">
	    <?php foreach( $lines as $line )
	    {?>
			myvar += '<?php print $line ?>\n';
		<?php 
		}?>
		$("#debuglog").text(myvar);
		</script>
	    <?php 
	    
	}
	
	function __destruct()
	{
	?>
		<script type="text/javascript">
	    $("#debuglog").html( myvar )
	    </script>
	<?php 	
	}

}
?>