Gareth Lennox

Thoughts from ScaleConf

Mon, 06 May 2013 00:00:00 +0000

Thoughts from ScaleConf

06 May 2013

I went to ScaleConf a couple of weeks ago, and have been meaning to write about some thoughts I had, so here it is!

The conference is, as per its name, primarily about scaling, and to be more specific, scaling software, and to be even more specific, a lot of the talks were about scaling software in your start-up.

Most of the speakers were from local companies, that are start-ups, or have recently grown past that stage. There were a couple of international speakers, notably from Facebook and AWS. They have completely different problems to the rest of us.

It was a fascinating look at building scaling systems and I found a couple of common themes kept appearing in most, if not all, of the talks. They seem to be good general practices, not just when you need your software to scale. Below are the seven common themes I identified.

1. Use the cloud

This might seem obvious today, but it’s important to not underestimate the ease and speed that you can scale up using the cloud, not to mention the cost.

One speaker said that any small company can compete with much larger companies in terms of capacity, via the cloud, something that was not possible 5/10 years ago.

2. Make sure you can scale horizontally

You want to be able to drop in another server, or ten and have it just work. You cannot keep buying bigger and bigger machines, as it starts costing large amounts of money, and there is a limit to the size you can buy. One huge server is also a single point of failure…

3. Don’t have a single point of failure

The cloud makes this easy to do by having multiple instances or everything, and it doesn’t cost huge amounts of money. A lot of the speakers mentioned that they still have single points of failure, but in all cases, they were actively working to get rid of them.

4. Use caching

Seems simple enough, but caching can make or break a system and will let you ride out waves of busyness.

5. Use queuing

Use queues to separate your backend processing from the front-end. This allows you to build more loosely coupled systems with the ability to take down parts of the system without affecting the rest. Pretty much everyone speaking mentioned that they were using some form of queuing mechanism. RabbitMQ seems to be the new hotness.

6. Push to production often

There is all sorts of talk about agile in software development, but I realised that being agile in releasing your software is important too. Some of the speakers showed us how they deploy new versions of their software upwards of 20 times a day. Most corporates I’ve worked with, you’re lucky to get a new version into production once every 3 months.

7. Every scenario is different

What works for one scenario doesn’t necessarily work for others, including the things I listed above. You can probably think of a case quite easily where the above do not apply.

Simple version numbers with git

Mon, 28 Jan 2013 00:00:00 +0000

Simple version numbers with git

28 January 2013

Each .net project has a version number (4 dotted numbers, e.g. 1.0.0.0). While you could leave it like that, it would ideally point back a unique revision in your source control system.

Unfortunately, they need to be a number, so you cannot dump the git hash in there and be done.

There are some projects that do this for you automatically (like MSBuild Versioning), but for small projects this is typically a bit overboard.

I usually just create a windows batch file, version.bat in the root of my projects, with the following content:

SET version=0
FOR /F "tokens=1" %%a IN ('git rev-list master') DO SET /A version+=1
FOR /F "tokens=*" %%a IN ('git rev-parse HEAD') DO SET commit=%%a
echo [assembly: System.Reflection.AssemblyVersion("1.0.%version%")] > src\AssemblyVersion.cs
echo [assembly: System.Reflection.AssemblyDescription("Compiled from %commit%")] >> src\AssemblyVersion.cs

This just gets a version number from git (basically the number of commits) and the current commit’s hash and pushes it into a AssemblyVersion.cs file that I include with my project.

Yes, using the number of commits as a version number has issues if you deploy code from different places (you will get different version numbers because of merges), but remember – this is for small projects that you’d deploy from one place and that you’d typically not have a build server.

I usually have a build.bat file that first calls the version.bat one, and then calls msbuild to build the solution.

If you have a mbuild file, you could just pass in the version number and commit hash as parameters (but then, rather use MSBuild Versioning linked above).

Delivering all mail to a local account

Fri, 26 Oct 2012 00:00:00 +0000

Delivering all mail to a local account

26 October 2012

I recently asked a question on Ask Ubuntu, regarding setting up my Ubuntu (12.04) virtual machine with a mail server, but to not actually send any mail.

What’s the point of this? Well, because I’m developing web applications that send mail, I want to test with all sorts of different e-mail addresses, other than my own, without actually sending these test messages. Ideally all the messages would get dumped into a single folder to me to review whenever I need to.

I didn’t get a response (yes, I’m impatient!), but managed to work it out myself, and posted the answer to the site. I’m putting it here for future reference. I used the postfix mail server, which is easy to install (sudo apt-get install postfix) and configure. To get it to deliver all mail locally, you need to do two things:

Install the postfix-pcre package (sudo apt-get install postfix-pcre)
Edit the main postfix configuration file (sudo nano /etc/postfix/main.cf file) and add the following lines, tweaking the MYLOCALUSER value to a user on the machine (using an answer on AskUbuntu):
```
luser_relay: MYLOCALUSER@localhost
local_recipient_maps =
```
Still editing the file, replace the mydestination configuration value using the value below (using an answer on serverfault) (note that you need the postfix-pcre package installed):
```
mydestination: pcre:/etc/postfix/mydestinations
```
Create and edit the mydestinations file referenced above (sudo nano /etc/postfix/mydestinations) and put the following line in it:
```
/.\*/ ACCEPT
```
Restart the postfix server:
```
sudo /etc/init.d/postfix restart
```

This creates a rule that basically tells postfix to treat all mail it processes as local domains. The luser_relay value and the local_recipient_maps value work together to define the default user to deliver mail to, your local user.

You can thereafter use whatever mail client you want, as all mail the server deals with is routed to your local user. Obviously you shouldn’t ever do this on a server that you want to, you know, actually send mail or anything, therefore it is primarily useful when developing software and you want to test you mail sending code.

Using Google Web Fonts

Tue, 07 Aug 2012 00:00:00 +0000

Using Google Web Fonts

07 August 2012

Google Web Fonts is a great resource for custom web fonts. It is free and has a huge library of fonts (508 different fonts at time of writing).

A lot of people are saying that while the library is large, the quality of most of the fonts is not that great. While this is true, you just have to find the high-quality fonts that do exist. There are a couple of useful sites on the web that help you choose fonts out of the huge selection. Beautiful Web Type is a great resource as is Google Webfonts that Don’t Suck. Alternatively, just play around with the various fonts (there is a handy chrome plugin).

It is really easy to include whatever fonts you want in you site, just beware about having too many, as it does increase your page size as well as not looking too good if you have tons of different fonts that don’t complement each other. A good guide for choosing fonts is A Beginner’s Guide to Pairing Fonts.

I’ve recently changed what fonts this site uses, using Google Web Fonts; I went with Oswald for headings, and Lora for the body copy.

Embedded ICC profiles and Firefox

Sat, 12 May 2012 00:00:00 +0000

Embedded ICC profiles and Firefox

12 May 2012

I have a separate blog where I post photos I take, and I recently noticed a problem with how the actual JPEGs display in Firefox. It was almost like they were showing the image in a lower colour mode. All other browsers displayed the images fine.

Here is an example using a recent photo off the blog. You can see the blurry background is cutoff in firefox.

The difference is that Firefox is one a couple of browsers in the market (Safari and possibly IE10 seem to be supporting it too) that applies embedded ICC colour profiles in images when displaying them.

I was saving my JPEGs with the ICC profile (sRGB) embedded in them and therefore Firefox was adjusting the colours to match the profile, messing things up.

I’m not 100% sure as to what is happening, but I can hazard a guess. The ICC profile is intended to remove the link between a device dependent colour space (e.g. the camera or your screen) and an independent colour space. I think the image is saved with the colours already converted to the target colour space, but with the profile embedded, Firefox applies the transformation again, thereby effectively applying the conversion twice. I couldn’t find much information about this on the web, so this is just a guess.

The information on the web is a bit thin… I did find this page, but it doesn’t seem to explain the issue I’m having.

So, after all that, what is the fix? Just save the image without the profile in it, and everything displays as intended, across all browsers and image viewers that I tested on (you can also remove the profile of an existing image using tools like exiftool).

XTS-AES implementation: SectorStream

Thu, 22 Mar 2012 00:00:00 +0000

XTS-AES implementation: SectorStream

22 March 2012

As a by-product of the XtsSharp project, I implemented a SectorStream, which is a stream that reads and writes whole sectors

(the size of which is configurable) to enable the XTS algorithm to operate on standard .NET streams.

I also implemented another stream based class that provides random access to an underlying SectorStream. This simulates random access by reading and writing entire sectors at once. This stream is called (yep, you guessed it) RandomAccessSectorStream.

The result is that you can use the random access stream to wrap a SectorStream and read and write to it as if it was just a normal .NET stream. It supports seeking to arbitrary points and reading / writing. The class handles the sector by sector reading and writing, which when using an XTSSectorStream, encrypts (when writing) and decrypts (when reading).

I’m not sure if this is usable outside of the XTS project, but it is completely separate code-wise, so maybe someone will find a use for it.

XTS-AES implementation in .NET

Mon, 19 Mar 2012 00:00:00 +0000

XTS-AES implementation in .NET

19 March 2012

I’ve been meaning to write about this for some time, but laziness, procrastination and work have managed to get in the way.

DISCLAIMER: I am not a cryptographer, cryptography is something that interests me and the library is a proof of concept; you use it at your own risk.

I recently (i.e. a year ago!) wrote an implementation of the XTS algorithm in .NET (C#). I had asked a question on stackoverflow, but didn’t get much of a response. The only other implementations I found were in C or C++.

I eventually implemented it myself by looking at the LibTomCrypt library as well as the actual IEEE standard document. The implementation supports AES-128 and AES-256 and is licensed under the BSD license, so do with it what you will!.

XTS’s main purpose is to enable encryption on block level devices (e.g. hard drives) while allowing random read / write access to the data. The key here is that each block can be encrypted / decrypted without touching the other blocks. Standard modes (like CBC) chain the blocks together, so to decrypt block 1000, you have to decrypt the previous 999. Encrypting each block separately using a mode like ECB is not secure as two blocks with the same content will encrypt to the same cypher text.

XTS gets around this problem by chaining two encryption steps together with a tweak step in the middle. The key input that is quite different from other cipher block modes in that XTS uses the sector number as part of the tweak step. This works quite well, as it is easy to calculate the sector number. Most available disk encryption software uses XTS mode.

The library uses the built in .NET AES implementation internally, and coordinates the two AES encryption transforms along with the actual tweak.

You can see my implementation in my bitbucket repository, and the LibTomCrypt one in their github repository. The entire algorithm is basically implemented in the single XtsCryptoTransform class. It is actually simpler than the LibTom one due to the nature of the .NET ICryptoTransform interface being the same for encryption / decryption. It can also be extended quite easily to use other encryption algorithms, as it depends on the ICryptoTransform interface only.

The library has unit tests that test it against the test vectors in Annex B in the IEEE standards document (which it obviously matches!)

A new home for my blog

Sat, 10 Mar 2012 00:00:00 +0000

A new home for my blog

10 March 2012

This site has now moved to it’s own domain (garethlennox.com), rather than sitting in a subdirectory under another domain.

I managed to move it in the space of a couple of hours, without too many hassles. The main problem was that wordpress doesn’t like to have it’s main url changed from underneath it, so I had to get into the database and tweak some values. There is a good guide in the wordpress codex on how to do this.

Otherwise, barring some obscure file permission problems, it was very straight forward.

One other benefit of having my own domain is that I get to set the favicon for it, which is a smaller version of my logo. I used this handy tool to convert the favicon into a .ico format.

New design!

Tue, 20 Dec 2011 00:00:00 +0000

New design!

20 December 2011

Finally, I’ve updated the design of this site, to brand-new spanking HTML5, including a fluent layout, that adjusts for larger and smaller screens

(it looks great on a phone too!). I created the previous design in 2005, so it was time for a refresh.

The site should work in any modern browser, as well as reasonably modern phone browsers. It even looks kinda ok in IE6!

I am still running Wordpress, with a large bunch of plugins. The new design is based off the Starkers theme - an extremely minimalistic theme (to see how minimialistic, see the demo). It is awesome in that it gives you the HTML framework without any CSS. The end-result? Much less CSS than if you’d started with an existing theme.

The design is intended to be minimal as well, with a focus on the content. I’ve stripped out all the twitter and delicious feeds. If you want those, links to them are at the bottom of the page.

Notable plugins that make this site run:

WP Super Cache: The standard Wordpress caching plugin. I’ve been using this for years, and it works really well.
WP Minify: Combines and minifies all the CSS and javascript files into single requests. This one is new, so we’ll see how it goes. Seems to work nicely.
Akismet: For comment spam. Also brilliant.

It always seems that re-designing this site is way more fun than actually posting on it, which is rather worrying. As usual, I’m going to try to find time to post here more often…

Picasa: Moving albums to another computer

Tue, 04 Oct 2011 00:00:00 +0000

Picasa: Moving albums to another computer

04 October 2011

Just spend hours working out how to do this. If you’re moving pictures to another PC or your Picasa db gets broken, you may lose all your albums.

The album information is stored alongside the Picasa database, in xml files with a .pal extension, which is irritating - you have to back up that folder too.

It is very simple to get your albums back, just do what this thread says, and all works fine: http://www.google.com/support/forum/p/Picasa/thread?tid=4267d3f621027f95&hl=en. All you have to do is copy your .pal files into any folder that Picasa is watching, and it will pick them right up.

This guide worked for me (copied here in case the thread disappears):

Transfer album definitions from Computer A to Computer B. There are the same photos, but placed in different location on the destination computer.

Edit the Albums in Picasa on Computer A

Exit the Picasa on Computer A. The PAL files are created in Picasa2Albums by this.

Copy files to temporary folder and edit the paths to the photos in all the PAL files to conform the photos location on the Computer B.

Start and exit Picasa on Computer B to refresh the Album status in the Picasa internal database

Start Picasa.

All referenced photos from edited PALs have to be visible for Picasa otherwise the Albums are not crated or empty. In order that, make the photos visible for Picasa in Tools -> Folder Manager …

Put the PAL files to any monitored folder. If it is “Scan Once” folder, you have to trigger licking by opening Folder Manager again - just enter, end exit by OK button.

The new Albums are imported to Picasa on Computer B having all photos inside. Source PAL files are deleted.

All the other crap you’ll find around the net about changing the dbid to null and stuff doesn’t work in the latest version of Picasa.

Hopefully the next time this happens to me, I’ll find my own blog post!

8 requirements that good backup software needs to fulfil

Thu, 20 May 2010 00:00:00 +0000

8 requirements that good backup software needs to fulfil

20 May 2010

I’ve been thinking about backup solutions, and have been trying to find one that fits what I need.

I have yet to find one, and this post is the first step in outlining what I need (and hopefully what other people need), with the end goal of building the perfect backup solution.

(disclaimer: these are 8 requirements for me, yours may differ!)

Something to remember: When your hard drive dies, do you have your data safely stored somewhere else? A good test would be if all the hard drives in your pc all stopped working right now, would you lose any critical or valuable data? Do you have backups? Can you restore your backups? What about if your whole house burned down?

I’ve found that available backup software out there is either to complicated, doesn’t do what I want, or too expensive. I’ve listed out what I feel are my requirements for backup software (which means that they’re probably not your requirements!). Note that since my primary machine is running Windows, I’m looking at requirements that Windows supports.

I don’t need a full disk backup, rather on a file-by-file basis. When my drive crashes, I’d probably want to use the opportunity to re-install my operating system anyway.

1. Simple and easy to use

Obviously all software should be easy to use, but why are a lot of backup solutions so complex? Ask me what to backup, where to store the backup and how often to run. Otherwise, leave me alone (obviously if there is a problem, let me know!).

2. Simple backup format

I should be able to open windows explorer, and view the files in a backup set without having to run any special software. This also helps with the first requirement – if your files are stored as files on the drive, anything should be able to read them. When you lose your main drive, it makes it easier if you can plug in your backup drive, and just copy files off it.

3. Actually backup (and restore!) your files

This seems obvious, but I’ve seen backup solutions that will silently backup all your files, but then barf when asked to restore them (looking at you, Windows Backup). The only way to be sure is to try restore your data at regular intervals, no matter which system you use. If the storage format is simple, as above, this is less of an issue, but you should still test it on a regular basis!

4. Stay out of my way

You should not have to remember to run a backup. Backup software should run automatically, and quietly in the background. Once you’ve scheduled it, it should do the rest. Disconnecting an external drive half way through a backup should not cause any problems when you plug the drive in again – it should carry on, ideally where it left off.

5. Handle multiple destinations

I would like to backup to external hard drives, USB flash drives and network computers. The software should automatically handle things like drive letter changes for external USB drives.

I want to define a set of files to backup, then point the software at multiple places. When plugging in a USB drive, the software should automatically kick off a backup if one is scheduled. When connecting to a network, if the destination machine is available, start the backup.

6. Keep history

Being able to go back a couple of months and retrieve a file from then is very handy. The software should do intelligent diffing as well as cleanup of old backup sets. If I’ve changed one 1MB file within of a couple of hundred gigabytes of files, I don’t expect the next backup to take much more than 1MB of extra space.

7. Offline

I have too much data to realistically store online. Living in South Africa exacerbates it because we have such constrained bandwidth availability (e.g. ADSL with 3GB limit). An online backup solution, while nice, is not really an option for most South Africans. If you have a couple of megabytes of word docs, then by all means install something like Dropbox or SugarSync and be done.

8. Quick

Obviously there is a limit to how quick backup software can be – it still needs to enumerate over all files you’re choosing to backup, but a backup of a couple of hundred changed files (from a typical day of work) should take under 5 minutes to complete.

Other nice-to-have features

While these would be nice, they are not critical for me at the moment.

Encryption: Encrypting your files would be nice, but this would break the “Simple format” requirement above. Maybe have an external usb drive encrypted using TrueCrypt?

Handle locked files: Files that are in use should also be backed up. This would involve using Volume Shadow Copy.

Primarily, a backup solution needs to be simple, just work and stay out of my way. Anything I’ve left out?

Wallpaper goodness

Mon, 15 Feb 2010 00:00:00 +0000

Wallpaper goodness

15 February 2010

I’ve uploaded some hi-res wallpapers on my photoblog site, from photos I’ve taken. Have a look, I think they’re quite good!

On Mono

Mon, 07 Sep 2009 00:00:00 +0000

On Mono

07 September 2009

So, I’ve been playing around with Mono on a virtual machine for a while now, and want to just put down some thoughts.

Note that this won’t be an in-depth review of Mono, but just my initial experiences coming from a MS .net background.

(for those of you that don’t know, Mono is an open-source implementation of the .net CLR. Go here for more details. It is actually quite a bit more than that, but coming from using .net, that is the part the interests me – at least at the moment.)

Firstly, one of the interesting things about Mono, is that you can take an assembly compiled using Microsoft’s tools, and run it under Mono, without any changes. Obviously, if you’re talking to the windows registry or using MS specific p-invoke calls, it wont work, but if your code is 100% managed, the chance that it will just work is pretty high.

Tooling

In terms of actually developing directly on Linux and using the Mono tools (rather than the Microsoft ones), there is an IDE called MonoDevelop that comes very close to the functionality of Visual Studio – in places its better (it can even open and save Visual Studio solutions and projects).

All the other usual command line tools are there, most importantly, there is a MSBuild clone called x-build, which lets you use your existing build scripts.

And something that is important for me, NUnit runs fine on mono.

For porting existing projects to Mono, there is a tool, called MoMA, which scans your source code, flagging any areas it identifies as being a potential problem.

Benefits

The most obvious one is that of licensing. Since Mono is open-source, and runs on an open-source operating system, there are zero licensing costs. This is attractive if you’re wanting to have multiple servers running your application. Windows licenses for 20 servers is a non-trivial cost.

Mono is also cross-platform. It runs in environments and on devices where you won’t find the Microsoft .net framework. Taken from the mono site:

Mono is built to be cross platform. Mono runs on Linux, Microsoft Windows, Mac OS X, BSD, and Sun Solaris, Nintendo Wii, Sony PlayStation 3, Apple iPhone. It also runs on x86, x86-64, IA64, PowerPC, SPARC (32), ARM, Alpha, s390, s390x (32 and 64 bits) and more. Developing your application with Mono allows you to run on nearly any computer in existence.

Distribution

I’m not sure about older versions, but the latest version of Ubuntu has Mono installed already. A couple of the pre-installed applications are written in C#.

To be honest, I’ve only used Mono under Ubuntu, so cannot comment on its availability in other distributions, but the Mono website recommends openSUSE.

Conclusion

So, in conclusion, I’m very impressed, well done guys. I’ve not yet had time to take one of our larger projects and get it running on Mono, but that’s the next step.

This is more a note to myself, but below is a list of things I want to investigate further around Mono:

Ease of porting a largish existing application to Mono
Do common frameworks run on Mono? (things like ASP.NET MVC, NHibernate, Castle Windsor etc)
Windows Forms support (this was spotty when I looked at it a couple of years ago – its meant to be better)
Performance vs the Microsoft stack

MSI version matching upgrade woes

Wed, 08 Jul 2009 00:00:00 +0000

MSI version matching upgrade woes

08 July 2009

We’re distributing our .net application using Microsoft’s MSI installer technology (using the very powerful WiX to do it).

We’ve recently seen a problem with the installer not upgrading older versions, but rather just installing on top of the existing version, which results in the application being listed twice in the “Add/remove programs” (Programs and Features in Vista) section, one for each version. This results in weird things happening, like uninstalling one of them leaves the other one still installed with all the application files removed.

I ensured that we kept the same upgrade guid, with the version number incrementing, but all looked good. After searching and trying different things, I finally found this gem, hidden away in the MS documentation:

Note that Windows Installer uses only the first three fields of the product version. If you include a fourth field in your product version, the installer ignores the fourth field.

Why?!??

Our version numbers are in the order of 3.2.0.1234, 3.2.0.1456 (where the 4th number increments based on the Subversion revision number). The installer was seeing the version numbers as 3.2.0, i.e. the same, and not uninstalling the old version. Changing the installer version number to 3.2.1234 fixes this.

WP Super Cache not caching (solution)

Wed, 01 Jul 2009 00:00:00 +0000

WP Super Cache not caching (solution)

01 July 2009

The WP Super Cache plugin for Wordpress is a pretty awesome plugin. The other day I was having problems with it not doing any caching

and spitting out the following just before the head tag (where the wp_head() call is):

<!-- Page not cached by WP Super Cache. No closing HTML tag. Check your theme. -->

This was very odd. Everything I could find on the web regarding this problem was seeing this tag at the end of the html. Eventually I found a post that sounded similar to mine (and of course now I cannot find it again, unfortunately). It mentioned a plug-in using output buffering incorrectly. This makes sense, as looking in the source code of WP Super Cache, it outputs the message above in its output buffer handler if it cannot find a closing tag. Needless to say, after a couple of hours fiddling, it turns out that it was a plug-in that I had written especially for the site, and it was doing exactly what the post mentioned (i.e. it was calling ob_end_flush() without calling ob_start()).

So it was all my fault, and I’ll probably do it in the future, hence this post!

So, note to self: If you see this in the future, check the plug-ins.

Don’t use @@IDENTITY

Tue, 26 May 2009 00:00:00 +0000

Don’t use @@IDENTITY

26 May 2009

If you’re using SQL Server, and you’re inserting records into a table with an auto increment primary key, I suggest you don’t use @@IDENTITY to retrieve the previously inserted Id.

Rather use SCOPE_IDENTITY().

The reason for this is that @@IDENTITY gives you the last generated Id in the current session vs. SCOPEIDENTITY() gives you the last generated Id in the _current scope.

The current session includes any triggers that run on the table after the insert. The current scope is just the current stored procedure that you’re in.

We were bitten by this recently. We were using @@IDENTITY when inserted into a table that is replicated. Upon insert the replication trigger was firing, inserting a record somewhere else. @@IDENTITY was giving us the identity of the record the trigger inserted.

Changing the @@IDENTITY to SCOPE_IDENTITY() fixed the problem.

This was even nastier in that by total fluke, the identity was matching a record that already existed, so when we used the retrieved identity to insert into another table, we didn’t violate the foreign key constraint.

Lesson learnt!

Update: I’ve found a page with an in-depth look at the different techniques. Its for SQL Server 2000, but applies to 2005/2008 too.

Google and search ranking

Mon, 04 May 2009 00:00:00 +0000

Google and search ranking

04 May 2009

I’ve recently finished a new site for a client, and in the process of developing, I uploaded a test site to a web server for the client to review.

Since going live with the actual site, I left the test site up and running (in the assumption that its less effort to leave it up than to take it down).

Unfortunately, google managed to find the test site and ranked it higher than the proper site for a couple of the keywords. Not good! Its a simple fix to put in a mod_rewrite redirect rule, but its quite incredible how quickly google managed to find a site that has no incoming links.

I have a feeling that its related to the site map plugin that we’re using for wordpress, I think that it pinged google when we were developing the site, and google happily went along and indexed the site.

So, if you’re developing a site, and put up a test version, make sure that you either take it down when making the site live, or ensure that it redirects to the correct site.

SQL Compact 3.5 Performance and NHibernate

Tue, 07 Apr 2009 00:00:00 +0000

SQL Compact 3.5 Performance and NHibernate

07 April 2009

We’re using SQL Compact 3.5 in a smart client desktop application, talking to it with NHibernate and are experiencing performance problems in a couple of places.

In general, we’re very happy with the database, it is most of the time pretty fast, and easy to work with. The fact that it runs within our process also helps, as its easy to install and use. This is especially useful, as we are going to be running on relatively low-spec machines (laptops with 500MB ram).

We are however seeing a performance problem in SELECT statements with multiple joins (> 5).

This article warns:

In general, if a significant number of your queries require joins of more than five or six tables, you should consider denormalization.

We are seeing queries that take around 4 minutes, whereas running the same query on a proper sql server takes less than a second.

When dumping out the query plans, and reviewing it, we are seeing in most cases, a full table scan taking place, and joining of all rows before filtering. Running the same query on the full sql server (with the same schema), it uses the indexes, and filters as expected. I’m going to demonstrate this issue, and a possible solution to the problem (tests were done using SQL Compact 3.5 SP1).

I’ve created a simple schema, as defined in the diagram below (should be pretty familiar):

(all tables had 1000 records in them, with the exception of the order_line table, which had ~16 000 records)

I’m going to use the following query, which is a bit contrived, but it is complex enough to display the issue:

SELECT product.name,
 COUNT([order].id) AS order_count
FROM order_line
 LEFT OUTER JOIN [order]
 ON [order].id: order_line.order_id
 LEFT OUTER JOIN product
 ON product.id: order_line.product_id
WHERE [order].id IN ('{list of ~100 integers}')
 AND product.quantity_in_stock < order_line.quantity
GROUP BY product.name
ORDER BY order_count DESC,
 product.name

In SQL Compact, this generates the following query plan:

On the right of the query plan, you can see it joining the order and orderline tables (takes ~97% of the time). The problem is that it does this join _before it filters out the orders, thereby reading all the records in the order_line table (which is obviously big!). This query runes noticeably slower with more data. Running the query on the full SQL Server results in it using the indexes as expected.

The solution to this, is to manually optimize the query yourself, instead of relying on the database engine to do it for you, by removing the LEFT JOINs and inverting the query, using INNER JOINs rather, to force the engine to filter rows out as early as possible (you’d usually write the query like this anyway, if you were doing it by hand, but remember, we’re using NHibernate to generate the query).

Here is the above query, optimized:

SELECT product.name,
 COUNT([order].id) AS order_count
FROM [order]
 INNER JOIN order_line
 ON order_line.order_id: [order].id
 INNER JOIN product
 ON product.id: order_line.product_id
WHERE [order].id IN ('{list of ~100 integers}')
 AND product.quantity_in_stock < order_line.quantity
GROUP BY product.name
ORDER BY order_count DESC,
 product.name

and the corresponding query plan:

If you examine the query plan, you’ll notice that it’s filtering the rows up front, rather than joining and then filtering.

So, in conclusion, it doesn’t seem that SQL Compact optimizes LEFT JOINs properly, and you have to optimize the query yourself to avoid table scans and poor performance.

One important fact to note is that we’re using NHibernate to perform the querying, and its generating sql with lots of LEFT JOINs in it (nothing wrong with that, most databases optimize this quite well). The number of joins are determined by what base object you choose to query on, and how many values from other objects you’re pulling back.

We’ve since replaced the generated queries that show this performance problem with hard-coded named queries, written with inner joins as described above.

Empty unit tests

Tue, 10 Mar 2009 00:00:00 +0000

Empty unit tests

10 March 2009

I was reviewing some code, and noticed a couple of unit tests that had TODO comments in them (and no code).

The problem is that when running these tests, they were passing. Someone reading the list of tests would assume that the test was testing what it claimed to be testing, when in actual fact its doing nothing.

The first option to avoid this problem is to not actually create test methods until you’re ready to implement them.

I know I like to sometimes lay out a list of tests I want to implement before actually writing them. If you want to do this, then I suggest you create the methods, but ensure that the test is ignored (in nUnit, put an [Ignore] attribute onto the method), or that it fails (in nUnit, Assert.Fail). This is especially important if you’re going to commit to your source control system your partially complete tests.

Doing it this way will ensure that

You don’t forget that you need to implement the test.
That someone doesn’t think that something is being tested when in actual fact it isn’t.

Using Yahoo Pipes to aggregate feeds

Tue, 10 Mar 2009 00:00:00 +0000

Using Yahoo Pipes to aggregate feeds

10 March 2009

I’ve got a list of links in the sidebar of this blog. What is not obvious is that this list is an aggregated list of links that come from multiple sources.

At the moment, I mark links as interesting from two sources:

Ideally, I don’t want to have 2 lists on the side, especially when most items I both bookmark in Delicious and share in Google Reader.

Enter Yahoo Pipes. Its an incredibly easy-to-use and powerful tool. You can see the pipe I’ve created here: http://pipes.yahoo.com/pipes/pipe.info?_id=wAcslJ753RGkTFabBR50VA. If you click the “View" Source” link you can see how its put together.

The general idea is to:

add multiple sources (fetch feed)
union them together into one feed
Ensure they are unique, based on the id
Sort them by the publish date
Output the result

The end result is one rss feed that contains the unique list of links and is sorted properly. I just then point the WordPress RSS widget at that feed, and it works.

You can do a lot more fancy things, but what I’ve described above is the simplest.

Refracted

Wed, 04 Mar 2009 00:00:00 +0000

Refracted

04 March 2009

So I’ve decided to start posting my own photo’s onto a dedicated photo blog, you may have noticed the photos appearing in the sidebar (the wonders of rss!).

If you’re interested in photography, you might find it interesting. The blog’s address is http://refracted.co.za/. I’m not going to be posting any more photos on this blog, other than the ones in the sidebar.

On the name

Its tricky coming up with unique, short names that can be turned into domain names. I came up with refracted.co.za the other day, and it seems quite fitting, as when a photograph is created, the light cannot help but be refracted and bent, firstly by the interface between the glass in the lens elements and the air (it’s related to the speed of light in the different mediums - see wikipedia for more information), and by the lens elements themselves.

How to sort rated items

Mon, 23 Feb 2009 00:00:00 +0000

How to sort rated items

23 February 2009

This is a fascinating article:

PROBLEM: You are a web programmer. You have users. Your users rate stuff on your site. You want to put the highest-rated stuff at the top and lowest-rated at the bottom. You need some sort of “score” to sort by.

The article goes into detail about how all the most common sorting algorithms are incorrect. We implemented a system a while back, using wrong solution #2 (from the article), and had all sorts of issues with things getting sorted incorrectly. The proposed algorithm looks rather scary though.

Need to keep this in mind when I next implement a rating-based system.

UPDATE: Another article, now bringing in a time component.

Commented out code

Mon, 16 Feb 2009 00:00:00 +0000

Commented out code

16 February 2009

One of my pet peeves is opening a source code file, and seeing half of it commented out, or blocks within it commented out. What’s wrong with this?

Commented out code serves no purpose, other than to confuse.

If you try uncomment it, chances are that the code around it has changed, and the uncommented code now won’t even compile.

Also, when reading the code and trying to understand it, the chunks of commented out code get in the way. “Should I be taking this code into account, or not?”

A lot of the time it is not obvious as to why it is commented out. I’ve seen lots of pieces of code commented out, with no reason given as to why. Maybe the developer was testing something and forgot to uncomment the code before checking in? Or is it old code that was not needed any more (then why is it still here?).

Code is usually commented out when the developer is not sure if the code will be needed in the future, and is leaving it there “just in case”.

Ultimately, you don’t need to keep commented code around. Your source control system can handle it. As long as you comment your commit properly, its a matter of 5 minutes work to find the old code and resurrect it.

Ribbons

Mon, 16 Feb 2009 00:00:00 +0000

Ribbons

16 February 2009

Took this the other day, just a ribbon hanging off the ceiling.

Turbulence

Fri, 23 Jan 2009 00:00:00 +0000

Turbulence

23 January 2009

Took this the other day from my flat’s balcony.

Looking Up

Thu, 11 Dec 2008 00:00:00 +0000

Looking Up

11 December 2008

So, after much thought, I’ve decided to start posting more photos here. Hopefully it will force me to post more often!

This picture was taken on a yacht for an end of year function.

Learning Zend Framework

Mon, 16 Jun 2008 00:00:00 +0000

Learning Zend Framework

16 June 2008

So, I’m playing with Zend Framework. My initial impressions are that it seems very cool.

The database stuff especially. One thing, and this is not limited to the Zend Framework, is that it’s such a big system, there is a lot of stuff to get your head around, so documentation is critical. Unfortunately, the documentation is written on a per-component basis, so it gives you next to no useful information on commonly encountered problems.

There is a quick start, but it ignores common issues, such as: How do you create a link to another controller (i.e. another part of the system)?

Now, I just spent 30 minutes trying to find this out. The official documentation is silent on this basic task. After searching all over the place, I finally found this site, that tells you to use “the url helper”. There is nothing (or at least as far as I can see) about this url helper in the documentation.

Shouldn’t the documentation cover this sort of thing? Or at least a FAQ somewhere? (there is one on the wiki, here, but its about frequently encountered problems, not questions, and does not cover the above issue).

Maybe there is something I’m talking about, but if it does exist, its not easy to find on google!

The Future of Web Apps summit

Fri, 21 Apr 2006 00:00:00 +0000

The Future of Web Apps summit

21 April 2006

You want to know what all the big guys are doing out there in the “web 2.0” world? Have a listen to the mp3s of the “Future of Web Apps” summit.

There are guys from del.icio.us, Flickr, 37Signals, Google, Yahoo and others speaking.

I have yet to listen to all of them, but the ones I have listened to are top notch.

Editing magic

Tue, 04 Apr 2006 00:00:00 +0000

Editing magic

04 April 2006

TinyMCE rocks! I have been integrating it into our content management system, and I must say its impressive.

Its very easy to use, and Just Works™, unlike some other web rich text editors out there. I managed to build a custom link screen in a couple of hours, with very little javascript. Another bonus is that it outputs nice valid XHTML, unlike other editors.

While using the editor., I have realised that I can write content much faster in our CMS using the default textile formatting, but people want the rich text editor, and it looks very impressive, so we are gonna give it to them :)

Hermanus

Sun, 02 Apr 2006 00:00:00 +0000

Hermanus

02 April 2006

Ahhh…. perfect for skiing! :)

Wordpress 2

Sun, 02 Apr 2006 00:00:00 +0000

Wordpress 2

02 April 2006

This blog is now running under wordpress 2. Its looks very spiffy, but I still need to explore all the changes.

Once thing I noticed is that you can add categories while you are editing a post (I mentioned it here a while ago).

I have turned off the WYSIWYG editor for the moment, because there is no functionality to paste code (i.e. html tags) into the edit box, as it doesn’t escape it properly.

Otherwise, looking good!

Web based feed-readers

Fri, 02 Dec 2005 00:00:00 +0000

Web based feed-readers

02 December 2005

Since I discovered rss feeds, I have tried many feed readers, but all the applications didn’t work for me, I needed a web-based reader.

Why? I need to be able to access it from multiple computers, making it difficult to sync read/unread items. A web app solves this by being accessible from anywhere.

I ended up settling on Bloglines, and used it for about a year. I have recently switched to using Gregarius. Its a nifty PHP feed reader that’s skinnable and has lots of plugins to customise it. The user-interface is also much better than Bloglines, and since its skinnable, if you don’t like how it looks, make your own skin!

Its obviously not a hosted service, so you need some web space, but I much prefer it. Its nice to be in control. :)

Recommended.

Lack of updates

Sat, 10 Sep 2005 00:00:00 +0000

Lack of updates

10 September 2005

Yes, I am still around :)

We are currently working on a big project for the US market, and are having to put in some substantial overtime. Hopefully this won’t go on for too much longer, but you never know…

I should be updating this site more often, but I seem to be just too lazy at the current time to bother writing about anything. (Note to self: stop being bloody lazy).

So, here’s to me not being so lazy…

Learning [insert language here]

Wed, 04 May 2005 00:00:00 +0000

Learning [insert language here]

04 May 2005

I was just reading a post over on whitespace, and it got me thinking about learning any computer language.

It is not enough to read a book and do some examples from the book - you actually need to write something substantial in the language. Granted, it will probably suck, since you would have built it while learning the language. The important thing to note is that it would have forced you to find out and understand exactly how the language works, how things are done, and best practices. The next thing you create will probably be better, and so on.

I am going to use CSS as an example here. Sure, buy that book on CSS, but after reading it, actually go and create a full site using CSS. It doesn’t matter if you need to continually go back and check the book, or the net, or colleagues. At first it won’t be easy, you will be lost, not know what’s going on and generally confused, but if you stick at it, you will find you pick up an amazing amount of information, just by using the language. Before you know it, you will be using the language as if you had always used it.

This is how I learn’t all the languages I currently know, usually without a book, just examples on the net, and a will to learn.

Ajax-enabled user interfaces

Thu, 28 Apr 2005 00:00:00 +0000

Ajax-enabled user interfaces

28 April 2005

There have been lots of people writing about ajax recently, and more importantly about ajax accessibility.

Keeping it simple, I want to focus on what happens when a user has javascript off, or the browser doesn’t support the XMLHttpRequest object. Now this may seem obvious, but your interface should work without javascript enabled. Maybe it won’t be as flashy, or user friendly, but it should work.

Our content management system (CMS) has a caching functionality to speed up the delivery of pages, and while it clears the cache when you change any content, there can be a time where you want to force a refresh. Below is the HTML for the clear cache link.

<a onclick="ClearCache(); return false;" href="clearcache/"> Clear page cache </a>

Notice that if the user has javascript off, none of the `onclick` code will execute, and they will navigate to the `clearcache` page, which clears the cache and informs them so, after which they can navigate back to the previous page. If javascript is on though, the functions `ClearCache` is called, which does all the ajax calls back to the server, which clears the cache, returning a status value. If the status is not an error a messagebox is displayed informing the user that the cache has been cleared, without refreshing the page.

Note that there is a `return false;` after the function call, this effectively cancels the click on the link, stopping the browser from navigating to the clearcache page. An important thing to note is that while the user experience is different in the two cases, they are both perfectly usable. It will even work in a text-only browser, such as Lynx.

One way to ensure that there is always some kind of fallback method is to first get your interface working without ajax. Then you carefully insert the relevant javascript calls.

You could also go the gmail route, and have two totally separate interfaces, one for modern browsers that support ajax, and another plain HTML interface for older browsers, however this is a lot more work, to support (usually) a small minority of users.

Behold! A design!

Wed, 27 Apr 2005 00:00:00 +0000

Behold! A design!

27 April 2005

Woohoo! No default wordpress template to find here any more.

I finally got around to creating a design for this web-log. If you find any dodgy looking areas, please let me know via the comments. I am sure I will be tweaking it, website designs are never finished!

By the way, the header is derived from a photograph of a rock face on Table Mountain, Cape Town.

A couple of (small) issues with Wordpress

Fri, 22 Apr 2005 00:00:00 +0000

A couple of (small) issues with Wordpress

22 April 2005

Having been using wordpress for a little while now, a couple of things bug me.

Don’t get me wrong, it is an amazing tool, and does its job well, but, as always with any software

product, there are a couple of little niggly things.

Maybe its possible to do some of the things I mention below, but I cannot see an obvious way. If there is, please let me know!

Adding categories from the write post page. This is a no-brainer. Currently, to add categories, I have to save my post, and navigate a couple of pages away to be able to add a category.
Intelligent URL transforming. Change images/something.png to something like (in my case) /g/images/something.png. This way if my blog moves in the future, all my image links don’t break. Admittedly, it is a small problem, so this one is a nice to have. Not sure if this is possible with the different plugins (I am using Markdown).

Thats it at the moment. Two slightly annoying things, both of which have a workaround, so I suppose I will just live with it. ;)

Panoramics

Sat, 16 Apr 2005 00:00:00 +0000

Panoramics

16 April 2005

I recently found some useful software that automatically stitches together multiple images to create panoramics.

Other panoramic software requires you to manually identify which picture goes where, but Autostitch does it all automatically.

You give it all the images, and it works out what image goes where, adjusting each one so they blend together. It works suprisingly well.

As an example, I have stitched together a panoramic I made while in the Cedarberg, during Easter 2005 (click the image for a larger version).

I didn’t do any tweaking at all, this is the raw output. Pretty impressive stuff.

Keeping a session alive

Thu, 14 Apr 2005 00:00:00 +0000

Keeping a session alive

14 April 2005

One of the problems we found with people using our content management system was that, frequently, they would take very long editing a page, causing the session to time out, thereby losing their changes.

We could have written code to catch the timout session, and somehow not lose their changes, but we decided on a different course of action, one that I think is much cleaner.

I made use of some javascript and ajax (aka XMLHttpRequest) to ping the server every five minutes. On the server I wrote a special page that would just refresh the session, when requested.

The code is quite simple, we use the setTimeout javascript method to initiate the ping, and ajax to make the call back to the server.

var timeout: 300000; //5 mins
function pingServer() {
 http: getHTTPObject();
 http.open("GET", "http://server/admin/ping/", false);
 http.send(null);
 setTimeout("pingServer()", timeout);
}
setTimeout("pingServer()", timeout);

the setTimeout method takes two arguments, a function name, and a timeout value in milliseconds. The getHTTPObject function, is defined on this blog.

Result? The session timeout of 20 minutes is never hit, as long as the user has their browser open. A much better user experience, I would say!

New blog

Thu, 14 Apr 2005 00:00:00 +0000

New blog

14 April 2005

Welcome to my new blog, we shall see if this lasts…

I will be talking about anything that interests me, the industry that I work in, and anything else I want (it is my site after all!). I work in the software development industry, developing mostly in C# and PHP. I also design websites, and am a keen supporter of web standards.

I am a part owner and director of Dwakn Consulting, a company a couple of mates and I started after varsity. We are concentrating on website design and web application development, and have developed our own content management system (hopefully, I will find some time to reveal some detail about this system).

This site is hosted on Dwakn’s server, and is powered by wordpress.

Hopefully, this won’t be the only post on this site :)