January 15, 2007
Last week we deployed Trade Me as an ASP.NET 2.0 application. We switched over early on Tuesday morning without even taking the site offline. With luck, nobody noticed. Nonetheless, this is an exciting milestone.
Eighteen months ago all four sites (Trade Me, FindSomeone, Old Friends & SafeTrader) were built using classic ASP, which was starting to show its age. We’ve been working off-and-on since then to bring this code base up-to-date. Most of the heavy lifting was actually done this time last year, when we took the opportunity over the quiet Christmas/New Year period to make a start on the main Trade Me site - taking it from ASP to ASP.NET 1.1.
The opportunity to work on this project was a big part of my motivation for heading home from the UK in 2004. It’s great to reach a point where we can reflect on the huge benefits it has realised, not the least being that we’ve been able to complete this work on our own terms. It’s an awesome credit to the team of people who have made it happen.
I’m pretty proud of the approach we’ve taken. To understand this you really need to understand the motivation for the change in the first place.
In 2004 there were a number of unanswered questions:
How much further could we push ASP before performance was impacted?
Back then, we were actually pretty happy with the performance of our ASP code. It had been tweaked and tuned a lot over the years. We’d ended up building our own ASP versions of a number of the technologies included in ASP.NET, such as caching.
The interaction between ASP and the database, which was (and is!) at the heart of the performance of the site, was pretty carefully managed. For example, we were careful not to keep connections open any longer than absolutely required, etc, etc.
At the application layer we had managed growth by adding more web servers. But, this was something we could only push so far before it would start to create problems for us in other places, most importantly in the database.
While we had confidence that we could continue to work with ASP, that wasn’t necessarily shared by everybody else.
Which lead us to the next problem …
How could we continue to attract the best developers to work with us?
It’s hard to believe now that we managed as well as we did without many of the tools and language features that we now take for granted: compiled code, a debugger, a solution which groups together all of the various bits of code, source control to hold this all together, an automated build/deploy process, … the list goes on.
For obvious reasons, we were finding it increasingly difficult to get top developers excited about working with us on an old ASP application.
And there was lots of work to do. As always seems to be the case, there was a seemingly infinite list of functional changes we wanted to make to the site.
So, that left us with the question that had been the major stumbling block to addressing these problems earlier …
How could we make this change without disrupting the vital on-going development of the site?
Looking at the code we had, it was hard to get excited about improving it, and hard to even know where to start. There was a massive temptation to throw it all out and start again.
But, inspired by Joel Spolsky and the ideas he wrote about in Things you should never do, Part I we decided to take the exact opposite approach.
Rather than re-write code we chose to migrate it, one page at a time, one line at a time.
This meant that all of the special cases which had been hacked and patched into the code over the years (which Joel calls “hairy” code) were also migrated, saving us a lot of hassle in re-learning those lessons.
The downside was that we weren’t able to fix all of the places where the design of the existing code was a bit “clunky” (to use a well understood technical term!) We had to satisfy ourselves in those cases with “better rather than perfect”. As it turned out, none of these really hurt us, and in fact we’ve been able to address many of them already. Once the code was migrated we found ourselves in a much stronger position to fix them with confidence.
Because we had so much existing VBScript code we migrated to VB.NET rather than C# or Java or Ruby. This minimised the amount of code change required (we enforce explicit and strict typing in the compiler, so there was a fair amount of work to do to get some of the code up to those standards, but that would have been required in any case).
We kept the migration work separate from the on-going site work. When migrating we didn’t add new features and we didn’t make database changes. When we were working on site changes we made them to the existing code, leaving it as ASP if necessary, rather than trying to migrate the code at the same time.
We focussed on specific things that we could clean-up in the code as part of the migration process. For example, we added an XHTML DOCTYPE to all pages and fixed any validation errors this highlighted. We moved all database code into separate classes. And, we created controls for common UI elements (in most cases replacing existing ASP include files). We also removed any code which was no longer being used, including entire “deadwood” pages which were no longer referenced.
To build confidence in this approach we started with our smaller sites: first SafeTrader and Old Friends followed by FindSomeone then finally Trade Me.
After each site was migrated we updated our plans based on what we’d learnt. The idea was to try and “learn early” where possible. For example, after the Old Friends migration we realised we would need a better solution for managing session data between ASP and ASP.NET, so we used the FindSomeone migration as a test of the solution we eventually used with Trade Me. The performance testing we did as part of the early migrations gave us confidence when it came time to migrate the larger sites.
We re-estimated as we went. By keeping track of how long it was taking to migrate each page we got accurate metrics which we fed into future estimates.
Finally, we created a bunch of new tools to support our changing dev process. For example, we created a simple tool we call “Release Manager” which hooks into our source control and is used to combine various code changes into packages which can then be deployed independently of our test and production environments. We created an automated process, using Nant, which manages our build and deploy, including associated database changes. More recently we implemented automated code quality tests and reports using FxCop, Nunit and Ncover. All of these mean that, for the first time, we can work on keeping the application itself in good shape as we implement new features.
This has been an exciting transformation. The migration was completed in good time, without major impact on the on-going development of the site - we made it look easy! We added four new developers to the team, all with prior .NET experience, and we got all of our existing dev team members involved in the project, giving them an opportunity to learn in the process. Having more people, along with process improvements and better tools, has enabled us to complete a lot more site improvements. We’re in good shape to tackle even more in the year ahead. We’ve even been pleasantly surprised by the positive impact on our platform, which has allowed us to reduce the number of web servers we use (there are more details in the Microsoft Case Study from mid last year, if you’re interested in this stuff).
As is the nature of this sort of change, we’ll never really finish. With the migration completed we’ve started to think about the next logical set of improvements. It will be exciting to see how much better we can make it.
If you’re interested in being part of this story, we’re always keen to hear from enthusiastic .NET developers. Send your CV to careers@trademe.co.nz.