Questions from Tim Haines, Part I

This is Part I in a two-part series. Part II covers the Trade Me development process and tools.

It’s been a while since we got geeky here, so …

After my recent post about our migration to ASP.NET I got sent a bunch of questions from Tim Haines. I thought I’d try and pick these off over a couple of posts.

To start with, a few questions about our application architecture:

Q: What’s the underlying architecture of Trade Me - presentation layer / business logic / data layer / stored procedures? All isolated on their own servers?

Q: Are there any patterns you find incredibly useful?

Q: Do you use an O/R mapper or code generator, or is all DB interaction highly tuned?

Q: What third party libraries do you use for the GUI? I see you have Dustin’s addEvent. Follow any particular philosophy or library for AJAX?

Here is a basic diagram I use to represent this application architecture we use on all of our sites at Trade Me (click for a larger view):

We’ve worked hard to keep this application architecture simple.

There are two layers within the ASP.NET code + one in the database (the stored procedures). I’ll start at the bottom of the diagram above and work up.

Data Layer

All database interaction is via stored procedures. This makes it easier to secure the database to threats like SQL injection. And it also makes it easier to monitor/trace the performance of the SQL code and identify where tuning is required.

Within the application we manage all access to the database via the Data Access Layer (DAL).

All of the classes within the DAL inherit from a common base class, which is a custom class library we’ve created (based loosely on the Microsoft Data Access Application Block). This base class provides all of the standard plumbing required to interact with the database - managing connections, executing stored procedures and processing the results.

The methods within the DAL classes themselves specify the details of the database logic - specifying which stored procedure to call, managing parameters, validating inputs and processing outputs.

So, for example, to process a new bid we might implement this DAL method:

Public Sub ProcessBid(ByVal auctionId as Integer, ByVal bidAmount as Decimal)
	ExecuteNonQuery("spb_process_bid", _
		New SqlParameter("@auction_id", auctionId), _
		New SqlParameter("@bid_amount", bidAmount))
End Sub

A couple of things to note here:

• All of our code is VB.NET, so that’s what I’ll use in these examples. Apologies to those of you who prefer curly brackets. Perhaps, try this VB.NET to C# converter 😉
• Obviously (hopefully!) this is not actual Trade Me code - just an example to demonstrate the ideas.

When we need to return data from the DAL we use Model classes. These are thin container classes which provide an abstraction from the data model used within the database and mean we don’t need to hold a database connection open while we process the data.

A simplistic Model class might look like this:

Public Class MemberSummary
	Public MemberId as Integer
	Public Name as String
End Class

Some Model classes use properties rather than exposing public member variables directly, and a few include functions and behaviours, but most are just a simple collection of public member variables.

Model classes are always instantiated within the DAL, never within the Web layer. We don’t pass Model objects as parameters (if you look closely at the diagram above you’ll notice the lines through the Model layer only goes upwards). This gives us an explicit interface into our DAL methods.

So, to get a list of members from the database we might implement this DAL method:

Public Function GetMemberSummaries() As IList(Of Model.MemberSummary)

	Dim list As New Generic.List(Of Model.MemberSummary)
	Dim dr As SqlDataReader = Nothing
	Try
		dr = ExecuteDataReader("spt_get_member_summary")
		While dr.Read()
			Dim item as New Model.MemberSummary
			item.MemberId = GetInteger(dr, "member_id")
			item.Name = GetString(dr, "name")
			list.Add(item)
		End While
	Finally
		If Not dr Is Nothing AndAlso Not dr.IsClosed Then</pre>
			dr.Close()	
		End If
	End Try
	Return list
End Function

DAL methods are grouped into classes based on common functionality. This is an arbitrary split - in theory we only need 6 DAL classes (one class per connection string variation), but in practice we currently have 47.

The two examples above show the patterns that make up the vast majority of DAL methods.

While we don’t use an O/R mapper, we have created a simple tool, which we call DALCodeGen. Using this we can simply specify which proc to call and the tool generates the DAL method and, if appropriate, Model class. This code can then be pasted into the project and tweaked/tuned as required.

Web Layer

All the remaining application code sits in the Web layer. This is a mixture of business and presentation logic, which in part is a reflection of our ASP heritage.

During the migration we created controls to implement our standard page layout, such as the tabs and sidebar which appear on every Trade Me page. These were previously ASP #include files. We’ve also implemented controls for common display elements such as the list and gallery view used when displaying a list of items on the site.

We have a base page class structure. These classes implement a bunch of common methods - for example, security and session management (login etc), URL re-writing, common display methods, etc.

Most of the page specific display code is currently located in methods which sit in the code behind rather than in controls.

We don’t use the built-in post-back model - in fact ViewState is disabled in our web.config file and only enabled on a case-by-case basis as required (typically only on internal admin pages).

With the exception of addEvent we also don’t currently use any third-party AJAX or JavaScript libraries. To date none of the AJAX functionality we’ve implemented has required the complex behaviours included in these libraries, so we’ve been able to get away with rolling our own simple asynchronous post/call-back logic.

Layers vs. Tiers

Each of the yellow boxes in the diagram above is a project within the .NET solution, so is compiled into a separate .NET assembly. All three assemblies are deployed to the web servers and the stored procedures, obviously, live on the database servers. So, there are only two physical “tiers” within this architecture.

Inspirations

There is no such thing as an original idea.

Most of this design was inspired by the PetShop examples created by Microsoft as a means of comparing .NET to J2EE. These were pretty controversial - there was a lot of debate at the time about the fairness of the comparison. Putting the religious debate to one side, I thought the .NET implementation was a good example of an application designed with performance in mind, which was obviously important to us.

Another reference I found really useful when I first started thinking about this was ‘Application Architecture for .NET, Designing Applications & Services‘ which was published by the Patterns & Practices Group at Microsoft. This is still available, although likely now out of date with the release of ASP.NET 2.0. It’s also important to realise that this book is intended to describe all of the various aspects that you might include in your architecture. Don’t treat it as a shopping list - just pick out the bits that apply to your situation.

Disclaimer

I’m a little reluctant to write in detail about how we do things. I’d hate to end up in the middle of a debate about the “right way” to design or architect an application.

Should you follow our lead? Possibly. Possibly not.

I can say this: if somebody has sent you a link to this saying “look, this is how Trade Me does it … it must be right” they are most likely wrong. You should at least make sure they have other supporting reasons for the approach they are proposing.

A lot of our design decisions are driven by performance considerations, given our size and traffic levels. These constraints probably won’t apply to you.

In other cases we choose our approach based on the needs of the dev team. We currently have 8 developers and ensuring that they can work quickly without getting in each others way too much is important. Smaller or larger teams may choose a different approach.

Also, a lot of our code still reflects the fact that it was recently migrated from an ASP code base. If you’re creating an application from scratch you might choose to take advantage of some of the newer language features which we don’t use.