I had a website with an issue. This site is successful on several levels… It gets a good number of visits (500,000/month), each visitor hits nearly 10 pages per visit (so around 5,000,000 page views/month), and ranks well with search engines. While this might sound like a slightly above-average site, the problem lies with it’s key feature… It is almost exclusively devoted to letting any visitor hotlink images and other files from this site onto their personal sites (blogs, social sites, etc). This means that on top of a good amount of what most people call legitimate traffic, there are also thousands upon thousands of pages all around the web that make requests to our server.

These files and images are each mostly between 8-20k in size. So, let’s separate the main issues:

Server Load
IIS doesn’t offer too much in the way of performance tuning, so I switched to Apache. Apache offered more of what I wanted, but similarly crashed and burned on a daily basis. I did try, out of desperation, hosting these files on a couple different shared hosting plans, but with both services I tried, I brought their servers down mere seconds after switching DNS over to them.

Page Load Time
Part of how the site works is offering visitors gallery-style pages to preview images they want to use on their sites. The average page has on it around 60 images, not including the look and feel images (logo, navigation, etc). IE and Firefox, by default, limit the number of simultaneous connections they will make to a server (I believe that with IE it’s 4, and with Firefox it’s 8), which was making the relatively small images appear to load quite slowly.

Fixing the page load time was actually the easiest task. I needed a way to trick the browser into making more connections. There were two key steps involved:

1. Setting up DNS. I set up four sub-domains to all point to the same unique IP address (server1.mysite.com, server2, server3, server4). I then had to adjust my website code to not just display a relative path to the image in question, I had to randomly display a fully qualified path to one of the four sub-domains I had set up. 60 images divided among 4 servers, with 4 simultaneous connections at the browser meant that the browser only has to make 3-4 queued sets of requests. Yes, this means that I would potentially be circumventing the browser’s built-in cache system, but most users have the cache set to default, which means the HTML will get cached, which in turn means the same images are called and retrieved from cache. After this, I would get around 16-100 requests/second just for images, bursting even higher, occasionally.

2. Now the server is well loaded. I was restarting the server nightly to combat the crashes, and even wrote a monitoring script that would automatically restart apache when it would go down 1-2 times each day. What I needed was an extremely lightweight and high-performance image server. Doing some research, I found my answer from the Ruby on Rails crowd…. Lighttpd (aka Lighty). It runs on multiple platforms, is well-documented, and is quite easy to set up. In fact, my config file is quite short:

I won’t go over the entire configuration, but the key settings are server.max-worker (number of threads to spawn), server.max-fds (number of file descriptors), and server.stat-cache-engine (the "fam" setting enables the File Alteration Monitor, a higher performance way of checking to see if a file has been modified since last read). I was able to set Lighty to the same root directory as IIS, and then excluded the ColdFusion and other non-asset files using the static-file.exclude-extensions setting. Of course, everyone will need to tune their settings to their site, but my settings are based on the performance section of the Lighty documentation.

Since switching to this IIS (for ColdFusion) and Lighttpd (for images and other assets) setup, I haven’t had any crash-related downtime. Page views are up year-over-year, and complaints are now few and far between. In fact, all these services are running on the same physical hardware, thanks to the magic of virtualization. But that’s for another blog post.