Tricks I Learned At Apple: Steve Jobs Load Testing

When I worked at the Apple Online Store, we would never load test the live website. There was rarely a need. However, it was always an interesting experience to turn the store back on after Steve Jobs walked off stage following one of his keynote presentations. As part of our postmortem, once the store was back online, we'd ask ourselves where the servers were constrained: CPU, network bandwidth, disk I/O, or memory? While it's hard to predict exactly how the entire system would behave in the real world, we had a good idea, before we flipped the switch, thanks to our thorough testing strategies.

Load Testing
Many companies use load testing software to see what kind of load their web app can handle. A common, but flawed way to load test a website is to launch the website and then turn on the load testers. The problem with this technique is that it doesn't give you any idea of how the website will perform if it goes down while it's live. When a website, that is in production, goes down, it must be brought up while under load and things will behave very, very differently. For example, it was discovered, when the iTunes Store first launched, that one of its trusted WebObjects components wasn't thread safe and this bug only presented itself under very heavy load.

Cutting My Teeth
When I first joined the Apple online store, I was paired up with an experienced software engineer so I could get up to speed on the code repository, build process, and unit and component testing. Since the online store was already live, we would never roll out new code without first testing it and gathering detail metrics.

My first task with my coworker was to implement a simple web service which retrieved product information, in the form of a plist, over the network. A simple service like this could normally be written in a day or two, but it took us most of the week as my mentor explained each step to me while we pair programmed our way through the process. (Although we pair programmed, our software methodology was Agile/Scrum, not Extreme Programing. Each team used whichever development technique they agreed on as long as they could stay on schedule. The teams I worked with were fortunate to have formally trained scrum masters who were supported by management.)

Before writing any production code, we'd write our unit tests. All software engineers should be taught to write their API unit tests first – it's a good discipline to learn. Next, we coded using WebObjects/Java with Eclipse/WOLips and we always ran the app in debug mode with key break points so that we could step through the code. I've frequently seen too many software engineers, elsewhere, who just code away as if they're throwing something against the wall to see what sticks.

As soon as we checked in our code, the repository would automatically build all of the applications and run the unit tests against them. If you broke the build then everyone on the team, plus a project manager or two, would receive a notification e-mail identifying you as the culprit.

Token

We had one, highly specialized piece of software code which could only be checked out, worked on, and checked in by a single engineer at a time. You were only allowed to touch this piece of code if you possessed a physical token. In our case, the token was a Darth Tater doll, which had to be conspicuously displayed on the top of your cube or bookcase.

Gathering Metrics

Once our service was code complete, bug free, and checked into the repository we began component testing to gather metrics on the new code. This is another step that's commonly overlooked in novice teams. I suspect that this "gather metrics" step isn't included in The Joel Test because Joel Spolsky's product was a desktop app and not a web app under heavy load (or, perhaps it's implicit in Spolsky's "Do you have testers?" step).

Before we could even consider including our code in the live code branch, we would hit it with many millions of requests. At Apple, we had very sophisticated caching algorithms which could store any number of entries we wanted, depending on our goals. Did we need a cache with only 500 products in it or 50,000? After a cold start, would we need to "warm up" the cache with specific products? How long should we wait, after no hits, before removing a product from the cache to free up memory?

As a side note, our caches were always hash tables. The beauty of a hash table is that it has a Big O notation run time that's constant: O(1). When you're asked, during a job interview, which is the fasted lookup function, don't, as is very common, say, "a binary tree." Perfect hash tables always win, hands down.

Tweaking and Done

We would tweak our code until we had acceptable metrics. Our metrics would measure how much memory the cache used and how long it took for each service request/response to be fulfilled. Depending on our needs, we might shoot for a goal to have 99.7% of our service requests returned within 35 ms, while 95% were returned within 10 ms with no single request taking longer than 50 ms.

These tests were run against a copy of the live database in a production environment. It's not a perfect indication of how the web app would perform once it was live. But it doesn't take long for this to be a great way to set expectations.

At the end of our sprint these metrics would be demoed as part of the Agile definition of "done." The code was now ready to be checked into the QA branch for functional testing before going live.