Hacks and Tricks of the Trade at Facebook

Software engineers love hacks that become tricks of the trade.

A hack is regarded as something that gets the job done in a clever way, but it's usually brittle. It's an inelegant, but effective, solution to a computing problem, sometimes referred to as a kludge or jury rig.

Any software engineer who's coded on a daily basis has written a hack. Every so often, a hack rises to the level of innovative breakthrough and is recognized by a prominent person in the industry.

As I've said before, innovation is anything that reduces the cost of a transaction in terms of time or money. The best hacks are the ones that already use the current infrastructure in an innovative way.

Until a decade ago, most every time a website served up a webpage it would go to the database to refetch data. For example, an e-commerce store would look up the products that are on sale each time a user requested that page. The problem was that many pages were served up at the same time, with redundant hits to the database. After all, the list of items on sale isn't going to change from one minute to the next.

About ten years ago, as servers needed to handle more load, there was a rise in open source caching technologies to minimize the number of trips to a database. This was a big gain for read only data, which doesn't change often. Storing data in memory (as a cache does) reduces the overhead of interacting with a database. A database's job is to ensure data reliability by running many checks, formally know as ACID properties. But many of these checks are unnecessary if the data doesn't change very often. So, rather than make a trip to the database, the data is simply stored in memory so it's retrieved much faster.

Facebook Tricks

I recently heard about a brilliantly simple trick that Facebook uses to speed up their site. When a Facebook user logs into their account, their data is fetched from the database. While fetching the data, the user has to wait. The amount of wait time could be imperceptible to the user, or it could be a noticeably long time if the website is under a heavy load. "Heavy load" is a relative term, but Facebook services more than one billion users per day, so saving any amount of time makes a noticeable difference.

Wouldn't it be great if Facebook's servers knew what data a user needed before the user formally requested it? Well, that's effectively what Facebook's done with their little trick that simply involves sending an encrypted UDP (datagram) ahead of the formal TCP/IP request. UDP requests are fire-and-forget, meaning there's a small chance they might not arrive at their destination. TCP/IP, on the other hand, guarantees delivery (or notice of a failed delivery). TCP/IP is the reason that webpages render perfectly compared to the BBS's of the 1980s that used unreliable dial-up modems where static and interference would be misinterpreted as data and displayed as garbage text.

So, the UDP datagram arrives well ahead of the TCP/IP request which enables Facebook's servers to pre-fetch the data and load it in its cache before the formal TCP/IP request arrives. A simple yet elegant way to optimize a website for speed.

Ethernet vs. WiFi: Why WiFi is Way Zippier

Speedtest: 94 Mb/s on LAN vs. 314 Mb/s on WiFi

This week, I bought a new TV and hooked it up at home. The key differences between this model and the previous ones are changes to the remote control and the new tvOS App Store. I figured hardwiring it, on my LAN, would be better than WiFi. My thinking was two fold. First, I expected less of a chance of interference on a LAN connection, than wireless, and more importantly, I also thought the LAN connection would be faster.

I was half right. Of course there's less of a chance of RF interference, since wired is better than wireless. But my TV sits less than six inches away from my wireless router. Interference is unlikely.

I was also half wrong. Surprisingly, the new TV's LAN connection is Fast Ethernet, not Gigabit Ethernet. That means the LAN connection to the TV tops out at 100 Mb/s. But the Internet pipe into my living room is several times zippier than Fast Ethernet.

As I said nearly two years ago, Common Sense Can Be Misleading. Even simple theories need to be tested.

12/25/2015 Update from an Apple senior software engineer who contributed to the AppleTV:
The WiFi chipsets implement more of the protocol stack than ethernet chipsets (this isn't unique to the ATV by a long shot). Thus, using WiFi consumes *less* of the main CPU than ethernet, which is counter intuitive in that a wire is more reliable and requires fewer re-transmits. This also means that a wifi only device will idle sleep using less power than a wired device (if power management is of great concern).

Flying vs. Driving

Flight planning, flight planning, and more flight planning.

When I learned to fly, I thought it would be like boating. Sure, I knew maintenance would be expensive, but I imagined jumping into my plane, at a moment's notice, and heading off into the wild blue yonder.

It turns out, while driving and boating can be relaxing, flying is a bit more stressful on a complex level. For starters, when flying, I have to always be "on," meaning highly attentive, lest I make a catastrophic mistake. Even when cruising on autopilot, I have to continually monitor the instruments while talking to air traffic controllers; and they're not patient if you miss their calls more than once or twice. Driving tends to be very linear, in a single direction. A driver maneuvers based on what's directly ahead. Rarely does a driver worry about what's coming from the sides, never mind above or below as is the case when flying through three dimensions. What's more is that most of the flying I do is to keep my skills and plane from getting rusty. I'd love to fly for leisure, every day, but that's not realistic.

A couple months ago, I noticed the big difference between flying and driving on a trip to Cupertino for a speaking engagement on Apple's design and marketing philosophies. My intent was to fly myself into San Jose Airport. I did my usual planning, the night before, and drove out to the airport at noon. As soon as I got out of my car I noticed it was eerily quiet; like the calm in the eye of a hurricane. Something didn't seem right since the airport, which is usually a whirlwind of activity, was too still. After a minute or two I heard several F/A-18s flying fast, low, and loud. I pulled up a digital chart (map) of the airport and saw that it was under temporary flight restrictions (TFR). I hand't noticed the pending TFR, the night before, which seemed odd. I called the airport operations manager and he confirmed my concerns. The airport had suspended operations while the Blue Angels practiced for the next day's airshow. He also mentioned that the TFR was a moving target since the times kept changing leading up to when it went into effect. The TFR began about 30 minutes before I arrived at the airport and it would be in effect for nearly six hours.

Buttoned up since I was driving instead of flying.

I took a few minutes to do some mental math as I sat in my car listening to the silence, pierced by the roar of jet engines. Flying commercial, on short notice, was prohibitively expensive. My next option was to wait until 5 PM, pick up my flight clearance, and then depart, along with many other flights. That would probably get me to Silicon Valley around 9 PM. My final option was to hit the road and drive for eight hours. That would get me to my destination around 8 PM. I chose to drive.

As I headed up the 5, I couldn't help but notice my immediate mental shift from being outwardly focused on flying to being inwardly focused on me, myself, and I as I daydreamed through LA traffic.