Hacks and Tricks of the Trade at Facebook

Software engineers love hacks that become tricks of the trade.

A hack is regarded as something that gets the job done in a clever way, but it's usually brittle. It's an inelegant, but effective, solution to a computing problem, sometimes referred to as a kludge or jury rig.

Any software engineer who's coded on a daily basis has written a hack. Every so often, a hack rises to the level of innovative breakthrough and is recognized by a prominent person in the industry.

As I've said before, innovation is anything that reduces the cost of a transaction in terms of time or money. The best hacks are the ones that already use the current infrastructure in an innovative way.

Until a decade ago, most every time a website served up a webpage it would go to the database to refetch data. For example, an e-commerce store would look up the products that are on sale each time a user requested that page. The problem was that many pages were served up at the same time, with redundant hits to the database. After all, the list of items on sale isn't going to change from one minute to the next.

About ten years ago, as servers needed to handle more load, there was a rise in open source caching technologies to minimize the number of trips to a database. This was a big gain for read only data, which doesn't change often. Storing data in memory (as a cache does) reduces the overhead of interacting with a database. A database's job is to ensure data reliability by running many checks, formally know as ACID properties. But many of these checks are unnecessary if the data doesn't change very often. So, rather than make a trip to the database, the data is simply stored in memory so it's retrieved much faster.

Facebook Tricks

I recently heard about a brilliantly simple trick that Facebook uses to speed up their site. When a Facebook user logs into their account, their data is fetched from the database. While fetching the data, the user has to wait. The amount of wait time could be imperceptible to the user, or it could be a noticeably long time if the website is under a heavy load. "Heavy load" is a relative term, but Facebook services more than one billion users per day, so saving any amount of time makes a noticeable difference.

Wouldn't it be great if Facebook's servers knew what data a user needed before the user formally requested it? Well, that's effectively what Facebook's done with their little trick that simply involves sending an encrypted UDP (datagram) ahead of the formal TCP/IP request. UDP requests are fire-and-forget, meaning there's a small chance they might not arrive at their destination. TCP/IP, on the other hand, guarantees delivery (or notice of a failed delivery). TCP/IP is the reason that webpages render perfectly compared to the BBS's of the 1980s that used unreliable dial-up modems where static and interference would be misinterpreted as data and displayed as garbage text.

So, the UDP datagram arrives well ahead of the TCP/IP request which enables Facebook's servers to pre-fetch the data and load it in its cache before the formal TCP/IP request arrives. A simple yet elegant way to optimize a website for speed.

Ethernet vs. WiFi: Why WiFi is Way Zippier

Speedtest: 94 Mb/s on LAN vs. 314 Mb/s on WiFi

This week, I bought a new TV and hooked it up at home. The key differences between this model and the previous ones are changes to the remote control and the new tvOS App Store. I figured hardwiring it, on my LAN, would be better than WiFi. My thinking was two fold. First, I expected less of a chance of interference on a LAN connection, than wireless, and more importantly, I also thought the LAN connection would be faster.

I was half right. Of course there's less of a chance of RF interference, since wired is better than wireless. But my TV sits less than six inches away from my wireless router. Interference is unlikely.

I was also half wrong. Surprisingly, the new TV's LAN connection is Fast Ethernet, not Gigabit Ethernet. That means the LAN connection to the TV tops out at 100 Mb/s. But the Internet pipe into my living room is several times zippier than Fast Ethernet.

As I said nearly two years ago, Common Sense Can Be Misleading. Even simple theories need to be tested.

12/25/2015 Update from an Apple senior software engineer who contributed to the AppleTV:
The WiFi chipsets implement more of the protocol stack than ethernet chipsets (this isn't unique to the ATV by a long shot). Thus, using WiFi consumes *less* of the main CPU than ethernet, which is counter intuitive in that a wire is more reliable and requires fewer re-transmits. This also means that a wifi only device will idle sleep using less power than a wired device (if power management is of great concern).

Flying vs. Driving

Flight planning, flight planning, and more flight planning.

When I learned to fly, I thought it would be like boating. Sure, I knew maintenance would be expensive, but I imagined jumping into my plane, at a moment's notice, and heading off into the wild blue yonder.

It turns out, while driving and boating can be relaxing, flying is a bit more stressful on a complex level. For starters, when flying, I have to always be "on," meaning highly attentive, lest I make a catastrophic mistake. Even when cruising on autopilot, I have to continually monitor the instruments while talking to air traffic controllers; and they're not patient if you miss their calls more than once or twice. Driving tends to be very linear, in a single direction. A driver maneuvers based on what's directly ahead. Rarely does a driver worry about what's coming from the sides, never mind above or below as is the case when flying through three dimensions. What's more is that most of the flying I do is to keep my skills and plane from getting rusty. I'd love to fly for leisure, every day, but that's not realistic.

A couple months ago, I noticed the big difference between flying and driving on a trip to Cupertino for a speaking engagement on Apple's design and marketing philosophies. My intent was to fly myself into San Jose Airport. I did my usual planning, the night before, and drove out to the airport at noon. As soon as I got out of my car I noticed it was eerily quiet; like the calm in the eye of a hurricane. Something didn't seem right since the airport, which is usually a whirlwind of activity, was too still. After a minute or two I heard several F/A-18s flying fast, low, and loud. I pulled up a digital chart (map) of the airport and saw that it was under temporary flight restrictions (TFR). I hand't noticed the pending TFR, the night before, which seemed odd. I called the airport operations manager and he confirmed my concerns. The airport had suspended operations while the Blue Angels practiced for the next day's airshow. He also mentioned that the TFR was a moving target since the times kept changing leading up to when it went into effect. The TFR began about 30 minutes before I arrived at the airport and it would be in effect for nearly six hours.

Buttoned up since I was driving instead of flying.

I took a few minutes to do some mental math as I sat in my car listening to the silence, pierced by the roar of jet engines. Flying commercial, on short notice, was prohibitively expensive. My next option was to wait until 5 PM, pick up my flight clearance, and then depart, along with many other flights. That would probably get me to Silicon Valley around 9 PM. My final option was to hit the road and drive for eight hours. That would get me to my destination around 8 PM. I chose to drive.

As I headed up the 5, I couldn't help but notice my immediate mental shift from being outwardly focused on flying to being inwardly focused on me, myself, and I as I daydreamed through LA traffic.

How Not to Get Rich

It's almost needless to say that business pitches like "Get Rich Now!" or "Grow Your Business Revenue by 10x!" or "Earn $90,000 Working From Home!" should be avoided. It's not that these businesses don't want you to succeed, rather it's that their priority is to get you to buy their system or program, at any cost.

Then

Twenty years ago, I was browsing magazines at my local Borders bookstore. A forty-something-year-old man approached me and said, "I can tell, by the magazines you're looking at, that you're an interesting person." Flattery will get you everywhere. After a couple minutes of chit-chat he told me that he trusted me enough to give me a cassette tape with some business opportunities. It only cost me my phone number. I listened to his tape. It was a recording of a 15 minute group presentation about getting rich, living your dreams, having enough money, etc. As I listened, I kept wanting to know more. Specifically, how do I do it? How do I get rich? Then I realized what it was. When he inevitably called me, I asked, "How is this different than Amway?"

"We are Amway," he exclaimed.

Thanks, but no thanks. One of many problems with Amway is that they treat every person as if they can be turned into a hard-sell sales person. That's like assuming we can make every person a software engineer. To each their own. Engineers and sales people are wired differently.

Now

This past week, I attended a free presentation with headlines similar to those I mentioned above. I knew exactly what to expect, and reality was inline with my expectations. These pitches follow the tried and true "amway-ish" techniques. "Would you like to earn an extra $2,000/month?" Of course you would. Who wouldn't?

Here are the simple tell-tale signs:
1. Pump the benefits.
2. Hide the features.
3. Offer a single solution: Buy my money making system.

You'll see the same routine over and over again. These companies will push their benefits hard, with details, without explaining a single, actionable feature other than buying their system. The tripwire is something like, "I charge $500/hour, but I'll give you a free hour to see if you can be accepted into my sales program." Adding scarcity is another key selling point.

Generally, the benefits of their system will be explained, in detail. "I used to do this, but now I do this." This is an excellent story telling technique I learned from Joyce Maynard. "I used to work hard for six months to earn $2,400. Now, I only need to work for six days to earn $24,000. And my program can show you how to do it." While the benefits are plentiful, the features are scarce. The "How is it done?" details are no where to be found during the initial pitch. And, when you hear it, it's almost always a let down.

My guard was down when I was in Borders, flipping through magazines. And cheap sales talk is designed to catch the attention of the unguarded. And, as I mentioned, you'll hear little to no details on the features of the system. In other words, "What do I need to do to achieve success?" or "How does it work?" is missing. That requires signing up for the program.

So, why do people continue to fall for it? PT Barnum is credited with answering that question.

Imagining the Invisible

In the 1990s, I read Information Anxiety. It's written by Richard Saul Wurman, the creator of the TED talks and the Access travel guides. Wurman wrote about how to manage information. The anxiety he speaks of stems from the explosion of information; the fact that an issue of the New York Times contains more information than the average person in seventeenth-century England would have encountered in a lifetime.

Wurman suggested ways of coping with the overload of information by visualizing the invisible, such as size or distance and compare them to tangible things. For example, an inch is the diameter of an American quarter coin, six inches is the length of a U.S. dollar bill, an acre is roughly the size of a football field, without the end zones.

Road Trip

A common analogy I frequently make and forget (which is my primary purpose of this blog post) is explaining how far San Diego is from San Francisco. In raw distance terms, it's about 500 miles. That's the same as driving from Washington, D.C. to Kittery, Maine, which is at the southwest tip of that state. That's a long distance, but it pales in comparison to the 830 mile trek from the southeast tip of Texas (Brownsville), due north, to that state's most northeastern point in Follett.

Telling a Great Story

My Silas Wood 6th grade teacher, Ms. Cooke, speaking about South Huntington Schools Hall of Fame inductee and Bank of America executive, Kieth Cockrell. 

What's the secret to telling a great story? It depends on a lot of things. Mostly, though, it's important to know your audience. The topic of your story doesn't even have to be interesting, rather, how you tell a story is key. A little levity and drama is helpful, when appropriate. Great storytellers have a way to pull in their audience without shutting them out; and the latter part is key – think about great mysteries with surprise endings.

A few years ago, I began writing fiction. I simply sat down at my computer and wrote a few short stories. And I made some classic mistakes, such as beginning a story with weather and writing the cliche story about a divorced woman and a sick dog.

I had no idea if my storytelling was good, so I went to a professional, Joyce Maynard. Joyce's biography always begins with the fact that she wrote her first book while living with the author of The Catcher in the Rye, J.D. Salinger. What adds to the allure of this tale is that, at the time, Joyce was 19 years old and Salinger was 53. But it was truly the high quality of her writing that got the attention of Salinger when, at 18, her article, "An 18-Year-Old Looks Back On Life" was the cover story of The New York Times Magazine in 1972. A key thing I learned from Joyce is that the story behind the story can be more interesting than the story, itself, since it's more exciting to show people what's going on behind the curtain.

The Cooke's In

Even more important is the storytelling technique of summarization that I learned from my sixth grade teacher, Ms. Cooke.

Last weekend, I had a chance to spend a couple days with Ms. Cooke – something I hadn't done for five years – and it was highly enthralling.

I, along with several other 6th grade classmates, still connect with Ms. Cooke because she had (and still has) a strong presence in our lives. She cared about us as students and she was a great storyteller. She's a smart, independent person who neither tolerates fools nor stupidity. As a teacher, she wanted to teach her students not only book smarts, but also key lessons in life. The quintessential "teach a person to fish" by showing us critical thinking skills.

After more than 30 years in the classroom she retired and opened The Cooke's In restaurant for more than a dozen years. Shortly after retiring from her restaurant, in 2009, I spent a couple days at her house scanning school photos covering three decades, followed by a mini 6th grade reunion where we got to hear more of her stories.

Ms. Cooke preparing her spectacular jerk chicken.

Fast forward to two weeks ago, the night before my 30th high school reunion, when about half a dozen of us from my 6th grade class descended on a classmate's house in Amagansett, NY, in the Hamptons, for a couple days of reminiscing. Of course, Ms. Cooke joined us and prepared her spectacular jerk chicken.

While listening to her speak, I learned another key storytelling secret: keep your story short and to the point. As we hung out in the kitchen and spoke about how good her food was Ms. Cooke said, "You know that the secret is to running a restaurant? Expediting."

Do you see what she did there? She summarized her entire story – a story she had yet to tell – in a single word that drew us all in.

"What do you mean?" we asked.

She told us that running a restaurant is about coming up with a recipe and being able to make it the exact same way, every time, and to do that quickly. She told us the obvious, but it carries more weight when spoken by the voice of experience. She told us just enough to pull in our attention and then she answered our question without waste of time or words. We speak about "active writing," but active talking is equally important since it's a key essence of storytelling, regardless if it's fact, fiction or marketing.

Guy Kawasaki: Lessons of Steve Jobs

Lessons of Steve Jobs keynote

Posted by Guy Kawasaki on Wednesday, October 28, 2015

The Apple Way of Design and Marketing

1 Infinite Loop: The Mothership

I figured, "Why not toot my own horn?" and brag about my Apple speaking engagements. I'm surprised how well it's been received, probably because it touches on a some key design, marketing, and branding topics.

I've given The Apple Way of Design and Marketing presentation a number of times, mainly to Chinese delegations of business people touring America's tech companies. Having worked at Apple, I forget the allure of how the company looks from beyond Infinite Loop and what makes Apple unique. Like a beautiful photo, things look amazing from the outside, where you wish you could touch the magic, on the inside.

My presentation touches on a variety of topics such as: why the Apple logo used to be upside down, to Apple's sophisticated unifying branding techniques, to simplifying UX and technology processes, along with Apple's brilliant decisions and mistakes.

Twitter Unique

Twitter is the haiku of the new millennium.

But, there is serious discussion to remove the 140 character limit in tweets.

Removing a tweet's 140 character limit is the equivalent of showing up at a haiku conference and telling everyone to stop using 5-7-5 and, instead, to use 7-5-7 since that would generate more content.

If Twitter removes the limit, what will differentiate it from other platforms, such as Tumblr, Medium, Blogger, etc? Twitter has recently removed this limit in their direct messages and I've already felt the negative effect of this decision.

In Corporate America, we detest e-mail because of its unlimited length. But Twitter's loved because it's short and to the point. The beauty of 140 characters is I always finish reading what I've started. ​It keeps everything short and to the point with the option to link to detailed content.

A few weeks ago, I blogged about the benefits of the 140 character limit in direct messages (DMs). Recently, I had a customer service issue I addressed through DMing. My DM was not read, in detail since it was more than 140 characters. The CSR asked me for information I had previously included in my original DM. At 500 characters, my DM, written like a short e-mail, was too long. It's a fast and noisy world out there – clear and concise communications is appreciated more than we realize. 

Unique

Twitter is unique. It's both a micro-blogging platform and a two-way communications medium. One key feature that makes Twitter unique is the mention. What other blogging platform enables this type of communication? Remove the 140 character limit and you have what we already have with Medium, or most any other blog.

We the People of the Internet used to enjoy Posterous.  It was a longer form than Twitter, but simpler than Tumblr. You could even blog on Posterous without setting up an account. Then Twitter acquired Posterous and shut it down (so as to not compete with Twitter). Twitter gave me my Posterous content, but it looked nothing like it did when Posterous was alive and kicking.

My favorite part of Twitter is the absence of an over-lawyered, one-sided, postscript disclaimer: This E-mail and any attachments are private, intended solely for the use of the addressee. If you are not the intended recipient... When has that disclaimer been of any use in an e-mail? Increasing the Twitter character limit beyond 140 characters will definitely add more noise. And I also fear that it would lose its simple two-way dialog via mentions.

Active Writing

For me, the personal beauty of Twitter is it has improved my writing, in a Hemingway sense. I enjoy writing fact, fiction, non-fiction, and narrative. A key to writing well is using the active voice which is direct and to the point, much like Twitter. Many times, I've drafted a "long-winded" tweet, that wouldn't fit into 140 characters, and had to pare it down to a more active voice:

never heard before vs. never heard
tell anyone else vs. tell anyone
a friend of mine vs. a friend
taking care of vs. fixing

But, in the end, Twitter, the corporation, needs to grow, so it must generate more content.

How much better would Twitter be if it expands beyond 140 characters? Fundamental changes like this are bold risks. Perhaps it will payoff. Then again, what is haiku without 5-7-5?

Simplicity is the ultimate sophistication.
–Da Vinci.

PS – Tweetstorms, on the other hand, are a great way to link tweets together to say more.

1 Infinite Loop Apple Store

Last weekend, the new Apple Store at 1 Infinite Loop reopened, after being closed for remodeling for a few months, and I had a chance to visit it during its opening week.

If memory serves, at the end of the last century, this store was called the Apple Store. After 2001, it was renamed The Company Store since it was unique in the truest sense of the word. Now, it's been rechristened, once again, as the Apple Store, manned by Apple retail employees who comprise 50% of Apple's 110,000+ workforce.

What makes the store at 1 Infinite Loop (called IL1 by Apple employees) unique is it's the only store that sells Apple logoware such as t-shirts, coffee cups, water bottles, pens, etc. I suspect that it's the only Apple Store without a Genius Bar, too.

See the video of the inside of the new Apple Store at IL1, below or, even better, checkout the raw super-high resolution HD video.

  1 Infinite Loop Apple Store.