Bread and Water

When I was a Midshipman at the Naval Academy I took a Law for the Junior Officer course. It covered the procedures, processes, and punishments of military law and the UCMJ.

Our instructor was a JAG Navy commander. He earned a Bronze Star during Desert Storm by monitoring radio networks and preventing a fratricide incident in the heat of battle.

He taught us a great lesson in creativity that he learned when he was a legal advisor to an aircraft carrier captain.

A Navy ship's captain, who is the commanding officer of the ship, can hold legal proceedings and dish out punishment. One punishment that's still on the books in the U.S. Navy is bread and water. To receive this punishment, a prisoner's first certified as healthy by the ship's doctor before confined to the brig.

My law professor told us that the bread and water punishment didn't work out as well as expected. The delinquents ended up bragging, after their incarceration, about how they survived the "old man's" most severe punishment allowed by law. Instead of a punishment, it became a point of pride.

What to do?

My professor had his ship's legal team take a closer look at the bread and water regulation:

(A) if imposed upon a person attached to or embarked in a vessel, confinement on bread and water or diminished rations for not more than three consecutive days;

A light bulb went off when they read, "diminished rations." Instead of serving bread and water, the prisoner was now fed baby food. Same caloric content, just a different medium.

Telling shipmates they survived three days on baby food ended the bravado.

Author: Joe Moreno

My Early Days At Apple

In the late 1990s, when I told people that I worked at Apple, it was like telling them that I worked at Atari or Commodore. "You don't hear much from Apple, anymore," someone once said to me.

The funny part, when I started working there in 1998, was that I had no experience on a Mac. I only knew that Macs were easy to use. I was hired at a job conference in San Diego and, after I accepted the verbal offer, I confessed to my soon-to-be boss that I had never worked on a Mac. "Don't worry about that - we use Windows NT," he said.

Seriously? I was about to go work for Apple on Windows?!?

I shouldn't have been hired due to my lack of web/Java experience, but it was the Dot-com boom and Apple was hiring like Gang Busters. I didn't even know what my job position would be when I signed up, so I was wondering what I'd be working on that required me to relocate to the Washington DC area. All I knew is that it was an unbelievably exciting time to be working in high tech. It seemed like everyone was getting rich!

It turns out that I was hired to be a WebObjects consulting engineer for the consulting division in the Apple Federal office (this office has since relocated to a different part of town).

Why Windows?
My first week at Apple, I was given a Toshiba Tecra and my job was to "build it from scratch" which meant installing Windows NT, WebObjects, and an Oracle database server. This was not an easy task - I had to hunt down lots of little things like network card, video, and printer drivers and then figure out which interrupts they needed to be configured to (if I recall correctly). But, before I did that, I had to use a floppy disk to install PKZIP so I could unzip files. Not fun at all. I felt so dumb that I'd be sure to ask different coworkers for help so that no one person could see my total ignorance.

The reason we worked on Windows NT was because, the previous year, Apple had purchased NeXT for both their operating system (OpenStep) and web app server (WebObjects). At the time of the purchase OpenStep and WebObjects ran on Windows NT.

Several years earlier, NeXT engineers had ported NeXT OS from their proprietary hardware over to Windows - a technique that would benefit Apple in 2005 when they seamlessly ported Mac OS X from the PowerPC to the Intel processor. Their secret was to develop against layers of abstraction rather than hard coding directly to low level hardware. For example, to get WebObjects, which was designed to run on UNIX with a Mach kernel, to run on Windows NT, they ported over a subset of the Mach kernel to sit between WebObjects and Windows. It ran as a Windows daemon to help WebObjects work with Windows.

Apple's Future
Apple's future was very precarious in the late 1990s. During my first few years, I kept wondering why I was purchasing Apple stock and not Microsoft's. The iPod, iPhone, and iPad weren't even a gleam in Steve Jobs's eye back then as he attempted to design four desktop and laptop computers for the consumer and professional markets.

While a lot of my coworkers had jumped ship in the late 1990s I hesitantly decided to stick it out. I had seen a glimpse, through WebObjects and OpenStep, of what the future could be. Needless to say, it turned out much better than I could have imagined.

Looking back, it seems that Apple was predestined to become what it has - but, living through it was a daily ritual of self-doubt.

All Servers Die So Take Care of Your Customers

Adjix, Epics3, et al, server farm with two dead (vertical) servers.
Serving up nearly 100K unique visitors daily on 512 Kbps.

Adjix

Born: August 19, 2008

Died: December 31, 2011

I wish that I could say Adjix died peacefully, in its sleep, but it was a sudden, unexpected death as it went down fighting.

RIP Adjix.

New Year's Eve 2012
I received a call from a couple of key users, on New Year's Eve afternoon, that my beloved Adjix was having problems. After the calls, I discovered that the Adjix database had unexpectedly stopped and needed to be restarted. We knew that the Adjix database didn't have much life left after the San Diego power outage of September 8, 2011 knocked out some servers in the cluster - but I wanted to keep Adjix running until the last possible moment.

After identifying the problem, during the afternoon of New Year's Eve, I restarted the database server and I could tell that it would take several hours to rebuild the indices and check the database integrity due to its large size. After all, you do want your database to be ACID compliant.

Then, ominously, just before midnight EST, the physical database server (hardware) stopped responding to any and all commands.

Today, six months later, I gained physical access to that server - the last one in that dying cluster - and this is the final entry from the logs (-0800 is PST).

2011-12-31 21:52:43 -0800: Rolling forward master [80.1% complete] 17154000 SQL transactions processed

Sure enough, the logs confirmed that, with less than eight minutes left before the New Year, the hardware stopped responding.

Future Proof
Although I never charged anyone to use the Adjix APIs, I still felt that everything I delivered should be future proof.

In theory, we can see that Adjix links redundantly work as expected:

http://adjix.com/ai26

But, I have to admit that I'm amazed to see, six months after Adjix died, that others' Adjix links are still alive and kicking:

http://twitter.com/search/adjix

(Note: ad.vu is the ultra-short version of adjix.com.)

Not only did Adjix automatically store every single shortened URL in its own adjix.com Amazon S3 bucket, but Adjix also gave every user the opportunity to store the Adjix shortened link in their own S3 bucket.

Applying Lessons Learned
In early 2010, I launched a photo sharing service, Epics3, which allowed users to share photos on Twitter and Facebook as well as store any photo they uploaded to their own Amazon S3 bucket. Just like Adjix, this was a simple solution to implement resulting in two separately owned and operated servers redundantly serving up the same content.

As an example, the first of the following links is dynamically generated from the Epics3 server and the second one is served statically, directly, from an Amazon S3 bucket.

http://pics.joemoreno.com/2gi3

That, my friend, is future proofing. You'd think, after four years, that it would be more popular.

On New Year's Eve 2012, Adjix died. Long live Adjix (and all other customer data).

Mathematics Debate

It's been about three and a half years since I blogged a math problem, so I'm way overdue.

Today, at lunch, my wife, who's a middle school math teacher, and I had a discussion about numbers. I mentioned to her something that Dave Winer had brought to my attention, this past April:

Does 0.99999 = 1?

"No - of course they're not the same," is most everyone's initial thought. Those two numbers are close... but not equal.

But, try this proof:

Q: Does 0.33333 = 1/3?
A: Yes!

Ok, next question...

Q: Does 0.66666 = 2/3?
A: Yes, again!

So, now for the punchline:

Q: Does 1/3 + 2/3 = 1?
A: Well, yes. But...

The beauty of mathematics is even though it's an abstract field, it's a pure discipline unlike, say, computer science. The laws of mathematics are constant throughout the entire universe.

So, does does 0.99999 = 1? Unfortunately, there's no simple answer.

From Steve's Lips to an Engineer's Ears

At Apple, the engineers are the talent and "This doesn't suck too bad," was high praise when I worked there.

Initially, when I began working at Apple, I thought that the company was primarily about design on many levels - industrial design, UI design, system architecture design, software design, etc.

I was wrong.

Design was just a means to an end - and that end was to provide the best possible user experience (BPUX) - the UX of highest quality. It meant, for the time being, there was no better way to design the UI for a better UX.

Think of BPUX as the nirvana of customer service. This is why consumers love to do business with Apple instead of, say, the phone company.

Holistic BPUX includes everything, from making the purchase and opening the box to using the product and calling on tech support. Apple's focus is not about squeezing every possible dollar from the customer, in the short term, at the expense of long term profitability. Sure, Apple cares about making money, but that is not what they design for.

Apple clearly understands that, as a consumer electronics company, they are not selling technology products. Rather Apple is selling a user experience and, to the user, the interface is the product. Steve Jobs inherently understood this decades ago and he said it best at WWDC in 1997:

You got to start with the customer (user) experience and work backwards to the technology. You can't start with with the technology and try to figure out where you're going to try to sell it. And I've made this mistake probably more than anyone else in this room and I've got the scar tissue to prove it.

But, when evaluating user experience, you have to look at it holistically with "rigid flexibility."

The first generations of iPods used an Apple developed connectivity technology, called FireWire, instead of USB 1.0 because FireWire was about 30 times faster. When USB 2.0 became the de facto standard on PCs - with speeds comparable to FireWire - Apple transitioned their iPods away from their own FireWire technology to the USB industry standard.

Steve's Requirements

Steve obviously understood design, but how, exactly, does a requirement from Steve Jobs travel to an engineer? It's a rather simple process that involves neither formal documentation nor written approvals.

In 2006, we were redesigning the Apple Online Store. My boss's boss was in a meeting with Steve when Steve directed that, once customers were at the online store, they should be able to find any Apple product within three clicks. The three click requirement wasn't invented by Steve, but he knew that time and clicks were two basic quantitive metrics for measuring BPUX. (As a side note, in this same meeting, Steve mentioned that he disliked contextual menus - right click pop-up menus - because they had become a dumping ground for unimaginative and lazy UI designers.)

So, we, the engineers, were simply told that each Apple product had to be found within three clicks. That might sound simple until you run a search for, say, headphones which returns about 200 different products. You can't paginate the results into batches of 10 or 25 because that would result in more than three clicks. On the other hand, you can't display all 100 or 200 products at once because that would overload the servers. And, you most certainly can't tell Steve, "It can't be done."

What do you do?

Infinite Scrolling

A coworker implemented a solution, called "infinite scrolling," for which he won an internal Apple innovation award. Today, in 2012, we take infinite scrolling for granted, but, back in 2006, AJAX was a fairly new web development technique with its key technology specification having only been drafted that year.

Simply put, infinite scrolling is the experience we have when scrolling down our Facebook or Twitter web page (example). As we scroll down to the bottom of the page, more of the web page is dynamically loaded which satisfied Steve's three click requirement, in both the letter and spirit of his intent, since a scroll isn't a click.

At the Apple Online Store, we implemented infinite scrolling by caching the product search results and then we'd trickle the results to the user for rendering in their web browser as they scrolled down the web page.

True Empowerment
Infinite scrolling isn't really a big deal and today's Apple Online Store implementation is different than the one implemented in 2006 --- the key thing to note about Apple is that neither Steve, nor any other manager, tried to solution the implementation. Instead, they pushed the problem down to the lowest level possible - to the talent: the engineers.

New Sandwich Shop, New Cash Register

This morning, my wife and I had breakfast at a new restaurant that just opened this morning. It's only a block away from our home and we'd pass by it frequently as they prepared for today's opening. We were the only customers in the restaurant since it was late in the morning when we went. But, we were told that business was moving along very nicely earlier in the morning.

When we bellied up to the counter to pay our bill, we couldn't help but notice that their cash register was an iPad connected to a cash till and credit card printer. They were excited to show off their high tech point-of-sale (POS) system which got me wondering if it was actually cheaper - in terms of total cost of ownership - than a traditional cash register.

Realistically, an alternative solution, like Square, seems practical when processing only credit cards on the go. A fixed POS system needs to securely store cash as well as process credit cards and it should be somewhat rugged.

A basic cash register, with a till, starts at about $200. Then add one or two hundred more for a printer and credit card swiper. On the other hand, a new iPad 2 (previous generation) starts at $399 plus the cash register app, printer and till will bump up the cost. A key benefit of an iPad POS is that it has more features and can collect more data than a basic cash register.

I still can't help but ponder how this will hold up. Even though most cash registers at retail chains are actually stripped down PCs which are networked into servers - I wonder if the iPad POS will work as expected or if it'll end up being more like a portable GPS system on your car's dashboard vice a navigation system built into your car. In other words, it's the difference between a dedicated tool for a specific job or, perhaps, a solution looking for a problem.

Of course, if the iPad POS was wirelessly connected to the kitchen - which I don't think it was - that would be a no brainer in terms of efficiency. But, regardless, the nicest benefit of a mom-and-pop shop using an iPad cash register is that they can take it home at the end of the day and use it for more proper iPad duties.

Pink Slipped on Pink Shirt Friday

Today was Pink Shirt Friday at work. I've known my boss/buddy for more than a dozen years and, about once a quarter, we coordinate wearing pink shirts to work on a given Friday. This week, we extended the invitation to all of the directors that work in the Product department.

The irony about today's Pink Shirt Friday is that pink slips were handed out and the entire Product department was dissolved. The layoffs - including me - were due to the fact that the Product department's original tasking had become obsolete. It's a shame because it seems like it was only yesterday when I wrote this piece.

In all actuality, it's very true that our departments's original tasking had become obsolete. Although I worked at Wyndham for only eight months, I noticed that the company had many processes and layers that could be streamlined. Slimming it down, by eliminating the Product department, will probably be a good thing. After all, lean processes are how we did it when I worked at Apple.

What Does the Product Department Do?
Generally, the Product department manages the product roadmap and it sits between Marketing and Engineering. But, it seems that our Marketing, Product, and IT departments ended up working so closely together that Marketing could deal directly with the engineers in many cases - especially once a project's requirements had been documented.

This video best describes the perception of the Product department.

Practically speaking, you can think of one duty of the Product department as a waiter/waitress who's filling a role between a restaurant customer and the kitchen.

It's About The People

In my short time at the company, I was part of a few key product launches that received a fair amount of attention including the Room Key joint venture and the mobile websites and apps launches.

The disappointing part is that I relocated from San Diego to the Greater NYC area for this job and it barley lasted eight months. But, I've been pleasantly surprised at just how much I enjoyed working with my fellow managers, directors, and VPs who are very thorough, smart, and hard working people.

It's not often that you work with a team of people who stay on task, no matter what, until the job is done; especially during the late night conference calls, like this one, where you end up hearing the birds chirping as you watch the sunrise before the call is over.

In the end, with the Product department no longer needed, all of the directors were given notice, today. And, although I'd seen many rounds of layoffs at Apple, this was a first for me. Fortunately, the company seems to have gone above and beyond and treated us well.

Oh, there's one other thing that I will certainly miss: Free Ice Cream Fridays.