Amazon Fulfillment Warehouse Tour

This afternoon, I took a one-mile walking tour of an Amazon fulfillment warehouse that’s mostly run by robots, here in San Diego. This facility spans 3,330,000² feet, employs 4,500 people, and operates 23/7 in three shifts with one hour of daily downtime.

I highly recommend taking one of these tours which are free and open to the public in nearly 20 states: Amazon Tours.

The most fascinating part was watching the robots disappear into the darkness as they moved shelves of Amazon products since they don’t need light to see where they’re going.

Sorry, no photos allowed.

A Love for Amazon

Amazon has cracked the code on keeping customers.

A coworker used to work at Amazon for four years. He told me that Jeff Bezos believed there was nothing more expensive than losing a customer. Hence, the reason why returns are so easy with Amazon, even though Amazon may eat those costs in the short term. Last night was no exception.

AWS Hack

In 2014, my Amazon Web Services account was hacked. Ten days into the month, I noticed that my bill was already a thousand times larger than my typical bill. Amazon recognized the hack and gave me amnesty for the entire bill. 

Spreading the Profits

Amazon had their first profitable quarter in the fourth quarter of 2001 when they earned $5 million in profit on revenues of over $1 billion.¹ Bezos wanted to share this profit with customers.

Tech companies typically don't pay dividends to investors. Even today, Amazon doesn't pay a dividend. But sharing profits with customers is very unusual. In this case, the profit shared with each customer amounted to less than $1/customer. As insignificant as that sounds, Bezos found a way to make it meaningful.

The price of a USPS First Class postage stamp was increasing in January 2002 from 34¢ to 37¢. Back then, "snail mail" was much more popular than today; not everyone had an e-mail account. Each price increase in postage was a challenge before April 2007 when the post office introduced the Forever Stamp. Before the Forever Stamp, a person had to buy 1¢ stamps to add to postage. This was a hassle and many people, including myself, would, for example, simply put two 34¢ stamps on a letter instead of a 34¢ stamp plus three 1¢ stamps. It seems like a waste of money but our personal time, to purchase additional 1¢ stamps, is worth something to each of us.

Bezos's solution was elegant. He recognized this inconvenience of needing to buy 1¢ stamps, so he had Amazon send a bunch of 1¢ stamps to their customers. I remember thinking how brilliant this was when I received them.

Last Night

Two days ago, I received my shingles #2 and pneumonia vaccines. The shingles second vaccine is the same as the first one, which I received on its own, without much discomfort. But, yesterday, I felt sick and I feared Covid because, on New Year's Day, I had breakfast with someone who tested positive for Covid, the next day – the same day I received my vaccines.

I took my temperature yesterday morning and it was fine – no fever. In the evening, when I took it, my thermometer was displaying a fever of 100.7°F. But the display was blinking which meant the battery was too low for a reliable reading. I looked on Amazon for a replacement battery. But there was no practical option that would arrive in time plus I couldn't buy a single battery which would be a waste.

Instead, I ordered a new thermometer around 6 PM and I paid an extra $3 so it would be delivered the next morning between 4AM and 8 AM. However, to my surprise, Amazon delivered it about an hour later. I took my temperature with the new thermometer and all was well – no fever. 

It's unusual for Amazon to move up a delivery by one day. I'm speculating that they know, if someone is paying extra for a faster delivery, for a medical item, that it's urgently needed. So, it seems they were able to expedite my delivery to give me peace of mind. I greatly appreciate that and it pays to live very close to downtown San Diego.

Part 2: Rx at Amazon Pharmacy vs Kaiser Permanente

Part 1: Rx at Amazon Pharmacy vs Kaiser Permanente

I tried Amazon Pharmacy for a Kaiser Permanente prescription for a second time, this week, and there was a significant savings. 

Earlier this summer, I compared Amazon Pharmacy to Kaiser Permanente for prescription fulfillment. Back then, the prices were exactly the same.

I tried it again, earlier this week, and Amazon Pharmacy was 60% cheaper than Kaiser Permanente since I was an Amazon Prime member.

Your mileage may vary, but it’s worth a gander. 

Part 1: Rx at Amazon Pharmacy vs Kaiser Permanente

Part 2: Rx at Amazon Pharmacy vs Kaiser Permanenet

I tried out Amazon Pharmacy for the first time.

TLDR; The prices were exactly the same as my healthcare provider.

Details

I ordered a prescription for a keto rash cream from my HMO (Kaiser Permanente) and received a little sticker shock when a 60g tube weighed in at nearly $60. In the past, this toothpaste-sized tube would cost me less than half that. So, I cancelled the prescription after seeing much cheaper options on the Amazon Pharmacy website.

I signed up for an Amazon Pharmacy account which also gave me the option to use a PIN for additional security – nice touch. I found the item I wanted (Fluocinonide), put it in my cart, and went to check out. Amazon Pharmacy asked me for the contact info for my dermatologist, which I provided, and they told me it would take a few business days to process my order.

When I hadn't heard anything a week later, I called Amazon Pharmacy's customer service phone number. It sounded like I was speaking to an American woman working from home due to occasional dog barking in the distant background. She dug into my account and said that Amazon Pharmacy attempted to contact my dermatologist several times via fax and phone with no luck. The phone hold times were too long, she told me.

I e-mailed my dermatologist at Kaiser Permanente with the Amazon Pharmacy phone number. A few hours later I received an e-mail response from Kaiser Permanente. They told me they called Amazon Pharmacy, spoke to Robin, and that my prescription was being processed.

The next day, I received an e-mail from Amazon Pharmacy with a price quote that exactly matched the price Kaiser Permanente quoted to me. I called Amazon Pharmacy to ask if I could order a smaller tube, but they said I'd have to contact my doctor to change the prescription. At this point, I realized that my copay with Amazon wouldn't be better than my copay with Kaiser Permanente and gave up on the idea of using Amazon Pharmacy. Instead, I e-mailed my dermatologist and asked for a prescription to be filled for half the amount, which was not a problem. The item shipped, with a USPS tracking number, at no additional charge from Kaiser Permanente.

While is was my first, and only, experience with Amazon Pharmacy I don't see how it would be a benefit compared to Kaiser Permanente which already provides free shipping along with a fully integrated system. Perhaps I'll try again, in the future. Don't fire Carolina!

macOS on AWS. Finally!

One of my "server farms" (more like a server garden)
with 4 Macs and 1 GSM modem

Finally, after nearly 15 years, macOS comes to AWS. I used to maintain two server farms totaling 11 Macs, mostly Mac minis, running OS X Server with 10 static IP addresses. Configuring and replacing Mac servers was my least favorite and most expensive task.

Amazon's AWS VP for EC2 shared a fun video about how macOS on AWS came to be.

How to Inventory Prime Now

More and more often, I find myself using Amazon Prime Now for two hour food and restaurant deliveries. As a former supply officer in the Marines, I quickly learned the importance of receipting for materiel as soon as it's received, before signing off on the delivery.

Pre-inventory

Receipting for a Prime Now delivery should also be done as soon as it arrives. To avoid any confusion or mistakes I simply unload everything in one spot, typcially on my kitchen counter (see my pre-inventory photo). Be sure to thoroughly check for any small items, especially in the bags with the cold packs.

Post-inventory

You can print out your order if it's a big delivery, but that shouldn't be necessary. I simply open my Prime Now app on my phone and start at the top of the list. As I scroll down, through each item, I pick it up and move it from my pre-inventory location to another counter (see my post-inventory photo). I've seen people simply take a quick look in their delivery bags and then put everything away, causing problems later when they're unsure if they received an item they ordered. Also, be sure to double check the quantities. 

Once you've gone through the entire list, you'll know within a two minutes if everything was delivered. If something's missing, I'll go through all the bags to triple check before calling customer service. 

If you have extra items, simply keep them. I've contacted Amazon when I was delivered a few extra items I didn't order and their response was "I'm truly sorry that you have received an extra bag of goodies. You are welcome to keep, dispose of or donate the items in that extra bag." Their loss is your gain. Not a bad deal.

If you're missing anything – or, if you've received the wrong order – then contact Amazon. They will probably give you a refund and/or a credit and then, if time allows, send out the replacement items.

In my experience, all of the drivers for Prime Now have been Amazon Flex Drivers which is like Uber/Lyft, but instead of moving people, the driver is moving goods. However, the Flex Drivers do not pick and pack the items in the warehouse; that's done by well-trained Amazon employees and highly obedient robots.

To speed up my delivery, I will use the app or website to track the Flex Driver on the way to my place and meet them between my front door and the street. Your deliverer will appreciate that, especially if you live in a gated complex. 

Like Uber/Lyft, no money is exchanged on the spot. It's all done ahead of time, through the app; even the tip.

Yes, Amazon's taking over the planet (and beyond). It's the second most valuable company in the world, if my calculations correct:

High Tech Market Capitalization

• Apple:     $939B
• Amazon:    $872B
• Google:    $824B
• Microsoft: $801B
• Facebook:  $599B
• Netflix:   $180B

PS – No, you don't have to queue up your food to inventory it. I don't. I only did it for these photos.

Prime Now, Amazon Flex (One way to spend money. One way to make money.)

Signed, sealed, and delivered.

A friend told me that she recently started driving for Amazon Flex which is like Uber for Amazon package delivery. As a contract driver, she simply drives to her local Amazon warehouse and loads up her car. She scans each package, using the Amazon Flex app, as she places it in her car and the app determines the optimal driving route. Drivers have three hours to finish their delivers and return with any undelivered items.

As I looked into Amazon Flex, I noticed that Amazon Prime Now delivers groceries, similar to Amazon Fresh. I gave it a try since Prime Now offered free delivery for orders over $20 (plus an additional $5 tip).

I started putting items into my shopping cart, yesterday afternoon. When I went to resume shopping, this afternoon, I noticed that a couple items were no longer available which was clearly displayed. After adding a few more items to my cart, I checked out at 1:30 PM with a 2 PM – 4 PM free delivery window. I had the option, for $7.99, for delivery within the hour.

Ding Dong

Cooler bag for dairy.

About 75 minutes after I placed my order, which was 15 minutes before arriving, I received a text message that Ryan was on his way, with a link to track him that updated in near real-time. I went out to meet him when I saw that he had arrived. As he walked toward me I said, "You're Ryan, from Amazon, right?"

"Yes... this was an easy delivery," said Ryan.

"Do you drive for Amazon Flex?" I asked, catching him off guard.

"Um, uh, yeah... do you know me, personally?"

He was a bit baffled as I explained that I did not know him but I understood how Amazon Flex worked. Best part is that my delivery was exactly what I ordered, plus the dairy was in a cooler bag with ice.

Amazon Prime Lessons

$20K for soap?

Last month, I signed up for Amazon Prime. Yes, I am very late to the show. I had the expectation that I could order most anything and it should ship, for free, and I'd receive it within two days. It turns out it's not that simple.

For starters, I did a quick search for Dial soap and added it to my cart. That's when I discovered, if I'm not careful, that I could end up paying almost $20,000 for simple bar soap. I also had to look, carefully, to distinguish between Amazon Prime, Amazon Prime Pantry, and Add-on Items.

Fulfillment by Amazon

Two weeks ago, a friend attended a seminar explaining how savvy entrepreneurs buy close out deals from places other than Amazon and then have their items shipped to Amazon for storage and fulfillment. At a glance, these items may look like they're being sold by Amazon, but they're actually only stored and shipped by Amazon. A third party is the actual seller. This can be a great service for an entrepreneur who has products manufactured in China and shipped to the US. Rather than the entrepreneur storing their inventory in their own garage, or leasing a warehouse, they can have the products shipped from China to Amazon for storage and fulfillment. The gotcha, for the customer, is when an entrepreneur sells an item on Amazon that Amazon already sells at a cheaper price. As the Syms Corporation used to advertise, which applies doubly so to Amazon, "An educated consumer is our best customer."

Prime Pantry and Add-on

Add-on items are another customer gotcha. Today, I placed an order for a few staple items that will be fulfilled tomorrow. A couple hours later I remembered that I needed to order one more thing (paper towels). I found some paper towels to order, but it was an Amazon Prime Pantry item which required an additional $5.99 in shipping, even if it's an Amazon Prime exclusive. So I passed on that. Next, I found some more paper towels, that were part of Amazon Prime, and not Prime Pantry. But, when I went to order it, I discovered it was an Add-on item meaning that it couldn't be shipped as a single item. I was hoping to find a way to add it to my pending order that was shipping tomorrow. But alas, I didn't find a way. In the past, I've cancelled a pending order, re-added the items I had just ordered and then included the Add-on item. Today, I decided to pass. Perhaps I'll remember to order it next time. I love that Amazon has enlisted the USPS to make delivers on Sundays but I wish there was a simpler way to modify my orders. Perhaps there is, but I don't know about it, yet???

Must Have Better TV

I spent today dog sitting at a buddy's house while he took his family to Universal Studios. He has AT&T U-verse with many, many channels. I haven't channel surfed TV in over a year.

In 2002 I hooked up a new TV to cable and let it automatically block out any channels that weren't broadcasting between channels 2 and 72. A decade later, more than ten of those channels were no longer broadcasting. The cable company kept moving channels above 72 to encourage their customers to forgo analog cable for digital TV.

On weekend mornings, more than a dozen of the remaining channels were broadcasting infomercials for weight loss and get rich quick products.

Why was I paying to see infomercials?

Unbundle Cable TV

Without a doubt, consumers would love to unbundle TV channels much like music albums. I'd really like to pay for, say, CNBC, CNN, KUSI, Spike, etc, without having to pay for QVC or channels broadcasting in languages I don't understand.

Ari Emanuel is a Hollywood talent agent who is is strongly against unbundling cable TV channels. In other words he doesn't want to give consumers the ability to pick, à la carte, which channels they want to pay for. Ari Emanuel spoke about this at D10:

Overall, from subscriptions to advertising, the market is about $100 billion now, while moving to à la carte pricing might only get $40 billion, which would be a major inhibitor [of developing premium content].

Basically, Ari Emanuel reasons that if the market shrinks so will the quality of the content like Hollywood movies and TV shows. Maybe. Maybe not. But Ari is one of those guys who you might say is a tad arrogant (see the end of my digression video below).

Companies like Netflix, Amazon, and Apple have been trying to unbundle channels but it has been a uphill battle. Unbundled content like House of Cards or Comedians in Cars Getting Coffee is rare. Eddy Cue once told us at a meeting that getting content from the music and TV industries onto iTunes was "simple" compared to the film industry. He said that the film industry was the most business savvy of them all since movies were sold to theaters, TV, cable, DVD, airlines, military, etc, for a very long time.

Perhaps, one day, we'll be able to pick the channels we want to pay for.

Digression...

Ari Emanuel is the brother of President Obama's former chief of staff, Rahm Emanuel, who is now the mayor of Chicago. A fictionalization of Ari Emanuel is played by the character Ari Gold (Jeremy Piven), in the HBO series Entourage. It's funny to watch Ari Gold bash others, like his assistant Lloyd, on the TV show. But, it's a bit uncomfortable to watch the real life Ari Emanuel tell people to shut up and sit down like he does in this Q&A panel.

$5,000 Security Breach, Part 1

$17,000 AWS bill in the making.

4/15/2014 Update: This story was picked up by ReadWrite.

Part 2 Here

The problem with the Heartbleed bug is you never know when and where you'll get hit. Actually, this is true for all security breaches.

Yesterday, I received an e-mail from Amazon asking me to update my credit card info for one of my personal Amazon Web Services (AWS) accounts. I logged in and saw that my running total for April was over $5,300. My typical monthly bill is less than $6.00 which is about 1,000 times less.

At first I thought it was a mistake. I hadn't fired up any EC2 instances this month and my account had no EC2 instances running in my region. I filled out a billing inquiry request form and selected the "call me now" option. Within a minute my phone rang and I was speaking to an AWS customer service rep.

I explained that I've been an AWS customer since 2007 and I've never seen a billing issue like this.

He said, "We've been seeing more and more of this. Check your spot EC2 instances in other regions and you'll see high end instances running."

Sure enough, he was right. In Tokyo, São Paulo, Sydney, and Singapore I had expensive server instances running.

"Your AWS credentials have been compromised," he said.

How did they get compromised? When did it happen? Was my development machine hacked? It couldn't be my Time Capsule since that's encrypted. Were one of my physical servers hacked? Did I have a backup, sitting on a server, somewhere, that was hacked? Am I about to get stuck with a $5,360 bill?

"'They' spin up spot instances which isn't subject to Billing Alerts. You'll need to cancel those spot instances, revoke your AWS credentials, and change your account password," he said.

"When did this happen?" I asked.

"Let me look," he replied.

My mind was still racing as I tried to figure out the source of the breach.

"These instances were spun up on April 2," he said.

Very smart; launch the attack early in the month so the victim won't know anything's wrong until they get next month's bill.

"Is this related to Heartbleed?" I asked. It had to be.

"No, it's just a case of your AWS credentials getting compromised," he answered.

He walked me through the steps to secure my account.

"Can you see what 'they' were doing with these spot instances?" I asked.

"No, we can't see inside the instances. But, they're usually mining for Bitcoin," he answered.

Ahh, now that makes sense. Spend $5,000 of someone else's money to mine, say, $1,000 of Bitcoin for yourself. Can't follow that money trail.

"I'm going to send you a questionnaire. Please fill it out describing what happened and, due to the large amount involved, I'll need a manager to review it. But you won't have to pay for what you're not responsible for," he said.

I let out a sigh of relief.

I was still dripping with sweat since I'd just returned from a run when I saw the initial e-mail from Amazon. While I was in the shower it hit me. I know how my AWS credentials were compromised. But I'll need to do a little more research first.

4/15/2014 Update: How did this happened? See Part 2 of this story to find out.

Author: Joe Moreno

1,000 songs in your pocket.

CNBC tweet on Amazon's Fire TV.

BLUF: Lead with the benefits before the features.

When Apple announced the first iPod they didn't market it as a 6.5 ounce MP3 player with a 5 GB hard drive. Instead, the copy simply read, "1,000 songs in your pocket."

When I saw this random tweet about Amazon's Fire TV I stopped and asked myself, "How does that compare to Apple TV?" Then I realized it didn't matter.

What's redline on your car engine? How fast is your smartphone's CPU? Is the CPU dual-core or quad-core? Does your car have four, six, or eight cylinders? How many cubic inches or liters does it displace?

Detailed tech specs matter to an engineer but they don't have the same marketing impact on consumers. This CNBC sponsored tweet, taken slightly out of context to illustrate my point, should probably be tailored for the latter in a world where color and style matter in consumer electronics. Tech specs were once important, but, nowadays, they're not a big deal for consumers. It's all about best possible customer experience.

Amazon CloudFront

Amazon's CloudFront brought CDN to the casual coder about five years ago. When Amazon announced CloudFront it had much of the same basic functionality as Akamai. But, the problem with CloudFront was many features were only accessible through APIs, not the AWS dashboard (management console). One of the key features missing from the dashboard was the ability to invalidate an object in the CDN. For me, this was a showstopper.

When caching an object (file) on an edge server, the CloudFront default setting stores it for 24 hours. To clear out a file before it expires requires an API call to invalidate it and fetch a new copy from the source. This setting was not originally available to the casual CloudFront user via the AWS dashboard. The API functionality was there, but, unlike Akamai's dashboard, it required a coder to put it in place.

I recently began using CloudFront, again. I'm happy to see the AWS dashboard can now invalidate objects with a simple click. Or, even better, the invalidation of a subset of objects is possible by specifying paths or patterns.

AWS just keeps getting better while dropping their prices. What could be better?

Amazon Product Management

Click to enlarge

Amazon has a very distinctive technique for bringing a new product to market. Instead of throwing stuff against the wall to see what sticks, they begin new product development with a notional press release followed by a first cut at the product's FAQ. Keep in mind that the press release will never be published in its original form – rather, it's a guide to keep the business, product, and engineering teams on the same sheet of music. I think of the press release as the high level, 30,000', strategic view of the product and the FAQ as the operational, 10,000', view.

Once the press release and FAQ are created the marketing team can layout their vision which is then passed to the product department to document the functional specifications, wire diagrams, and technical requirements for the engineering team to develop and the QA team to test against.

At the end of the day, though, it's a company's corporate culture that drives the success of their products. At Amazon, their corporate culture focuses on what the customer wants. This "the customer is always right" mentality may seem obvious, but it really does depend on each company's DNA. When I worked at Apple, our priority was to provide the best possible user experience (UX) and customer experience (CX).

Urban Life Conveniences

You can't miss the Amazon Locker by the front door of 7-11.

Today I had two firsts. I commuted on the San Diego public transit bus system and, this evening, I received a package from Amazon using their Locker delivery service.

Everyone's familiar with public buses, even people who never use them. But the Amazon Locker delivery service just started last year. It's perfect for people who live and work in big cities and don't want packages left sitting at their front door.

Instead of having your package shipped by Amazon to an address, they deliver it to a bank of lockers that they own, spread throughout the city. The shipping charges seem to be no different than if the package was shipped to your front door.

7-11 Amazon Locker at bus and trolly stop.

This evening, I received a confirmation e-mail that my package was ready for pick up. So, I hopped on the bus since the nearest bus stop is just a few hundred feet from my place. The last stop on my bus route was right outside – less than 100 feet – from the 7-11 where Amazon has a Locker delivery location. A perfect synergy since they're open 24/7.

Enter your code that pops open your locker.

Picking up my package was so quick I ended up reboarding the same bus that dropped me off because the driver had a five minute break. But, since my bus fare card is unlimited there was a "read error" when I tried to swipe it which the driver dismissed when he realized that I was the same person boarding the same bus.




Both the bus and Amazon Locker services were great customer experiences  – much better than I expected. Door-to-door was well under an hour for this two mile roundtrip  Best part is that the bus ride was effectively free and I didn't have to hunt for parking downtown.