What Hardware and OS are Inside Apple's Data Centers?

Here are a few things to consider about Apple's infrastructure.

Apple used to make the Xserve.

It was a beautifully designed piece of hardware, inside and out. Apple stopped shipping it about three and a half years ago.

Apple maintains it's own data centers.

What's inside these massive data centers? "Stuff," said Steve in this short video clip. Obviously, these data centers are packed full of servers.

So, what hardware & operating system are powering Apple's data centers?

The Apple data centers are most certainly not running Xserve hardware and they're not running OS X Server. I'd speculate they're running HP or IBM hardware with some flavor of Unix, perhaps even Linux.

Anyone else care to take a guess?

Author: Joe Moreno

At the Cafe: To laptop, or not to laptop

Banned at August First bakery: Laptops and tablets.

On the shores of Lake Champlain is a bakery cafe that's banned screens. This Burlington, VT cafe opened four years ago with free WiFi. Quickly, the owners noticed that patrons were camping out. The table space they took up started to affect their bottom line.

On the other side of the spectrum is San Diego's (and my) favorite coffee shop, Old Cal Coffee, in San Marcos, CA. They invite customers to spend all day. Frequently, I, along with others, have spent more than four hours there and I have yet to see a patron shooed away. I once saw a regular, who seemed to be living out of his car, sitting on the patio one holiday when the cafe was closed just to use the WiFi.

Right or Wrong?

The answer to the question, "Is this right or wrong?" is simple: It depends.

Sometimes I need to be offline when I want to be online and vice versa.

Keep in mind that these establishments are private owned businesses open to the public. The owners make the rules and we, the customers, are their guests. And, in the end, we vote with our money. In the case of the August First Bakery in Burlington, are customers going there for food or free WiFi? If it's the food, then the business should survive by banning screens.

Author: Joe Moreno

$5,000 Security Breach, Part 2

Every so often I write a blog post that immediately receives many thousands of views. Part 1 of this story fell into that category.

Where I last left off, on Thursday, I was in the shower when I had an epiphany. I had figured out how my Amazon Web Services credentials were compromised. At least I suspected, but I was running late, after my call with Amazon, as I got ready for the Spring Fling tech event. I didn't have time to comb through my public repository account so I deleted my entire GitHub account. I had only used it once, years ago, when I checked in an open source WebObjects project I had developed.

Jodi Mardesich interviewed me for the details and gave my story a great write up at ReadWrite.

Coda update: Amazon has confirmed that they'll grant me a one time exception for my faux pas.

Author: Joe Moreno

$5,000 Security Breach

$17,000 AWS bill in the making.

4/15/2014 Update: This story was picked up by ReadWrite.

The problem with the Heartbleed bug is you never know when and where you'll get hit. Actually, this is true for all security breaches.

Yesterday, I received an e-mail from Amazon asking me to update my credit card info for one of my personal Amazon Web Services (AWS) accounts. I logged in and saw that my running total for April was over $5,300. My typical monthly bill is less than $6.00 which is about 1,000 times less.

At first I thought it was a mistake. I hadn't fired up any EC2 instances this month and my account had no EC2 instances running in my region. I filled out a billing inquiry request form and selected the "call me now" option. Within a minute my phone rang and I was speaking to an AWS customer service rep.

I explained that I've been an AWS customer since 2007 and I've never seen a billing issue like this.

He said, "We've been seeing more and more of this. Check your spot EC2 instances in other regions and you'll see high end instances running."

Sure enough, he was right. In Tokyo, São Paulo, Sydney, and Singapore I had expensive server instances running.

"Your AWS credentials have been compromised," he said.

How did they get compromised? When did it happen? Was my development machine hacked? It couldn't be my Time Capsule since that's encrypted. Were one of my physical servers hacked? Did I have a backup, sitting on a server, somewhere, that was hacked? Am I about to get stuck with a $5,360 bill?

"'They' spin up spot instances which isn't subject to Billing Alerts. You'll need to cancel those spot instances, revoke your AWS credentials, and change your account password," he said.

"When did this happen?" I asked.

"Let me look," he replied.

My mind was still racing as I tried to figure out the source of the breach.

"These instances were spun up on April 2," he said.

Very smart; launch the attack early in the month so the victim won't know anything's wrong until they get next month's bill.

"Is this related to Heartbleed?" I asked. It had to be.

"No, it's just a case of your AWS credentials getting compromised," he answered.

He walked me through the steps to secure my account.

"Can you see what 'they' were doing with these spot instances?" I asked.

"No, we can't see inside the instances. But, they're usually mining for Bitcoin," he answered.

Ahh, now that makes sense. Spend $5,000 of someone else's money to mine, say, $1,000 of Bitcoin for yourself. Can't follow that money trail.

"I'm going to send you a questionnaire. Please fill it out describing what happened and, due to the large amount involved, I'll need a manager to review it. But you won't have to pay for what you're not responsible for," he said.

I let out a sigh of relief.

I was still dripping with sweat since I'd just returned from a run when I saw the initial e-mail from Amazon. While I was in the shower it hit me. I know how my AWS credentials were compromised. But I'll need to do a little more research first.

4/15/2014 Update: How did this happened? See Part 2 of this story to find out.

Author: Joe Moreno

First Eco-soap Self-serve Refill Store in San Diego

This article was run in the OB Rag and referenced on the cover page of the The Peninsula Beacon print edition (April 10, 2014).

What happens when a lawyer leaves Corporate America to get in touch with her inner hippie? She opens San Diego’s first eco-soap self-serve refill store in Ocean Beach to do her part to keep our world plastic-free.

Blue Dot Refill bottles for sampling and refill.

In February, Deidre Prozinski opened Blue Dot Refill next to Ocean Beach People's Organic Food Co-op on Voltaire Street. Within days of hanging out her shingle – and without any marketing or advertising – she hit her first milestone: $100 in sales in a single day, thanks, in part, to being right next to a co-op with like-minded customers.

Her business model is simple. Customers bring in their empty single-use plastic bottles for refill rather than throwing them away. They can sample any of the soaps and lotions and customers pay by the ounce. The best part is not only are customers keeping the plastics out of the environment but Prozinski said they are saving 10% – 40% over retail. For customers in a rush, Blue Dot Refill also offers a “drop and shop” option. They can drop off their empty “single-use” bottles and return later to pick them up.

“Recycling isn’t enough,” said Prozinski as she pointed out what most people don’t think about, “If you recycle a plastic bottle, it still exists on our planet. It doesn’t go away. Every single piece has to go somewhere.”

“Sixty percent [of plastics] don’t get recycled,” said Prozinski. The reasons are complex and she broke it down in simple terms, “At the end of the day, recycling is a business. Certain polymers can’t be mixed and someone has to be at the other end to buy the recycled plastic.”

Prozinski is passionate about reducing plastic waste. About a year and a half ago she began wondering why she hadn’t seen a soap and lotion refill store. “Stores buy rice and beans in bulk, why not this?” she asked herself.

Her idea was validated in November, on a trip to Placerville, about 50 miles northeast of Sacramento, when she saw S.O.A.P (Save Our Ailing Planet) doing exactly what she envisioned. “I went from idea to doors open in three and a half months,” she said. Customers, excited to see what she’s doing, continue to drop in and give her ideas such as selling yoga mat cleaners, organic pet shampoos, and massage lotions.

There’s a small irony in that her industrial sized plastic containers used to refill customers bottles can’t be refilled by her suppliers. But Prozinski hasn’t let that stop her. She’s partnered with a permaculture business that will use her empty containers for composting bins and aquaponics. “There’s always a way to make a difference,” she added.

With the growing popularity of her little shop at 4799 ½ Voltaire Street she’s decided to expand with refill shops in Cardiff and South Park/Golden Hill. After that, she wants to have a refill truck she can drive to events just like a food truck. In the meantime, Prozinski offers a 10% Farmer’s Market Discount Day on Wednesdays.

“Refill is the new recycle,” she said. It’s not just her company’s tag line, but her vision for the future.

Author: Joe Moreno

To Support and Defend Heartbleed

I've seen flag officers testify about intelligence gathering techniques that involved spying on Americans. They've defended their possible Fourth Amendment violations by stating that they acted in the interest of national security and protecting the country.

My sticking point with these arguments is military officers take an oath of office that's similar to the Presidential Oath. These oaths make no mention of protecting the country. Rather, it's about protecting the Constitution. I have no doubt that Edward Snowden would argue that he acted in the spirt of this oath, more so than the NSA.

Here are some questions to consider:

1. If a criminal notices a security vulnerability at a bank, would you expect him/her to notify the bank? No.

2. If a security company, charged with protecting the bank, noticed the same vulnerability, would you expect them to notify the bank? Of course.

3. If the NSA had discovered the OpenSSL Heartbleed bug would you expect them to notify the U.S. in the interest of national security? Would you?

At what point should an agency or organization stop defending America in the interest of attacking or spying on others?

Perhaps a government agency did leak the details of this OpenSSL bug. Then again, perhaps they've been exploiting it in the interest of national security. But, I seriously doubt either is the case.

Author: Joe Moreno

Walking Backwards, In Reverse

Whoever thought that nine hours of walking backwards, shown in reverse, could be so interesting?
Full story.

Author: Joe Moreno

Living in a Safe House

The architecture of my Nairobi safe house reminded me of San Diego.

In 2005 I lived in a safe house in Nairobi. Actually, I lived in a couple safe houses, but the layout was essentially the same. What's different, in the real world compared to the movies, is that most safe houses aren't obscure hidden buildings with desolate interiors. Rather, like the White House, they are reinforced houses to prevent home invasions.

Sleep safely on the second floor, behind bars.

My safe houses were located in compounds with six to twelve other homes surrounded by walls topped with concertina (razor wire) and 24/7 guards. Another group I worked with lived in, what we affectionately called, the cathouse (Civil Affairs Team house) which had everything my house had plus an electric fence.

The windows in my safe house were covered with reinforced burglar bars and the bedrooms were located on the second floor where I could lock myself in, similar to a jail cell. There was a lever on my bedroom wall that I could pull to blow out a section of the bars to escape if there was a fire.

The red lever on the wall would blow out a section of bars.

It's hard to believe it's been almost a decade since I lived in Nairobi. This metropolis is the tech center of East Africa and the weather's better than San Diego. The city is located about 1° from the equator and, at a mile up, it's above the mosquito line. The summer highs were in the 80s and the winter lows were in the 50s. Nairobi gets a little more rain than Southern California so things we irrigate for in San Diego, like banana plants and birds of paradise, grow naturally without any humidity.

I highly recommend a trip to Nairobi and safari in Maasai Mara.


Author: Joe Moreno

Today's Cold War is Cyber

What happens when the government of China or North Korea attacks the US?
You'd expect retaliation similar to 9/11 or Pearl Harbor.

What if the goal of the attack isn't to directly harm the US government, but rather a specific business, say, a bank? And, what if it's not a physical attack (with atoms) but, rather, a cyber attack (with electrons)? In this case, since attribution for the attack is difficult, a response can be dicey.

While not an act of terrorism, a cyber attack is similar to terrorism in that it's asymmetrical.

DIME on PMESII 

As I wrote three years ago, defenders in the cyber world do not have the advantage they have in the real world. About ten years ago I studied DIME on PMESII at the Joint Forces Command. When a government wants to impose their will on another less-than-friendly government they have options other than military attacks or spying. Specifically, the actions they can take in irregular warfare are diplomatic, informational, military and/or economic (DIME). Cyber attacks definitely fall under the informational.

U.S. Response

Let's say the NSA discovered, hypothetically, that the government of China was behind the cyber attack that compromised millions of Target's credit cards. How would the United States respond to these attacks? NPR's Fresh Air covered this topic in depth a few days ago. But, the bottom line is, in the name of a proportional response, a counterattack would probably be just as undetectable as the initial offensive. After all, it wasn't a direct attack against the U.S. government or the Constitution, nor was anyone harmed or equipment damaged.

Is it time for commercial ventures to do more than simply provide defensive options?

Watch the entire episode.

Author: Joe Moreno

Colonial Infrastructure

Last night I had a conversation with a fellow former Marine officer. We talked about our experiences while on deployment. One thing that struck me was the differences between a former British colony and a former French colony.

From my experiences in Africa and South West Asia, I noticed that the French colonies were not the place you wanted to be, especially if there was a French Foreign Legion garrison stationed in the country, like Djibouti prior to 2011. British colony infrastructure and public education were noticeably better, on many levels, than the French which is why countries like India and Kenya really shine when it comes to lines-of-communication and STEM.