Absolute Poor

Antony and I on my last day in Nairobi.

It's hard to imagine living in absolute poverty.

I encounter the relative poor everyday. It's not uncommon to see overweight homeless people walking the streets of America. In developing countries, the poorest of the poor are skinny. When I lived in East Africa I could usually figure out who was the local village chief since he was the least skinny guy.

Absolute poverty didn't hit me until I lived in Nairobi. My predecessor told me it was customary for us to pay for meals for our drivers. My driver, Antony, took me to a gas station with fast food restaurants for dinner. He had chicken and rice and I had a Margherita pizza. Each of our dinners was the equivalent of $4.

We sat outside as I listen to him tell me about Kenya. After I finished eating I went back inside and bought a Nestle Drumstick. I was surprised it cost as much as my dinner. Pizza, chicken, and rice are inexpensive since their key ingredients are grown and processed locally. But, food that's imported, like name brand ice cream, is pricy.

When I sat back down at our table I asked Antony how much he thought my ice cream cost. His guess was about $1.50.

I'll never forget his response when I showed him my receipt. He casually tapped on the total, at the bottom of the receipt, and said, "That is more money than I make in a day."

More money than I make in a day?!?

Those words have haunted me ever since he spoke them.

My new best friend in Kenya was living on just over $3/day. It should have occurred to me sooner. I was responsible for managing my team's rental car contracts. I knew that we paid $70/day for our SUVs. It didn't dawn on me that it was only an extra $6/day to have a driver. That left just over $3 for Antony after the rental car company took their cut

Living on three dollars a day. Amazing.

Author: Joe Moreno

To Offshore or Not to Offshore at Apple?

When I worked at the Apple Online Store we were organized in teams of six.

My team consisted of four software engineers, one project manager (Scrum Master), and one QA engineer. The QA engineer was a permanent part of our team. He was a white-box tester. We wrote our own unit tests and demonstrated scalability with our component tests while our QA engineer reviewed our logic. He'd look for obvious issues like uncaught null pointer exceptions while digging deeper in search of ambiguous cases like poor security implementations.

New Code Here

Any issues our QA engineer found in our code, during a sprint, were fixed by the software engineers on our team. The rest of the bugs (outside of our recent sprints) were entered into Apple's bug tracking system, Radar. Then, once a week, we'd meet to prioritize bug fixes in Radar. We off-shored the bug fixes to India since it wasn't sexy work. Once it was fixed, we reviewed the code and verified it before integrating it into the main branch.

Bug Fix There

Offshoring bug fixes worked beautifully. Each bug was clearly documented: what really happened vs. what was suppose to happen. I had no idea who was on the other end fixing our bugs but I realized they were intelligent and hard working. However, I could tell they weren't experienced with our technology (WebObjects) or conventions. As one example, I reviewed code where the offshore team had hard coded SQL queries directly into Java. Other times, I saw Java objects instantiated simply to access static Java methods.

The beauty of offshoring a bug fix is we could focus on new feature development.

New Code There

Since offshoring bug fixes worked so well we decided to give them a shot at new development. We quickly discovered that was a mistake. The offshore team didn't have enough context to write good code. Their implementations were too brittle.

This problem frequently happens in any coding organization that's offshoring new development. Without a product roadmap, the offshore team simply writes code to do exactly what you asked for; and no more.

I never saw requirements sent to an offshore team to refactor code. That would be too nebulous of a task. By the time I would have documented all the ins and outs of a refactoring requirement,  I could have written the code myself. The real problem is that we don't know how our code will behave until we run it.

And that was the crux of the problem with the code written by the offshore teams I've dealt with. They could only do exactly what you asked for, now, without knowing what was coming. Plus, the requirements had to be very explicit. 

Software engineering isn't an event – it's a process. It's a process of continue improvement and refinement. It's iterative.

Work Life Balance at a Startup

Last April I heard Brad Feld speak at UCSD.

It was a common VC talk. Brad discussed his ideas about growing the San Diego startup community.

Then came the final question from an audience member. It was about maintaining a healthy work-life balance while running a startup. Brad's answer was the most practical relationship advice I've heard.

Nothing is More Important Than Her Call

Whenever Brad's wife calls, he answers. Always. Brad told us that she knows he'll take her call so she doesn't abuse it.

Update: Big thanks to Jerry Jones for pointing out an example of Brad taking a phone call while on the air.

Someone told me they heard Brad take her call during a podcast interview. But that was hearsay. This morning I was having coffee with the Giesen twins who validated that Brad practices what he preaches. The twins watched Brad take his wife's call, on stage, a couple years ago.

Monthly Dinner of Dumping

If something is bothering Brad or his wife then they wait for their monthly dinner. That's when they get to unload anything and everything on each other. The key is to do this every month on the same day. Sometimes this is a simple 20 minute process. Other times the dinner lasts for hours with tears and food servers who won't go near their table until things simmer down.

Life, like work, requires dedicated perseverance.


Author: Joe Moreno

World's Largest Food Fight?

One of three wings of King Hall

Midshipmen at the Naval Academy eat in King Hall. It's the most impressive dining facility I've ever seen in terms of logistical operations. The Brigade of Midshipman, 4,500 strong, march into this mess hall at the same time. Within a few minutes of sitting down every table of twelve is served a hot meal. I know of no other place where thousands are seated and served so quickly. It's quite a sight to behold.

Spontaneous food fights broke out two of the four years I attended the Academy. Always on Halloween.

The first year it happened I was a plebe on duty, so I missed it. The second time it erupted I was right in the middle of it where the three wings joined. It started at the far end of King Hall and I saw a wall of food traveling in my direction. It only took a second to reach me. Midshipmen – especially the plebes, since this was their chance to rebel – picked up food and threw it.

Within 30 seconds the melee was over. There was no food left to throw. It was all on the wall, floor, or us. I was fortunate enough to take cover in a defensive position between the end of my table and the wall. This act spared my uniform from battle damage and undeserved "fruit salad." My shoes, on the other hand, were a mess by the time I made it out of King Hall since I was stepping and slipping on sauces, gravy, salad and veggies.

How, why, or who started it remains a mystery.

Biggest Battle?

A King Hall table of 12 represents a squad.

So, was I involved in the world's largest food fight? That official title goes to the town of Buñol, Spain. Their annual food fight began in the mid-1940s and it's grown to about 20,000 people who show up to throw tomatoes at one another.

I chose to ignore the Spanish event as a food fight. It doesn't compare. For starters, it should be called a tomato fight since that's the only weapon used to engage the enemy. Second, the festival in Spain is planned – I'm talking about the difference between a boxing match and barroom brawl.

I wish I had video or photos of my epic battle with nameless heroes. And, for the record, I never threw a single piece of food. I always thought it was a tremendous waste.

If you happen to know of a bigger, unplanned, food fight involving more than 4,500 people then please let me know at joe@usna93.com.

Author: Joe Moreno

What Hardware and OS are Inside Apple's Data Centers?

Here are a few things to consider about Apple's infrastructure.

Apple used to make the Xserve.

It was a beautifully designed piece of hardware, inside and out. Apple stopped shipping it about three and a half years ago.

Apple maintains it's own data centers.

What's inside these massive data centers? "Stuff," said Steve in this short video clip. Obviously, these data centers are packed full of servers.

So, what hardware & operating system are powering Apple's data centers?

The Apple data centers are most certainly not running Xserve hardware and they're not running OS X Server. I'd speculate they're running HP or IBM hardware with some flavor of Unix, perhaps even Linux.

Anyone else care to take a guess?

Author: Joe Moreno

At the Cafe: To laptop, or not to laptop

Banned at August First bakery: Laptops and tablets.

On the shores of Lake Champlain is a bakery cafe that's banned screens. This Burlington, VT cafe opened four years ago with free WiFi. Quickly, the owners noticed that patrons were camping out. The table space they took up started to affect their bottom line.

On the other side of the spectrum is San Diego's (and my) favorite coffee shop, Old Cal Coffee, in San Marcos, CA. They invite customers to spend all day. Frequently, I, along with others, have spent more than four hours there and I have yet to see a patron shooed away. I once saw a regular, who seemed to be living out of his car, sitting on the patio one holiday when the cafe was closed just to use the WiFi.

Right or Wrong?

The answer to the question, "Is this right or wrong?" is simple: It depends.

Sometimes I need to be offline when I want to be online and vice versa.

Keep in mind that these establishments are private owned businesses open to the public. The owners make the rules and we, the customers, are their guests. And, in the end, we vote with our money. In the case of the August First Bakery in Burlington, are customers going there for food or free WiFi? If it's the food, then the business should survive by banning screens.

Author: Joe Moreno

$5,000 Security Breach, Part 2

Every so often I write a blog post that immediately receives many thousands of views. Part 1 of this story fell into that category.

Where I last left off, on Thursday, I was in the shower when I had an epiphany. I had figured out how my Amazon Web Services credentials were compromised. At least I suspected, but I was running late, after my call with Amazon, as I got ready for the Spring Fling tech event. I didn't have time to comb through my public repository account so I deleted my entire GitHub account. I had only used it once, years ago, when I checked in an open source WebObjects project I had developed.

Jodi Mardesich interviewed me for the details and gave my story a great write up at ReadWrite.

Coda update: Amazon has confirmed that they'll grant me a one time exception for my faux pas.

Author: Joe Moreno

$5,000 Security Breach, Part 1

$17,000 AWS bill in the making.

4/15/2014 Update: This story was picked up by ReadWrite.

Part 2 Here

The problem with the Heartbleed bug is you never know when and where you'll get hit. Actually, this is true for all security breaches.

Yesterday, I received an e-mail from Amazon asking me to update my credit card info for one of my personal Amazon Web Services (AWS) accounts. I logged in and saw that my running total for April was over $5,300. My typical monthly bill is less than $6.00 which is about 1,000 times less.

At first I thought it was a mistake. I hadn't fired up any EC2 instances this month and my account had no EC2 instances running in my region. I filled out a billing inquiry request form and selected the "call me now" option. Within a minute my phone rang and I was speaking to an AWS customer service rep.

I explained that I've been an AWS customer since 2007 and I've never seen a billing issue like this.

He said, "We've been seeing more and more of this. Check your spot EC2 instances in other regions and you'll see high end instances running."

Sure enough, he was right. In Tokyo, São Paulo, Sydney, and Singapore I had expensive server instances running.

"Your AWS credentials have been compromised," he said.

How did they get compromised? When did it happen? Was my development machine hacked? It couldn't be my Time Capsule since that's encrypted. Were one of my physical servers hacked? Did I have a backup, sitting on a server, somewhere, that was hacked? Am I about to get stuck with a $5,360 bill?

"'They' spin up spot instances which isn't subject to Billing Alerts. You'll need to cancel those spot instances, revoke your AWS credentials, and change your account password," he said.

"When did this happen?" I asked.

"Let me look," he replied.

My mind was still racing as I tried to figure out the source of the breach.

"These instances were spun up on April 2," he said.

Very smart; launch the attack early in the month so the victim won't know anything's wrong until they get next month's bill.

"Is this related to Heartbleed?" I asked. It had to be.

"No, it's just a case of your AWS credentials getting compromised," he answered.

He walked me through the steps to secure my account.

"Can you see what 'they' were doing with these spot instances?" I asked.

"No, we can't see inside the instances. But, they're usually mining for Bitcoin," he answered.

Ahh, now that makes sense. Spend $5,000 of someone else's money to mine, say, $1,000 of Bitcoin for yourself. Can't follow that money trail.

"I'm going to send you a questionnaire. Please fill it out describing what happened and, due to the large amount involved, I'll need a manager to review it. But you won't have to pay for what you're not responsible for," he said.

I let out a sigh of relief.

I was still dripping with sweat since I'd just returned from a run when I saw the initial e-mail from Amazon. While I was in the shower it hit me. I know how my AWS credentials were compromised. But I'll need to do a little more research first.

4/15/2014 Update: How did this happened? See Part 2 of this story to find out.

Author: Joe Moreno

First Eco-soap Self-serve Refill Store in San Diego

This article was run in the OB Rag and referenced on the cover page of the The Peninsula Beacon print edition (April 10, 2014).

What happens when a lawyer leaves Corporate America to get in touch with her inner hippie? She opens San Diego’s first eco-soap self-serve refill store in Ocean Beach to do her part to keep our world plastic-free.

Blue Dot Refill bottles for sampling and refill.

In February, Deidre Prozinski opened Blue Dot Refill next to Ocean Beach People's Organic Food Co-op on Voltaire Street. Within days of hanging out her shingle – and without any marketing or advertising – she hit her first milestone: $100 in sales in a single day, thanks, in part, to being right next to a co-op with like-minded customers.

Her business model is simple. Customers bring in their empty single-use plastic bottles for refill rather than throwing them away. They can sample any of the soaps and lotions and customers pay by the ounce. The best part is not only are customers keeping the plastics out of the environment but Prozinski said they are saving 10% – 40% over retail. For customers in a rush, Blue Dot Refill also offers a “drop and shop” option. They can drop off their empty “single-use” bottles and return later to pick them up.

“Recycling isn’t enough,” said Prozinski as she pointed out what most people don’t think about, “If you recycle a plastic bottle, it still exists on our planet. It doesn’t go away. Every single piece has to go somewhere.”

“Sixty percent [of plastics] don’t get recycled,” said Prozinski. The reasons are complex and she broke it down in simple terms, “At the end of the day, recycling is a business. Certain polymers can’t be mixed and someone has to be at the other end to buy the recycled plastic.”

Prozinski is passionate about reducing plastic waste. About a year and a half ago she began wondering why she hadn’t seen a soap and lotion refill store. “Stores buy rice and beans in bulk, why not this?” she asked herself.

Her idea was validated in November, on a trip to Placerville, about 50 miles northeast of Sacramento, when she saw S.O.A.P (Save Our Ailing Planet) doing exactly what she envisioned. “I went from idea to doors open in three and a half months,” she said. Customers, excited to see what she’s doing, continue to drop in and give her ideas such as selling yoga mat cleaners, organic pet shampoos, and massage lotions.

There’s a small irony in that her industrial sized plastic containers used to refill customers bottles can’t be refilled by her suppliers. But Prozinski hasn’t let that stop her. She’s partnered with a permaculture business that will use her empty containers for composting bins and aquaponics. “There’s always a way to make a difference,” she added.

With the growing popularity of her little shop at 4799 ½ Voltaire Street she’s decided to expand with refill shops in Cardiff and South Park/Golden Hill. After that, she wants to have a refill truck she can drive to events just like a food truck. In the meantime, Prozinski offers a 10% Farmer’s Market Discount Day on Wednesdays.

“Refill is the new recycle,” she said. It’s not just her company’s tag line, but her vision for the future.

Author: Joe Moreno

To Support and Defend Heartbleed

I've seen flag officers testify about intelligence gathering techniques that involved spying on Americans. They've defended their possible Fourth Amendment violations by stating that they acted in the interest of national security and protecting the country.

My sticking point with these arguments is military officers take an oath of office that's similar to the Presidential Oath. These oaths make no mention of protecting the country. Rather, it's about protecting the Constitution. I have no doubt that Edward Snowden would argue that he acted in the spirt of this oath, more so than the NSA.

Here are some questions to consider:

1. If a criminal notices a security vulnerability at a bank, would you expect him/her to notify the bank? No.

2. If a security company, charged with protecting the bank, noticed the same vulnerability, would you expect them to notify the bank? Of course.

3. If the NSA had discovered the OpenSSL Heartbleed bug would you expect them to notify the U.S. in the interest of national security? Would you?

At what point should an agency or organization stop defending America in the interest of attacking or spying on others?

Perhaps a government agency did leak the details of this OpenSSL bug. Then again, perhaps they've been exploiting it in the interest of national security. But, I seriously doubt either is the case.

Author: Joe Moreno

Walking Backwards, In Reverse

Whoever thought that nine hours of walking backwards, shown in reverse, could be so interesting?
Full story.

Author: Joe Moreno

Living in a Safe House

The architecture of my Nairobi safe house reminded me of San Diego.

In 2005 I lived in a safe house in Nairobi. Actually, I lived in a couple safe houses, but the layout was essentially the same. What's different, in the real world compared to the movies, is that most safe houses aren't obscure hidden buildings with desolate interiors. Rather, like the White House, they are reinforced houses to prevent home invasions.

Sleep safely on the second floor, behind bars.

My safe houses were located in compounds with six to twelve other homes surrounded by walls topped with concertina (razor wire) and 24/7 guards. Another group I worked with lived in, what we affectionately called, the cathouse (Civil Affairs Team house) which had everything my house had plus an electric fence.

The windows in my safe house were covered with reinforced burglar bars and the bedrooms were located on the second floor where I could lock myself in, similar to a jail cell. There was a lever on my bedroom wall that I could pull to blow out a section of the bars to escape if there was a fire.

The red lever on the wall would blow out a section of bars.

It's hard to believe it's been almost a decade since I lived in Nairobi. This metropolis is the tech center of East Africa and the weather's better than San Diego. The city is located about 1° from the equator and, at a mile up, it's above the mosquito line. The summer highs were in the 80s and the winter lows were in the 50s. Nairobi gets a little more rain than Southern California so things we irrigate for in San Diego, like banana plants and birds of paradise, grow naturally without any humidity.

I highly recommend a trip to Nairobi and safari in Maasai Mara.


Author: Joe Moreno

Today's Cold War is Cyber

What happens when the government of China or North Korea attacks the US?
You'd expect retaliation similar to 9/11 or Pearl Harbor.

What if the goal of the attack isn't to directly harm the US government, but rather a specific business, say, a bank? And, what if it's not a physical attack (with atoms) but, rather, a cyber attack (with electrons)? In this case, since attribution for the attack is difficult, a response can be dicey.

While not an act of terrorism, a cyber attack is similar to terrorism in that it's asymmetrical.

DIME on PMESII 

As I wrote three years ago, defenders in the cyber world do not have the advantage they have in the real world. About ten years ago I studied DIME on PMESII at the Joint Forces Command. When a government wants to impose their will on another less-than-friendly government they have options other than military attacks or spying. Specifically, the actions they can take in irregular warfare are diplomatic, informational, military and/or economic (DIME). Cyber attacks definitely fall under the informational.

U.S. Response

Let's say the NSA discovered, hypothetically, that the government of China was behind the cyber attack that compromised millions of Target's credit cards. How would the United States respond to these attacks? NPR's Fresh Air covered this topic in depth a few days ago. But, the bottom line is, in the name of a proportional response, a counterattack would probably be just as undetectable as the initial offensive. After all, it wasn't a direct attack against the U.S. government or the Constitution, nor was anyone harmed or equipment damaged.

Is it time for commercial ventures to do more than simply provide defensive options?

Watch the entire episode.

Author: Joe Moreno

Colonial Infrastructure

Last night I had a conversation with a fellow former Marine officer. We talked about our experiences while on deployment. One thing that struck me was the differences between a former British colony and a former French colony.

From my experiences in Africa and South West Asia, I noticed that the French colonies were not the place you wanted to be, especially if there was a French Foreign Legion garrison stationed in the country, like Djibouti prior to 2011. British colony infrastructure and public education were noticeably better, on many levels, than the French which is why countries like India and Kenya really shine when it comes to lines-of-communication and STEM.

Lazy Programming

There are two types of lazy programming, good and bad.

Good Lazy

Lazily instantiating and populating data structures is a perfect example of a good design pattern. This technique is also how CDNs populate their edge servers. Don't create or store anything until the last possible moment.

When implementing this technique, I use accessors that have the same name as my instance variables (ivars). Below, my _employees ivar is set to null when a class is instantiated and it's not populated until the first time it's touched (accessed). This is the beauty of key-value coding accessor methods.

private NSMutableArray _employees = null;

public NSMutableArray employees()
{
    if (_employees == null)
    {
        this.setEmployees(new NSMutableArray());
    }
    return _employees;
}

public void setEmployees(NSMutableArray newEmployees)
{
    _employees = newEmployees;
}

Depending on my performance requirements, this design pattern would work if I needed to save memory. However, if memory isn't an issue, but, rather speed, this might not be an ideal solution since each time the employees() method is called there's a an O(1) test performed to see if the private ivar is null. In cases where speed needs to be optimized then it's best to pre-populate the data structures (caches) before the web app begins accepting requests. At the Apple Online Store, we pre-populated only when necessary. In every case, though, the key is to avoid premature optimization.

Bad Lazy

The goal of a software engineer is to provide the best possible user experience (BPUX).

As a programmer, I'm not shooting for perfection but I know when something can be done better. (If I went to sleep last night then I had time.)

If I have to code something that's singular or plural I'll go out of my way so it doesn't read:
You have 1 item(s) in your cart.

It's not very hard to code:
You have 0 items in your cart.

You have 1 item in your cart.

You have 2 items in your cart.

There is no shortage of websites where I've entered my phone number (760.444.4721) or credit card number (4111-1111-1111-1111) only to hit enter and been told I made a mistake and my digits need to be reentered with only numeric characters.

Some programmer had to go out of their way to search the string I entered, confirm there was a non-numeric character, and then return an error message to me. This is my big pet peeve – it's too in-your-face. I entered all the information the programmer needed and they could have parsed out the digits. When I'm coding, I simply write a cover method to return only the numeric digits.

Software engineers aren't sales people, so they don't live the ABCs.

1,000 songs in your pocket.

CNBC tweet on Amazon's Fire TV.

BLUF: Lead with the benefits before the features.

When Apple announced the first iPod they didn't market it as a 6.5 ounce MP3 player with a 5 GB hard drive. Instead, the copy simply read, "1,000 songs in your pocket."

When I saw this random tweet about Amazon's Fire TV I stopped and asked myself, "How does that compare to Apple TV?" Then I realized it didn't matter.

What's redline on your car engine? How fast is your smartphone's CPU? Is the CPU dual-core or quad-core? Does your car have four, six, or eight cylinders? How many cubic inches or liters does it displace?

Detailed tech specs matter to an engineer but they don't have the same marketing impact on consumers. This CNBC sponsored tweet, taken slightly out of context to illustrate my point, should probably be tailored for the latter in a world where color and style matter in consumer electronics. Tech specs were once important, but, nowadays, they're not a big deal for consumers. It's all about best possible customer experience.