DNS Hijacking?

I have a DNS hijacking theory.

Route 53 is Amazon's elegant DNS web service. DNS is the part of the Internet that converts domain names, like apple.com, into IP addresses such as 17.172.224.47. This is how humans contact computers on the Internet. While DNS is robust, resilient, and redundant, it is the Internet's single point of failure.

So, here's my theory. Websites, like adweek.com, use Route 53:

dig adweek.com ns

returns:

adweek.com. 172800 IN NS ns-304.awsdns-38.com.

adweek.com. 172800 IN NS ns-532.awsdns-02.net.

adweek.com. 172800 IN NS ns-1322.awsdns-37.org.

adweek.com. 172800 IN NS ns-1571.awsdns-04.co.uk.

This means the first time you visit adweek.com, your web browser/ISP will ask one of the Internet's root servers for adweek.com's registrar (i.e., where is the domain name, adweek.com, registered). The root servers will tell your web browsers that adweek.com is hosted at moniker.com (Moniker is a domain name registrar, similar the well-known GoDaddy). The next step is that your web browser will ask Moniker where adweek.com's DNS servers are located. These are referred to as the DNS name servers, or NS for short. As seen above, the response will point your browser to Route 53 which answers with four different servers for redundancy. The final step is that your web browser will query any one of these four servers for the physical IP address of adweek.com. All of these steps happen in the blink of an eye.

Now here's the hijacking part. What if I go to my own Route 53 account, create an entry for adweek.com, and start adding records? When I did this, Route 53 assigned the following four NS servers to me:

ns-715.awsdns-25.net.
ns-1787.awsdns-31.co.uk.
ns-396.awsdns-49.com.
ns-1263.awsdns-29.org.

There should be no hijacking problem since Route 53 assigned four NS servers to me that are different than adweek.com's NS servers. In other words, I cannot hijack adweek.com's Internet traffic in this case. But what if Route 53 had assigned to me an NS server that was the same as adweek.com's NS server? Then, I'm speculating, I could redirect at least a small portion of adweek.com's traffic to wherever I wanted to.

Perhaps this isn't an issue because Route 53 ensures that it never duplicates NS servers names. That would be an expensive proposition, but certainly doable. From there, if my theory holds true, then what about simpler DNS hosts, such as GoDaddy whose DNS servers seem to be limited to nsXX.domaincontrol.com, where XX appears to be a double digit number? This means that many different domain names are using the same DNS server names. Would that make it possible to hijack some traffic from websites sharing the same DNS server? I'm sure that DNS implementations are robust enough that this isn't an issue, otherwise it would have occurred by now. But, with my understanding of the DNS RFC, I don't know how this hijacking issue has been avoided.

So, how has this DNS hijacking scenario been prevented? I'd love to know.

A Busy Day at the Office

What does a busy day look like?
Ten years ago it was reading and writing e-mails. Today, it's bouncing between Slack, Trello, Basecamp, and Facebook.

More tools seems more complicated, but it actually allows for better filtering of information while establishing well defined boundaries. 

I recently started working as the CTO at It's Borrowed, which is like Airbnb for your stuff. Why not earn money renting out that camping equipment, golf clubs, power tools, or wheel chair that's sitting around in your garage? Looking to borrow a ladder or ice chest? Check the It's Borrowed app.

Since It's Borrowed is a small team of five, I'm also the product manager for our API servers and mobile apps. When I last worked in this capacity at Wyndham we only used e-mail and SharePoint, which was marginal. What made it worse, at Wyndham, was that I had to manually sort and archive e-mails in Outlook, my key communications tool. Otherwise, when my e-mail storage grew to more than about 4 GB, Outlook would stop working reliably. Allow me to digress by pointing out that Outlook's horribly engineered to stuff every single e-mail and attachment into a single file which grows and grows until it eventually collapses in an unpredictable way. Compare that to macOS which organizes your e-mail boxes into an elegant hierarchy of folders; each e-mail (body and attachment), is then stored as a separate file. File systems are much better at managing (CRUD) ten-million 1K sized files than two 5GB sized files.

Workflow Filters

Today's workflow, with Basecamp, Slack, Trello, etc., simplifies my life by managing filters. A ding of an incoming e-mail is an interruption; and a message could be from my manager or my mother, each having a unique sense of urgency and importance.

Here's my workflow when using Slack, Trello, and Basecamp...

Basecamp
When I meet with the business (marketing), we put high level tasks into Basecamp such as "Create an e-commerce shopping cart so people can buy our stuff." Think of the tasks in Basecamp as the business's vision – a high level goal.

Trello
Since I'm the two-way bridge between the business and engineering, I take the vision of the business and break it down into a single engineering task that can fit into a Sprint since we're closely following Agile/Scrum. In Trello, our tech lead has organized boards for each Sprint and the backlog. Each column has a card which is akin to an electronic Post-it note. Inside each Sprint board, we have columns such as Blocked, To Do, In Progress, Review, Done, etc. The blocked column are impediments that I take for action, while the tech lead focuses on the other columns.

Each Sprint begins with the engineers reviewing the backlog along with the CEO and me. The engineers get to pick, from our prioritized list, each task (card) they'll do in the upcoming Sprint. Only the engineer picking the task can assign a weight to a task – a weight, meaning how long it'll take a task to be completed. That part is sometimes a hard pill for management to swallow. But, over time, as engineering delivers on all they promise, management will gain confidence even though they can't interrupt a Sprint with new requirements.

Slack
I think of Slack as Twitter on steroids, across different teams and channels. The key Slack teams are the ones that I have with the engineering department and each individual engineer. Since the engineers fall under the tech lead, I rarely communicate directly with an engineer so as to not interrupt them. This is harder than is sounds when the CEO asks me a pressing question which I could simply have answered by interrupting an engineer. A couple of our channels are automated and tied in to third party services so, at any time, I can see code check-ins or deployments to production in real-time.

The beauty of slack is I can see the history of a channel conversation without having to search through e-mail looking for specific threads.

Discipline is key. As long as the team has the motivation to follow this format then the information will quickly and effectively flow, increasing productivity. It really does work well.

The Right Way to Address Bad Service

Customer service is a pet peeve of mine. There's no way for a business to never have a problem. But, what a company does to address the problem is their moment to shine.

I've written many times about stellar customer service here, here, and here. I've also addressed how to get the customer service you deserve. The blog post is one of the latter.

I don't believe in receiving bad customer service and then posting to Yelp about it unless I've addressed it with the business, first, and given them a chance to fix the problem. Today, that happened to me and the business failed.

Blindsided

I've never given anything but 5 star reviews on Yelp. I believe in sprinkling the world with good karma. If I have a problem with a business, then I'll address it, on the spot, to give the business a chance to fix it. Otherwise, I'll let it go. Speak now, or forever hold your tongue. In other words, don't blindside a business with a bad review. Instead, address the issue on the spot with tact and understanding. Tact is the lubricant of human relationships, so use it liberally.

Today, I rented a truck from a local mom-and-pop U-Haul franchise. This is my third time renting from these owners, and, up until now, I was surprised that their seven Yelp reviews averaged out to 1.5 stars. It turns out there's a reason.

The husband and wife team do a good job when everything goes well. But, when something goes wrong, they come across as lazy. My lazy characterization is a combination of my observation, today, and the Yelp reviews.

Today, when I rented my U-Haul truck, I specifically asked for a few extras such as a hand truck, blankets, and a pad lock. The first two items cost extra... no problem, I've rented U-Haul trucks before and I knew that. The last item... the pad lock... is chained to the truck and the key to lock it is on the ignition keychain. When I tried to use the key to unlock the padlock, it wouldn't budge so the owner tried it and she didn't have any luck either. This pad lock is the only thing that secures the back of the truck. I didn't want to leave it parked on the street, overnight, without securing it. Too many homeless in my neighborhood looking for blankets, etc.

When I saw that the owner couldn't open the pad lock, I asked if she could replace it with a working on. She said no. She said that they didn't have any and she recommended that I buy a lock. I asked if there would be a discount and she said no, the pad lock is free. Then I asked if we could get a different truck and she, again, said no because "I have no other trucks," even though there were clearly other trucks, of the same exact model, on the lot. Even if those were reserved, she could swap out the trucks. I have no doubt that she lied to me simply to avoid refilling out new paperwork. She said that she was sorry and she was on her way.

After getting home and trying a bicycle lock, which didn't work, I saw no other option but to buy a new padlock, which I did. Not a big deal, but still an inconvenience, costing me extra time and money. So, it looks like they'll be getting a 4.0 from me on Yelp (my first). Not a big deal for them, since it'll help their 1.5 star average, but perhaps I'm being insensitive. Maybe this husband and wife are on the verge of bankruptcy. Who knows, but they certainly could have done better; or, perhaps I'm too easy going and I was taken advantage of.

Little Stories to be Told

1st Grade, Silas Wood, Huntington Station, Long Island

42 years later, in the Hamptons.

Top: 1st grade at Silas Wood Elementary School.

Bottom: 42 years later I found myself polishing Brian Gerken’s shoes at a dinner party at Victoria’s home in the Hamptons. Brian literally gave me the shirt off his back as a “thank you.”

Photo credit: Our 6th grade teacher, Ms. Cooke, who had Brian, Vicky, and I as her students, snapped the shoe shining photo in the Hamptons.

Coda: There were five people from my first grade class photo in attendance at our dinner party in the Hamptons. It's not that were remained close, over the years, it's simply that timing was on our side as we came together for a 6th grade reunion the evening before our 30th high school reunion.

Good Design is Apparent

The lip (red arrows) wrapping around the sink looks like styling, but it isn't.

Good design is apparent.
Great design is transparent. 
– Joe & Joe (Sparano & Moreno)

Styling is concerned with surface treatment and appearance – the expressive qualities of a product. Design, on the other hand, is primarily concerned with problem solving.
– "Design of the 20th Century" by Charlotte & Peter Fiell

One day, I plugged up the sink on the right with a stopper and stepped away while I left the water running to fill up the basin. I quickly forgot about it until about 20 minutes later. As I hurried back to the kitchen, I could hear the water spilling over before I could see the sink. In my mind's eye, I envisioned how much of a mess of water I'd have to clean up on the floor. Guess what? Not a drop spilled onto the counter or ground. I was floored when I saw that every drop of water had overflowed into the sink basin on the left, thanks to the lip around the sink. For all these years, I simply thought that lip was styling; it turns out it's functional. It's a design element that solves a problem – the exact problem I created by leaving the water running until it overflowed.

Database Architecture & RDBMS Best Practices

The most popular database is Excel. Knowledge workers use Excel spreadsheets as simple, non-relational databases more often than as an accounting worksheet. That's OK, because it works. The big boys (software developers) use relational databases, which is like tying together each worksheet in Excel; the tying together (joining) of database tables is the relational part.

In my computer science college classes I was taught that each row in a database requires a primary key. Just like each row on your Excel spreadsheet has a unique number: 1, 2, 3; each row in a relational database needs a way to find that row of data, quickly. In academia – in other words, in theory – I was frequently taught to use something like a person's social security number (SSN) as the primary key for a database table. But, in practice, that doesn't work nicely. In theory, there's no difference between theory and practice. In practice there is.

One of the key tenets of primary keys is that they must never change unless every single reference to that primary key (the relationship joined by a foreign key) is also updated which is not a trivial task

There are a few issues with using a person's SSN as a primary key. One issue is that it's a long number, much longer than a 1 – 5 digit integer typically used as a primary key, so it takes a database engine longer to compare numbers during a lookup.

More importantly, what happens if a person's SSN changes? SSNs can change in domestic violence cases, identity theft, when two people are mistakenly issued the same SSN, etc. You may think, since these are such rare cases, that it's not a big deal for a database; but software systems have to be developed to handle every possible case to work effectively. Good software should reflect a true reality, not a theoretical reality. So, to solve this issue, most databases issue a sequence of integers for each row. In the case where databases are in a cluster, each database might tag a digit on the end representing which database server issued a specific primary key. In other words, instead of 1, 2, 3; database server #4 in a cluster might issue 14, 24, 34, etc. as primary keys to avoid any primary key collisions.

Database Best Practices

Having done a bit of database work at Apple, I've learned some best practices that are worth sharing. A RDBS should start off in third normal form which means only primary/foreign keys are duplicated in separate tables – the rest of the data should be unique to each table. Later, when there's a performance issue, the database can be optimized, as needed, by denormalizing the data.

1. Primary keys: Never build intelligence into a primary key – a primary key is simply an artifact of the database and it should represent nothing more than a way to access a row in a database table (i.e., don't use SSN as a primary key). Just like a software developer doesn't care what GUID is assigned to an object in memory, neither should a database designer care much about the contents of a primary key. Creating a primary key that's a simple integer is highly efficient since a computer can quickly find and compare numbers (in the case of integers) much faster than a string of nine characters (in the case of SSNs).

I once consulted with a retried Army first sergeant at DTIC, around 1999, who made her RDBS primary keys positive and negative integers. The primary keys for active personnel were positive integers and the retired personnel primary keys were negative integers. While this sounds smart, it's actually too clever. This was pre-9/11 so it was rare for a retired person to come back on active duty. The reason this was a bad decision is that it can be dangerous to change primary keys, and then update all related tables' foreign keys.

2. Table Names: Database table names should be singular (Employee, Order, Transaction, Statistic, etc); they should be named for what each row in the table represents, not the entire collection. The reason is that, typically, there's a one-to-one mapping between a row in a database table and an object used in code. For example, in code, an instance variable referencing an Employee object should represent a single employee from the database while an instance variable that's plural, such as Employees, should represent a collection of objects such as an array or dictionary.

3. Lookup Tables: A lookup table is a simple static database table that's used to populate a list or collection. For example, a list of countries that your company ships to. Perhaps, your company only ships to the U.S. and Canada. Later, when you start shipping to more countries, how do you update the pop-up list of countries on your website or mobile app? With a lookup table, you simply add another row to the table with the new country that you ship to. Updating the database table is easier than changing your code, recompiling, and deploying. Typically, a look up table also has a column representing a sort order. This is done so the list can be displayed in a specific order with, say, the U.S. listed first, instead of Afghanistan, since most of your customers are located in America.

4. Compound Primary Keys: A database table should have a single primary key for a typical one-to-many relationship to another table (join). Sometimes, you need a many-to-many relationship. For example, a Person table/object joined to an Address. A person might have multiple addresses (homes), and an address might belong to multiple people. In these cases, where a many-to-many relationship is needed then a simple middle table is set up that contains only two columns which contain two primary keys propagated from the two joining tables. One of the primary keys in the middle table is the primary key of the Person and the other is the primary key of the Address. Technically speaking, the two primary keys of the middle table are propagated foreign keys. One of the middle table's primary keys is a foreign key from the Person table and the other is a foreign key from the Address table.

I'm not aware of a practical case where more than two primary keys are needed in a database table. In cases where I've seen three (or more) primary keys in a database table I realized that the database designer didn't have a good understanding of relational databases. What they typically needed was a single primary key and indices created for their other columns to optimize their lookup speeds.

5. Number vs Varchar: 

Do not use a numeric type for defining data fields which won't be used for calculations (i.e. "math"). In other words, credit card numbers, phone numbers, SSNs, etc., should be string types (i.e. varchars) in a database model. One specific problem I've encountered on a production system is when a developer stored the credit card security code (CSC) as a numeric data type. Although this credit card code is always numeric, it can contain an important leading zero. When I saw my CSC repeatedly failing at checkout on an e-commerce web site, I immediately knew the problem and confirmed it by reaching out to the DBA.

Military Misconceptions

I had two misconceptions about military retirement when I was a young buck, probably because I’m not a military retiree.

My first misconception was that my four years attending a federal Service Academy (Annapolis, West Point, Air Force Academy) would be applied to my retirement when I reached 20 years of active service. It turns out that’s not the case. However, my four years on active duty at Annapolis would apply to retirement from a civilian federal job.

My second misconception was that I could retire at 20 years. Although retiring at 20 years (or a lesser amount for medical reasons) is effectively how it works for nearly all veterans, there’s a small nuance that’s often overlooked. The military (especially the Navy and USMC) technically calls “retirement” a paid retainer for a period of time, which means that they can still call you back to active duty, involuntarily. What happens if you don’t return to active duty for the call up? Well… the federal government knows where you live since they’re sending you a “retirement” check so they can simply stop paying you.

As we used to joke at Annapolis, NAVY stands for Never Again Volunteer Yourself.

The Beauty of Binary

Boolean algebra was invented by George Boole in the mid-1800s, long before binary numbers had any practical purpose. While binary, which is base 2, is not the simplest numeral system for humans, it's ideal for computers. It's a simple way to store and transfer information. As a matter of fact, you can think of DNA as binary since it only has two combinations (CG or AT) that store all the genetic information of our makeup.

(The simplest and oldest numeral system for humans is unary, which is base 1. Think: tallying numbers with four ones, 1111, while the fifth tally is a diagonal line striking through the four tallies to make one group of five. Actually, traditional tallying seems more like a cross between base 1 and base 5, but I digress.)

There is an elegant simplicity in binary in that each digit is either a one or a zero. On or off. No room for any gray area, even though fractions and negative numbers can still be represented in binary. Additionally, some numbers that can't truly be expressed in one base, for example, 1/3 in base 10, can be simply written in, say, base 3 as 0.13.

Since computers use binary, some integers operations are child's play to a computer, especially bit shifting. As humans, we can't easily figure out multiplication of large numbers in our head. For example, what is 123 x 45? That will require a pencil and paper or calculator. But, we can easily figure out the answer to 12345 x 100, even though the latter deals with much larger numbers because we simply shift the digits three places. But, for humans, this calculation only works for powers of 10, since we think in base 10. Computers, however, get this luxury when they're multiplying by integers that are a multiple of the base. Multiplying a base 2 number by 2, 4, 8 is as simple as shifting the bits by one, two, three, or four places. For a computer, like a human, this is a much simpler task than working through the traditional arithmetic.

Simple Influences

Simplicity is the ultimate sophistication.
– Leonardo da Vinci

The Marines and Apple have been the biggest influences in my professional life... my way of thinking... my work philosophy.

At the end of the day, they both respect and seek simplicity. Simplicity of design in products. Simplicity of design in tactics. While both organizations are well respected leaders in their fields, they go about solving problems in different ways. One's procedural and the other's artistic. One creates and the other destroys.

But, there's nothing wrong with creating good and destroying bad.

Mental Health in America

How does someone end up on the streets? There are many different reasons such as abuse, drugs, loss of a job, and mental health issues. Over the past few months, I've watched the mental health of a  friend and former coworker from Apple, decline until he ended up living in his office, car, and then ended up on the streets. Help and love from family, friends, his church, and coworkers did no good since he wouldn't accept any aid.

Last time this happened to him, he ended up being arrested and committed. Unfortunately, until someone exhibits some harmful behavior – what authorities refer to as "fitting the profile" – there's not much that can be done by others. Sure, we tried talking to him, in good faith, but when one's brain can't accept reality, logic does no good. From his point of view, the entire world doesn't understand his genius and he things we're all squelching his creativity.

Today, a few of us, including his wife and mother, had to pack up his office, which he had been squatting at, for a couple of months, until his sympathetic landlord had no other choice. We wondered what street he was living on as we boxed up his computers, routers, books, marketing collateral, and training materials.

But, that's life in America. We're free until we harm.