Software Cities

Yesterday, Dave Winer posted about why in software, we're always starting over. Software engineering is about managing complexity, and it seems to be approaching an asymptotic limit of what can be managed by individuals and companies. It's not that we won't be able to create more sophisticated software, we will; but the growth will be slower and the benefits less noticeable. We're running out of low hanging fruit; we're running out of simple software that performs a useful service as software engineering becomes more prolific.

The real world problem I'm seeing today is that software is becoming so complex that it won't work as expected. Our expectations need to readjust. Yesterday, I couldn't play iTunes Radio or iTunes Match because iTunes simply skipped from song to song without playing any of them. It's not that Apple engineers are incompetent – that's far from the case. Rather, it's a two fold problem. First is what I've already mentioned: software is becoming more and more complex. Second is the fact that new engineers come into the workforce that need to understand legacy software and then either build upon it or reengineer it. Either way, it requires a lot of time and effort. And, unless there's a simplification breakthrough, it's going to result in more complexity for the software engineer.

When pondering this issue, holistically, I look for other examples where I've seen similar problems. Instead of looking at it as a software engineering issue, I look at it as a systems engineering issue. This analogy works well when breaking down problems. For example, we can think of data packets transversing the Internet as cars (packets) carrying payloads of people (data). In this example, we see the redundancy of our roads. Destroying a bridge in Syria has no effect on the roads in the U.S. Or, destroying the Internet's "single point of failure," i.e. DNS, would be the dire equivalent of removing every road sign in the world. As systems fail in ways we didn't imagine, other pathways must handle the load resulting in cyber traffic congestion or even failure to access a network node endpoint.

Gentrification of Software

Software engineering has many similarities to constructing homes and buildings. We even use the same word, architect, in both disciplines. But, in the world of software, we are no longer simply creating buildings. In other words, we are no longer simply making standalone software applications. Instead, we are building entire cities, which, like computers, are networked together. And, like a city, every road can't be open all the time – there's constant construction preventing access. Most of the time, we can plan ahead. But, similar to real world infrastructure failures, like a water main break, we have problems, usually in the form of bugs or hardware failures, in the online world. 

All software needs to be checked for bugs, either by a compiler, coder, tester, or customer. Every new line of code increases complexity, but this is an oversimplification since we usually don't want to compress four lines of code into one. Code written must be debuggable and there's a balance between engineering, over-engineering, and making code intuitive for people to read. One never wants to be too clever when writing code. Too-clever code can end up fooling everyone like debugging a multithreaded race condition. I'm not aware of a formula to compute how dense code is, but an experienced software engineer will get an intuitive feel for it with years of experience. 

As towns and cities require building codes and permits, we may see the same thing in high-tech. Obviously, a bridge failing is catastrophic while Amazon going down is comparatively minor, no one is physically hurt in the latter. Lost revenue is vastly different than lost lives, but, that will change. What if an airplane auto-pilot breaks in-flight? Or, worse, what if it begins misreporting or misinterpreting flight data? In the physical world, our building codes are about safety. Online, our issues are about security – and the two are related. Our online world focus is on attacks rather than infrastructure failings.

While I don't see a need for software performance inspections by third parties, I do see a day when software will be inspected by independent agencies for security.