complexity — John Levy Consulting

July 23, 2013

We all ask this question now and then, particularly when the computer or its application software is not working as it should. Or at least when it’s doing things we don’t understand.

Our smartphones are no different – they are primarily computers, with some radio communications thrown in to connect them with the world. Of course, those radios include 3G and 4G cell phone bands, WiFi and Bluetooth.

Let’s focus for a moment on what every computer has to deal with inside its “kernel” – the core of the operating system. No matter what else is happening at the application level, here are some of the events and processes going on underneath the apps.

1. Memory management – keeping track of who is using which parts of RAM and ROM (often Flash memory). Not only is there constant competition among the applications for RAM space, the operating system itself has to allocate space for its own processes. Furthermore, applications often are using more RAM space than is actually available in hardware. How is this possible? By using “virtual memory” which means that some of the “space” is actually on disk or in Flash or Solid-State Disk (SSD) memory. Only the active “pages” of an application are actually in RAM, while others are “paged out” to disk.

2. Disk Input/Output (I/O) – whether it has a real spinning disk or an SSD, an operating system has to manage the traffic in and out. Since it takes a lot longer to move data to and from disk that it does to compute anything, scheduling has to happen for multiple such transfers at the same time. This means building queues of pending I/O operations, dealing with retries if there are errors, and cleaning up the queues when they are finished.

3. File System – on top of the disk I/O, an operating system usually provides a file system. This creates named collections of data, allocates space for them on disk, and keeps track of all the pieces of a file. Sometimes it also manages redundant file information (such as RAID or block replication) to guard against data loss.

4. Network protocols – there is a separate protocol for each network, such as GSM phone, CDMA phone, WiFi (to Internet), and Bluetooth. Each protocol has multiple layers of interactions going on. The system has to manage each layer by generating and responding to messages coming and going on the network.

5. Error recovery – not all operations in the computer are successful. Here are some of the ways they can fail.

a. When a transmission over a radio link or a disk I/O operation is initiated, there is always a timer set up that counts down and fires off an interrupt when it gets to zero. The interrupt means that the operation has failed, because it didn’t complete within the allotted time.

b. There are other ways to fail, too. Sometimes, a radio transmission will be garbled. This gets detected when a checksum or other code check finds a mismatch in the message. Then the recovery involves sending a reply message that says, “I didn’t get that last message.” And there are multiple layers of network protocol at which such messaging is being done.

c. The hardware of the computer itself fails. A bit gets dropped in the memory, or a bus from one part of the computer to another part loses a pulse and sends the wrong data. This gets detected when the data is checked and found to be an impossible value (a value that is not normal). In this case, the whole operation has be repeated, or else the program that was running and using this data has to be terminated.

6. Power management – in addition to turning the system on and off, mobile systems, such as smartphones, are actively managing power to various subsystems so as to minimize the load on the battery. Most systems also manage screen brightness (more in bright light, less in dim light) and disk spinning (turn off when not in use).

7. User interactions – A user may be typing on a keyboard, moving a finger around on a touch screen, pressing buttons, and reading what is displayed on a screen. There can also be other attachments, such as fingerprint readers or credit card readers, providing data to the system. There is also sound output and voice input. Determining what the user is asking the system to do may require sophisticated software, including motion detection on the touch screen and parsing the voice input.

To support application software, the system may include “frameworks,” such as a Java language framework, which is a collection of facilities that enable programs in Java to be run using a Java Virtual Machine. And there may be other facilities provided by the system, such as an indexing and search capability for emails, calendar items, address book items, web pages, and files. These require sophisticated data structures to be built and added to whenever a new item arrives in the system.

These are a sampling – but not all – of the activities and facilities in your system. There may be hundreds of thousands of lines of code involved in them. And someone – a team of someones – is maintaining that code as it evolves from generation to generation.

Is it any wonder that software project management – and particularly software lifecycle management – is a critical part of a successful vendor’s technology?

What’s wrong with complexity?

March 19, 2013

We tend to design things that are complex, and that can be our undoing.

Technologists love intricate mechanisms. That’s why many of us, as kids, took things apart, and some of us even put them back together again.

In my training as an engineer, I enjoyed learning how mechanical, electrical and chemical things worked. And the more elaborate the mechanisms, the better the challenge and the satisfaction of getting the understanding.

We tend also to design things that are complex, particularly if we’re in software design, because software is layered into abstractions almost without limit. Database systems linked via networks to computational engines and on to user-interaction devices are full of opportunities to exercise our power of design in the face of complex interactions.

Yet complexity can also be our undoing. Consider this from Andre Zolli’s article about the crash of Air France Flight 447:

It was complexity, as much as any factor, which doomed Flight 447. Prior to the crash, the plane had flown through a series of storms, causing a buildup of ice that disabled several of its airspeed sensors — a moderate, but not catastrophic failure. As a safety precaution, the autopilot automatically disengaged, returning control to the human pilots, while flashing them a cryptic “invalid data” alert that revealed little about the underlying problem.

Confronting this ambiguity, the pilots appear to have reverted to rote training procedures that likely made the situation worse: they banked into a climb designed to avoid further danger, which also slowed the plane’s airspeed and sent it into a stall.

Confusingly, at the height of the danger, a blaring alarm in the cockpit indicating the stall went silent — suggesting exactly the opposite of what was actually happening. The plane’s cockpit voice recorder captured the pilots’ last, bewildered exchange:

(Pilot 1) Damn it, we’re going to crash… This can’t be happening!  

(Pilot 2) But what’s happening?

Less than two seconds later, they were dead. …

We rightfully add safety systems to things like planes and oil rigs, and hedge the bets of major banks, in an effort to encourage them to run safely yet ever-more efficiently. Each of these safety features, however, also increases the complexity of the whole. Add enough of them, and soon these otherwise beneficial features become potential sources of risk themselves, as the number of possible interactions — both anticipated and unanticipated — between various components becomes incomprehensibly large. [Want to Build Resilience? Kill the Complexity by Andrew Zolli, 9/26/2012]

This is certainly a cautionary tale about messages that don’t convey important meaning. But it’s also a warning about interactions that were designed but couldn’t be tested or evaluated in all their combinations. That’s what complexity leads to.

Disasters like Flight 447 nearly always require a complex system interacting with a human. Remember the key learnings of the Apollo disaster: NASA’s safety analyses were not being followed up because of a dual-agenda management system. The bottom line was that they relied on the fact that heat-shield tiles had never yet caused serious damage.

When you’re responsible for a project that is complex, you need to address that complexity in two ways.

First, you need to be sure that the people doing the analytical and design work know what the possible failure mechanisms are, how to compensate for them without adding a lot more complexity, and have scheduled adequate tests to validate the robustness of the design.

Second – and this is the more difficult – you have to be sure that the people implementing the project and the people managing the project (including yourself) are not harboring private agendas that may undermine the effectiveness of the analysis and design and testing. Adding ship-date pressure on a team, for example, can cause them to short-change the test plan and declare a product ready to ship when it still has serious faults.

The second area is where your experience with people doing projects will help you most. Listening a lot to project team members and following up on hints of conflict over goals or processes will help you stay current on the health of your project.

Finally, you can become an advocate for simplicity. When faced with a choice in a project between a more complex solution and a simpler solution, go for the simpler one. Often this will allow you to discover sooner whether or not the solution is adequate.

Some projects, of course, become excessively complex no matter what you do. This may be a time when the most responsible thing you can do is recommend that the project be cancelled. Better to have no product than one that kills.

What’s going on inside my computer?

What’s wrong with complexity?