What is the worst disaster that can be attributed to the negligence of a software engineer?

Home - Quora Link

The problem with your question is that disasters are never the fault of an individual engineer. The worst software engineering disaster I personally can think of, the Knight Capital incident that destroyed $10mm worth of value a minute for 44 minutes, could and should not be attributed to the failure of a single engineer, as much as one might hope.

No, the poor schmuck who takes the fall for such incidents is always just that, a poor schmuck who just happened to be in the wrong place at the wrong time. Even if they were actually the one to run the command to destroy the database / wipe the code repo / wreck the server, it’s not really his fault.

The reason why comes down to two principles. Safety and architecture.

If I hired you to paint my fence, and gave you $15 an hour to do so, sounds like a good deal for the right person, right? What if my fence happened to be enclosing my alligator refuge? Let’s say you took the job anyway because you needed the money. If you get attacked by an alligator, whose fault is it? Yours because you happened to piss off an alligator, or mine because I didn’t take proper safety precautions and hired the wrong person for the job hoping to save a buck?

Any system complicated enough to be worked on by engineers is dangerous, people’s jobs ride on the proper functioning of that system. Hiring skilled developers / engineers is intended to mitigate that risk. But there just aren’t enough highly-skilled and experienced engineers out there for everybody to fill their staffs with. Invariably it is required to hire junior developers.

Junior devs need safety rails and sandbox systems in which they can try stuff out and learn without putting production systems at risk. Because every system’s different and there are myriad setups out there, and you won’t always be able to find someone that’s got experience in your particular setup. This stuff times time and resources to build and maintain. The company has to value these things. When done right, safety becomes a culture within the organization.

Second, the company needs to hire someone skilled enough to do proper architecture. Architecture is a high level plan for how the various moving parts of the system interact with each other. Ideally architecture is set once and then only changed extremely rarely, these changes planned for and executed with the utmost precision. In most outfits, architecture is an afterthought and everyone is in firefighting mode most of the time.

But with good architecture, there are always multiple fall-back points where you can get back to a known-good system state quickly should it ever be required. For example, everybody says they back up their systems, but very few people take the necessary next step and think about just how much disruption restoring from a backup is going to cause. They don’t run simulations where they take down production systems and restore from the backup, just to find out what’s likely to happen in these events. As a result, when disaster does strike, you run into issues with the restoration process at the worst possible time to run into them at.

I’m not saying proper architecture and safety culture is common, in fact, it’s very uncommon for these things to be done right. This is because you can drive business value without them, and do it more cheaply. Being that the usual approach of business is to do things more cheaply when possible, skimping on safety culture and thoughtful architecture is the far more common occurrence. And then they blame the poor junior schmuck for negligence when it all breaks.

Blame him all you want, but it’s your millions that are going to be lost.