Category: Humor

Circa 2010, Becca‘s employer, Initech, was growing. “Growing”, in this case, meant “acquiring competitors who had niche products that their customers wanted added to Initech’s portfolio”.

One such product was a content-management/workflow tool, already sold to some big-name, multi-billion dollar companies. The tool fell into that niche between “really useful to the people who need it” and “buggy as an ant hill inside of a termite mound under a wasp nest”. Sales were good, but the devs were underwater and the backlog of feature requests and bug fixes were growing. So Initech bought the vendor, fired most of the developers, and handed it to an Initech team.

“It’s just some bugfixes,” management said, “what could go wrong?”

The first thing that went wrong was the senior dev assigned to the project rage quit after two weeks with it. “I ran the install scripts, which are supposed to provision a new database and deploy the web app files to a web server. There are thousands of interlocking scripts to make this happen, and half of them don’t work. Along the way, it creates hundreds of SQL tables, all with names like table1, or important_table1, and many of them are never queried by anything in the code, despite having data.” What followed was an ultimatum: take me out of this project or I’ll take myself out of the company.

Given that this senior dev was central to many other projects, the company shuffled things around and put another sacrificial lamb in place, a different senior dev. This dev needed to add a small feature- a new field to a screen. Six weeks later, the dev had something that mostly worked. It was a hacked on retrofit- there was no “right way” to add a new field to a screen; any modifications to any of the front end or back end or database code tended to cause explosions, thunderstorms, and the gnashing of teeth. This dev, also very high up and respected, repeated the ultimatum.

When adding a new field to a screen takes six weeks and burns a developer out, you can imagine how hard it is to fix bugs. The junior devs put on bug fixes were burning out just as quickly, but producing far fewer results. The bug backlog kept growing far faster than bugs could get patched.

if(hasrating)
{
thechkbox = currRating[0].parentElement.parentElement.parentElement.parentElement.parentElement.parentElement.children(2).children(0).children(0)
thechkbox.disabled = false;
}

That was some of the better code in the application. Much of the code was a bizarre mix of Classic ASP server-side code and JavaScript- with the ASP code frequently generating the JavaScript. It also generated SQL queries through string concatenation. It’d often stuff those queries into data- attributes in HTML elements because that was the only way it new how to link data together in the UI. Many times, raw SQL queries would get placed as GET parameters on the URL.

As you could imagine, this meant plenty of options for SQL injection attacks, but they were rarely needed: your privilege level was stored as text in an <input type="hidden">. Anyone who knew how to use the browser debugger could promote themselves to Admin.

Staring at this, the company decided to do the natural thing: dedicate all of their developers to a gigantic, ground-up, rewrite. “If we put everybody on this, it’ll be done in, what, three months? We can totally do this.”

Well, they couldn’t put everyone on it full time- the existing package was so buggy and dangerous that developers constantly needed to be pulled back into the old version to just keep it running. But also, when trying to understand the requirements for a rebuild, they rapidly discovered that every customer had a different version, with different features, and completely different needs.

The original developers had never used source control. When they built the software, they built it for one customer, and deployed it. When the next customer asked for the software, they copied all the files to a new directory, and tweaked and modified it to suit that customer. When the third customer came on, they looked at the versions they deployed for Customer A and Customer B, decided which one made a better starting point, and then copied-and-pasted for Customer C. That, by the way, was why there were hundreds of database tables that seemed to have no purpose: they didn’t for this version of the software, but did for other customers.

The attempt at a ground up rewrite lasted all of two days before management admitted defeat. That was still too little, too late. The general awfulness of this new product, the fact that developers kept getting shuffled onto it and burning out in weeks, and the sense that the company had no plan to get to a reasonable software development lifecycle for it meant that anyone with experience jumped ship- including Becca.

Within three months of purchasing this white elephant, Initech had lost about 70% of its developers. Their core products started to be negatively impacted, and the company very nearly went under.

Now, so-called “vulture capital” investment tends to take loans to buy companies that aren’t very profitable (but have plenty of assets). They transfer the debt to the company (making them liable for their own purchase), strip them of any assets of value, and then send them off to collapse under the weight of debt obligations they can’t possibly fulfill. Venture capitalists get rich, companies go out of business.

Initech’s “fix” for all of this was to steal a page from that playbook. They went into debt to buy another company with some promising products, transferred those products to Initech’s portfolio, and took all of the tech-debt products and shoved them into the victim’s portfolio. Initech kept a generous licensing agreement to continue to sell and host the broken product, but got to blame the “vendor” for its crappiness- and paid nothing to its ongoing maintenance or support.

Power outages are never good, and they’re even worse when your facility needs to run 24/7. Now, Jaroslaw‘s organization didn’t do a great job setting up for round-the-clock, always-on operations. It was the kind of thing where the organization grew, annexed the neighboring building, and kept growing. The result was hundreds of workstations, two separate power lines, two server rooms, three different Internet uplinks, and huge piles of switches responsible for making this network work.

Which added the problem that after a power outage, nothing came back on exactly right, either. It always took some time to find the one switch that opted not to reboot.

Now, many years earlier, someone had the bright idea of installing a generator. The hookup offered no easy way to switch over to generator power, and thus required an electrician with keys to the elecrtical boxes to actually make the change. While the servers had small UPSes, enough of the environment went down during a power outage that, by the time they had the generator on and everything powered back up properly, the outage was usually over.

And so it went for years, until someone higher up looked at the problem and freed up the budget to fix things. The generator was replaced, and there was a plan to change the wiring so that it was faster to switch over- but it turned out that would have tripled the budget and shut the facility down for days while electricians redid the whole electrical system. Instead, the budget was used to upgrade from small, consumer-grade UPSes to a big hocking, 10kW unit.

It was the size of a large refrigerator, and had enough power to keep all the critical elements of the facility powered on for twenty hours- time enough to switch over to the generator, if needed.

And then, miracle of miracles, they tested their switchover plan. They cut main power, saw the UPS come on, ensured work could continue, then had the electricians switch on the generator, and then reversed the whole process. It went off without a hitch.

And then a week later, the UPS screamed about an overload. It lasted for about 40 seconds, then cleared up. Considering that the UPS had way more capacity than they needed, that seemed like a serious problem. Two hours later, it happened again. And again. And again. Jaroslaw went through everything in the server room, trying to find the badly behaved device. At one point, he found an unplugged electric kettle sitting not far from the server room, and went on a hunt to see if anyone had been making tea in the server room, thinking that was the culprit. No one had.

Over three days, after checking all the equipment, Jaroslaw went to the building wiring diagram and started checking every outlet. He found one, hidden in the back of the server room, ostensibly unused, that had an extension cord plugged in. The cord was neatly tucked into the cable chase, as if it was part of the plan. Jaroslaw tracked the cable, and followed it around the room until he found a hidden refrigerator. Some of the 24/7 staff wanted easy access to snacks and drinks, and didn’t want to constantly badge in and out of the server room to get them. While there were plenty of non-UPS protected outlets they could have used, someone had decided this was a better option.

And sure enough, while Jaroslaw was looking at the fridge, he heard the compressor kick on, and the UPS scream about an overload at the same time.

The immediate fix was easy: remove the fridge and extension cord, and have a serious discussion about proper server room safety. The longer term fix was spending the last bits of the budget to add keyed switches to all of the outlets in the server room, ensuring no one could plug things in without going through the proper channels.