fail

The Projects That Failed

At the skankworks.net we’ve worked as independents in the IT sector since thirty years. We’ve been involved in many great projects over that time and many readers will be using software that we have worked on. Naturally, we’ve also seen our share of project failures.

Up until about five years ago, that is. When we first encountered an Agile project. Then things changed. For the worse. Now the only thing we see are project failures.

Over the last five years it has been our misfortune to be involved almost exclusively with projects in which the Agile methodology was used, if such a hodge-podge of wooly-thinking and inexperience could be called a method. They’ve all failed.

Before you leap to any TL/DR conclusions, let’s clarify this. We were not developing these failed projects, we work in a QA capacity, testing and reviewing product development for our clients. For most of these years we have been sitting around with very little to do, paid to sit at our clients’ desks waiting for agile software deliveries that never arrive.

Defective by Design

Defect reports are generally dismissed by Agile teams, who seem happy to deliver software that they know does not work. Agile test cases are designed such that they are impossible to fail. We’ve seen test cases, that the agile team intends to submit to regulators, that would require the test team to solve the Halting Problem before they could judge if the software works or not.

On our most recent project the development team created a “secret branch” in the software repository unknown to everybody else from the test team to the FDA. But that was the branch they delivered the builds from. It was what the SPL wanted.

It’s not the only time we’ve witnessed agile teams attempting to bypass software testing and QA by developing undocumented units “off-board” in order to hide them from the V&V team. Their subsequent “demo” consisting of a series of screenshots created in photoshop displayed sequentially as if it was a running system.

Think, Colin Powell at the UN waving his test tubes, or Netanyahu with his cartoon bomb. In such a world, why wouldn’t lowly software development teams do the same?

Deliver First, Develop Later

About the only way to convince agile developers that there is a defect in the code is to sit down next to them at their desks and demonstrate it to them on their own PCs, as if you were scolding an errant child. There are few exceptions to this. Mostly coming from junior developers who are still naive enough to believe that business needs working code to make a profit.

The young ones are often keen. They will want to fix their bugs, if the scrum master gives them permission that is. And there’s the rub. Often the scrummaster will not have the resources for bug-fixing and will demand that there aren’t any, creating his own reality when there very obviously are.

He – scrummasters are usually men – will allocate insufficient resources. Scrummasters aren’t meant to be project managers after all. Rework doesn’t fit well into Agile plans. Rework increases the backlog, when the scrummaster is supposed to be grooming it. There is no project manager around to sort things out. Thus, any excuse not to accept a defect will do. The precious sprint will go on.

The constant refrain of “continuous delivery” one hears from agile is never matched in practice. Most software deliveries we get from agile are untestable. In order to complete their story points agile developers will deliver half-finished units of code, which they insist on referring to as “stories”, that while “half complete” have no functionality that is testable. They are stories with no ending.

Next sprint, they invariably move on to something else, and deliver that half-complete as well.

Contracts Cancelled / Projects Canned

We’ve cancelled our contracts on two projects, one because we never had any work delivered for us to evaluate, another because the development firm would routinely ignore defect reports and fire members of the test team (externals) who insisted on raising them.

We don’t work this way. Nobody can.

Here is a brief overview of the projects, to give some sense of the scale of the waste Agile inculcates.

Hall of Shame

New eCommerce Front-End, Thin-client and web.
Planned: Budget, $15,000,000. Delivery Due: One Year
Actual: Spent, $45,000,000 Delivery: One year late
Customer: Investment Bank
Customer Response: Stopped using the new product after one month evaluation, renewed their license with the previous third-party supplier the new system was intended to replace.
Market Response: N/A

New eCommerce Front-End, Thin-client, mobile, and web. Version 2
Planned: Budget, $20,000,000. Delivery Due: Two years
Actual: Spent, $35,000,000 Delivery: Never delivered
Customer: Investment Bank
Customer Response: Abandoned the project during UAT, cancelled the contract one month before go-live. Project canned.
Market Response: N/A

System Self-Test
Planned: Budget, $15,000,000. Delivery Due: Three Years (at 30% effort)
Actual: Spent, $5,000,000 Delivery: Never delivered
Customer: Pharmaceutical manufacture
Customer Response: Internal project, abandoned after one year due to duplicaton of requirements. Project canned.
Market Response: N/A

Workflow Design GUI
Planned: Budget, $5,000,000. Delivery Due: One year
Actual: Spent, ~$6,500,000 Delivery: On time
Customer: Pharmaceutical manufacture
Customer Response: System rejected. Did not meet requirements, incompatible with customer API. Project canned.
Market Response: N/A

Medical device, front-end GUI
Planned: Budget, $6,000,000. Delivery Due: One year
Actual: Spent, ~$6,500,000. Delivery: On time with reduced feature set
Customer: Medical Device Reseller
Customer Response: Accepted.
Market Response: Three subsequent recalls from regulators due to potentially harmful software defects, cost to fix ~$1,000,000 each, not including losses incurred by customers due to downtime/lawsuits. Additonal $500,000 cost due to incomplete software development documentation submitted to regulators. Post-launch remediation costs thus far: $3,500,000, lawsuits pending.

Embedded Device Control System
Planned: Budget, $12,000,000. Delivery Due: Eigtheen months
Actual: Spent, $35,000,000 Delivery: Still in development three years beyond delivery date.
Customer: Medical Device Manufacture
Customer Response: 501(k) suspended at customer request, one pre-market warning and service pack issued to permanently disable some core functionality, contract renegotiation ongoing. Canning the project, being one of the options under discussion.
Market Response: N/A

Wastage

So that’s around $150 million dollars we’ve seen our corporate customers spend on Agile software over the last three years, and just one of them has any marketable software to show for it. Software that has got them into persistent trouble with the FDA and put people’s lives at risk. Software which they are probably going to get sued for. Hardly software that one could describe as “working”.

All of these projects were staffed mainly by externals and temporary hires, who are paid by the hour and who will lose their jobs and be put back on the unemployment lines the moment their part in the project is complete. There simply is no incentive for them to deliver on time, while delays and cost overruns are in their own best interests.

Regulators such as the FDA need to take note of this. Current manufacturing processes are creating an environment which rewards poor productivity. They need to update 21 CFR good manufacturing processes to include a ban on unreliable error-prone and counter-productive development methods such as agile and require the use of formal methods in regulated software.

This will of course increase the intial cost of development and have corporate executives and their immaculately attired lobbyists screaming the place down. Software engineers who can design and develop using formal methodologies are not much more expensive, but they are difficult to find.

The first thing the executives would have to do is hire some competent recruiters, and who are they going to get to do that for them?

Nevertheless, a software engineer that can, for example, write code in ADA from a Z-Notation specification is always going to be better than those trading under the ludicrous title of “solution architect” who get most of their algorithms from hurried searches on stackoverflow, the software engineering equivalent of the back of a cornflakes packet, cutting and pasting whatever looks like it might work.

On the other hand, well-designed, formally developed, tested, and documented software tends to get delivered on time far more frequently than Agile software and, more importantly, does what it is supposed to do.

But when you’ve just seen your agile project’s “solution architect” looking up on google what the word “idempotent” means, you know the project is going to fail. In fact, it’s already failed. You ask yourself, “What am I doing here?” and you start looking for another project to join.

Military Exception

There is one exception, not really a business project, but noteworthy.

On one of these project there was an intern, working as a developer through the summer as part of his university course. They gave him a project to “write an iPhone app”. Anything he liked, and get it on the app store. This he did, not using Agile. He published the app a week or two before he left the firm to go do his university exams, and held a little celebration because his app had been downloaded twenty times in the first week and had earned him a few much-needed extra dollars.

They hired him as a full-time member of staff under the graduate-entry program six months later. Upon his return we asked him, “How’s your app doing? Did you get anymore downloads?”

“I took it down”
“Why?”
“After three months I had nearly five thousand downloads at a $1 each, but Apple wouldn’t pay it out. They said that downloads of my app had exceeded the usage limits and that my account had been blocked until I paid to upgrade it.
“How much would that cost?”
“Five thousand dollars”
“I see.”

There’s a young man who has learned a valuable lesson about the software industry, on his very first project.

He’s now thinking about quititng IT and joining the military.

Medical Device Advice

That’s Agile for you. Every project, no matter if the lives of the end-users or the health of the economy depends on it, is treated as no more than an app, designed only to ensure that the developers get something in their pay packets this month, their leaders a bonus cheque this quarter.

Trust us on this. We’ve seen it happening.

Woefully inexperienced developers, who have never produced anything more complex than a web app designed to sell google ads, put onto writing code that determines the results of chemical and biological analysis of life’s processes. Where displaying NEG or POS on-screen may well mean the difference between life and death for the patients who are being diagnosed by such shabbily produced and flawed, perhaps fatally flawed, software.

If your doctor says she needs to analyze your precious bodily fluids, ask her how the analysis will be done. If she says a medical device, we advise you to seek a second opinion – and not one that derives from the same kind of device. We advise this because there are many former software developers of our acquaintance who have quit their jobs while working on such devices and would not allow them to be used to diagnose their own maladies.

That’s another, all too predictable, consequence of Agile for you to mull: So many highly skilled developers with a sense of integrity and personal responsibility that no longer desire to work in software development.

If your business has adopted a methodology that is driving away the very people you need to hire, then we’ve only one word for you: Fail.