As the Client Engagement Manager for Bit-Wizards, I have spent a lot of my career answering the same question in many different forms:
• Why didn’t you as experts know this new wrinkle was going to be a problem ahead of time?
• Why is this software project over-budget and over the deadline?
• Why does software development cost so much?
• If this feature is so simple, why is that similar feature so difficult?
• If an auto mechanic can provide and stick to an estimate, why can’t software developers?
I think these questions are all reflections of an unstated assumption:
Software should be a consumer good – affordable, repeatable and interchangeable.
Get your integration products on aisle six and your development expertise can be picked up on aisle nine.
It’s easy to look at daily life and create two categories of objects:
- Individual purchases are things like cars, groceries, phones or clothing.
- Institutional purchases are things like buildings, bridges, space shuttles or power plants.
Which of these does software development fall under? The answer is both – depending on what you mean.
- Software licenses can be individual purchases – you buy a Mac OS X license with your home computer, and express it as ‘I have OS X’. Actually, what you have purchased is an OS X license – one use of the software.
- Software development is always an institutional purchase – your company hires a construction firm to build the new plant and similarly hires a development firm to build the new software system. Your company owns the software outright for unlimited uses.
To explore this difference further, let’s take that last question because it’s so common – comparing software development to hiring an auto mechanic. What is the real nature of software development vs. that of auto repair?
What is software development really?
Software development is the process of creating a never-before-seen software system, one that is uniquely tailored to the purchasing company. During that process, software developers stitch together a complex tapestry of human desires, business processes, marketing imperatives and computer technology. The result is a new creation that will almost always be used by different groups of people than those that created it.
There are a lot of steps to the software development process: estimation, requirements gathering, visual design, system integration, writing code, testing, revision, and launch, to name a few. Only for the simplest of software projects can one person be intimately involved with every one of these steps – complexity like this requires many different disciplines to come together to make the whole. The process of creation isn’t the product, though, and focusing exclusively on the process misses the point of the activity. The point was to have a new software system come out of the process, something that took that mix of desires, business logic, marketing, and technology and made them into a single business asset.
The steps in a software development process only FEEL like an MC Escher staircase.
Complexity is the Key Word Here
Software development is a strategy to manage complexity. Humans created software systems to hold complexity for them – intricate processes, detailed information, massive storage, crossing long distances. This is because humans are not good at all these things on our own. Having a computer take the complexity out of record keeping, or hotel room availability, or market trend analysis, frees humans up to do things we’re good at instead of focusing on the mechanical minutia.
Unfortunately, there is a hidden cost to so many people hiding their own complexity inside software systems. A person performing one role can create a large number of variations even they can’t keep full track of without lots of notes. Combining many roles and many people into one system results in so many layers that no one person nor team of people can keep track of every detail and moving part.
Over time, software development has gained more and more reliability and reusability through commoditization of common components. Often, however, those common components accomplish very different goals within the system and achieve very different outcomes. Both the commodity components and the overall software systems are often very brittle, requiring exactly the same environment to run or exactly the same type of input to produce usable results. Just imagine trying to use a smartphone to file your taxes online, and you’ll see what I mean. Even when the system itself doesn’t fully break down, the process it’s modeling is completely unusable because the screen size is smaller. The limitations and boundaries of software systems remain obscure and counterintuitive more often than not.
Whether software development teams are internal to an organization or brought in from external firms, they are being paid for their expertise in the process of development, using some specific tools, to work toward a specific goal. The process of hiring developers hinges on their expertise in a certain area relevant to the project at hand. In most cases, internal hiring or externally contracted, the developers have only their time to sell; however, their expertise is built over time, and a particular developer’s skill set is often relatively rare.
Taking the road less traveled by isn’t remarkable in software development – all software development is trailblazing over unmarked territory.
And What is Auto Repair?
At the risk of stating the obvious, cars are mechanical objects that rapidly turn their wheels on roads to move themselves and their contents from place to place. The history of the automobile is one of mass production and minor refinement over more than a century, yet the basic function is identical: wheels on the road. Over time, the non-essentials functions of a car have been refined, increasing things like safety, comfort and reliability through millions of tiny changes. Sitting side by side with a horse and buggy, any modern car looks entirely unrelated to the buggy, when, in fact, they are extremely similar in complexity.
When a car needs repair, mass production makes almost every problem a relative breeze. Nearly any worn out part can just be replaced with an identical component. Due to the long history of the evolution of a small number of components, and the obvious limitations of use, almost no entirely new problems are discovered in any car. While a car is well protected against water and rain, almost nobody tries to build an oceangoing car – most people could agree that driving a car into the sea would result in unpredictable damage and risk.
All this means that the auto repair profession works within a fairly well-defined set of rules and with the benefit of a lot of documentation about every model and every component. Almost no one decides that commercially available transmissions just weren’t designed properly for their specific needs and that they need to create their own. Car trouble is due to worn out parts and is fixed by replacing them. Troubleshooting can take a few hours while the time required to replace the most major components is measured in at most a handful of days (watch Fast and Loud, Counting Cars, or any other semi-accurate auto show on TV to find out).
To give an estimate, an auto repair estimator has a large body of prior work to draw on. This work history documents a lot of detail about the common problems that arise in the model of car in question, from millions of nearly identical repair jobs performed on this model car in the past by many different mechanics. The time to complete similar repair jobs have been averaged into reliable time estimate ranges. The estimator also has costs for every part available, because the parts are interchangeable commodities. Between these two, the vast majority of car repair jobs can be estimated fairly accurately – fortunately for the individual consumer!
Taken together, this means that the mechanic is chosen primarily on the basis of budget and convenience. The mechanic is paid based volume of repairs performed, given that typical repairs have only a small variability in cost and profitability. Additionally, no repair will increase the value of the car, repairs will only restore it to the value it had before the problem arose. The repair process itself is a commodity, with nearly identical results available from any number of providers.
But isn’t my CMS platform the "base model" and custom software development just adds dealer options?
Often the auto repair analogy is extended to the particular software platform used in the development. For Bit-Wizards, this is usually Kentico CMS, and the question is “Isn’t Bit-Wizards the expert in Kentico?” Let’s examine this in light of the comparison of software development expertise to automotive commodities above.
Software platforms like Kentico CMS are often called ‘engines.' It’s rare for consumers to choose their car based on the engine alone; instead they are buying an off the lot product that includes an engine. However, businesses purchasing software development have needs that can’t be satisfied by off the lot vehicles. They need something that meets specific requirements, like James Bond’s submarine car, or the Chitty Chitty Bang Bang flying car.
In these situations, the software developer’s expertise with the ‘engine’ of Kentico helps qualify them for the work, but their expertise is still what’s being purchased. They know how to mount the engine so that it's safe while the sub-car is underwater, but they probably haven’t made a submarine semi-truck before (you can only make so many submarine-cars using luxury cars, before the bad guys expect you to drive it into the water to escape. Maybe they won’t suspect that you’ll drive that 18-wheeler on the sea floor, though!)
Being an expert in Kentico CMS means that the development team has used it in many projects before, but not that they’ve modified every component or used it in every kind of project. There are a lot of nuances and modifications necessary when ‘mounting’ the Kentico CMS in a new environment, whether it’s one with custom marketing automation requirements, or needs to get user data from legacy systems, or needs to provide Amazon-like e-commerce flexibility.
Invariably, customers aren’t buying Kentico CMS because they want it out of the box – in fact, it’s pretty useless without the rest of the system assembled around it. Sometimes that’s simple stuff, like a templated design and some content. Other times, Kentico is the engine around which a highly tuned drag racer has been built, and every tweak can have unexpected ramifications on the system’s performance – or can simply not fit into the available space without extra modification.
The requirements document did not include airboat pontoons.
OK, OK. What is Software Development Like?
Since software development is not like a commodity service, what is it like? Let’s try some analogies in the world of consulting professions:
- Software development is like civil engineering consulting. Before a bridge can be replaced, an extensive and expensive study must be conducted. The study will determine current and future usage, review alternate ideas, explore in depth whether the location will support the proposed new bridge ideas, and take exacting measurements. And then there’s still a good chance that more supports will be required than expected, or materials will be delayed, or political needs will change midway through, and the project will still run over budget and deadline.
- Software development is like business growth consulting. The growth consultant has experience with helping companies grow in many situations, but he can’t guarantee the desired results. He will adapt his experience to the client company’s situation, but he may not resonate with the employees’ particular experience and, therefore, may not give the most effective advice for that company. He still charges his time to the company regardless of the eventual results.
- As much as it hurts to say, software development is like legal representation. The lawyer gives his best counsel, but he might be blindsided by some arcane precedent in the case law. He endeavors to place his client in the best possible legal position, but he can’t erase the client’s past actions.
- Software development is like hospital medicine. Many different disciplines need to come together to address one complex case. General practitioners must coordinate with surgeons, internists, anesthesiologists, pharmacists, psychologists and many others to provide the best care possible. The hospital won’t know ahead of time exactly which disciplines are required, nor what the costs could be in many cases.
Outcome not guaranteed, but money sure helps tip the scales.
Be honest here. What’s the real problem?
There’s a lot of literature both inside and outside the software industry written about this very problem: why is software development so hard? There are many conclusions out there, but my best summary comes down to the two themes in this post.
- There is huge complexity in software development
- There are significant perception problems due to comparing software development to anything else.
Really, software development is unlike anything else in human history. It’s a process where many people come together to create something none of the participants can fully grasp – in some sense, every software system is a new mind bigger than the capabilities of its creators. The systems are not limited by the types of physical boundaries we’re used to seeing. The best practitioners in the world, the NASA’s, Google’s, Oracle’s and others have built many best practices, but the potential is so huge that I think the precision and reliability we’re used to in cars is a long way off for software development.
And it’s all coming faster and faster.