Software is a discovery process

The development of software is a relatively new discipline, and one that is constantly changing. The last couple of years have been an earthquake with the arrival of AI, but even before that, the field changed beyond recognition from one decade to the next.

Still, there’s something that I think is part of its essence, and it hasn’t changed. It has been widely recognised for the last quarter of a century, since the Agile Manifesto.

It is borderline impossible to know exactly what functionality a software project needs until it’s done.

And software is never really done. Either it’s in use and needs to be endlessly adapted and tweaked, or it’s abandoned. Twenty years ago, some software had to be frozen and packaged to ship in boxes, but even then, the next version was already being prepared.

I should’ve taken that left turn at Albuquerque!

It is probably because software is so malleable and abstract that it needs constant thought to turn an extremely fuzzy vision into a concrete one. After all, imagining software in enough detail is exactly the same as writing it1. That still holds now that models write most of the code: describing what we want precisely enough for a model to build it is still development, just at a higher level than source code.

This is elusive work. In our heads, we all feel we know how it should behave, but once we start, each detail needs to be carefully thought out and critically analysed. Many things don’t work in practice.

This makes development a process of discovery: an exploration of a vast space of possibilities towards something concrete, one step at a time, pivoting often.

This discovery happens on many fronts: what the users of the software really do, what they need and what they want. What the alternatives are doing. How the software actually behaves when running, how it copes with failures nobody anticipated, and how the failures we did anticipate play out in real life.

All this discovery should feed continuous learning and adjustment, which keeps the cycle going. To me, this is the main element in software development. Keep learning. About your code, about your tools, about your processes, about your team, about your industry… And use that knowledge to improve. This is not a natural instinct for most people. Developers are forced to learn it early, since every bug is a lesson, but it never gets easy, and it frustrates many. Self-reflection is tough!

That’s why learning is best done with others. It’s far easier to spot someone else’s blind spots than our own. Collaboration and the debate of ideas make better products, and a shared sense of teamwork and accountability helps the improvements stick.

If you want to go fast, go alone. If you want to go far, go together2.

Going together is easily forgotten in the age of AI, when we have superpowers and can command legions of agents to write software for us. But agents don’t bring experience of their own. You can ask them to critique your ideas, but they haven’t watched a user struggle with a confusing screen, been woken up by an outage, or lived with the consequences of a decision, and they tend to go along with whoever is prompting them. Their bias towards action can make them less likely to suggest removing features or stopping work that is going in the wrong direction. A good, varied team with different skills and views will challenge the project and keep real usage in mind.

Teamwork is tough, but fun

Software development will keep changing, now faster than ever, but I believe its essence is still the same3. We don’t know exactly what to build until we’ve built it, and it’s never really finished. Building software is a discovery: of what users do and need, of how the system behaves once it meets the real world, and of what the alternatives are doing. That discovery lets us learn and improve, and that is easier with others. Other people see what we miss and bring opinions and viewpoints that complement our own.

AI may make producing code faster and cheaper than ever, but imagining software in enough detail is still the same as developing it. In Fred Brooks’ terms, AI removes much of the accidental complexity, but the essential complexity remains. The real hard part was never typing the code; it has always been working out what to build and solving a messy, real-world problem. That takes exploration and discovery, done together, to get somewhere worth going.

  1. Jack W. Reeves made this argument in 1992 in his essay “What Is Software Design?” ↩︎
  2. And if you want to go very fast, but perhaps in the wrong direction, go with agents. ↩︎
  3. In “No Silver Bullet” (1986), Fred Brooks distinguishes between essential and accidental complexity. Forty years later, it remains one of the most relevant pieces ever written for understanding software development. ↩︎

Leave a Reply