In physics, when you try to measure certain properties of an object with increasing accuracy, other properties of the object must neccesarily become less well known. Wikipedia tells it the way it is:
The uncertainty principle is a fundamental concept in quantum mechanics. It states that there is a limit to the precision with which certain pairs of physical properties, such as position and momentum, can be simultaneously known. In other words, the more accurately one property is measured, the less accurately the other property can be known.
So, in physics, there are the complimentary variables of Position and Momentum, which can only be measured to a certain extent. To be more precise, both can in fact be measured at the same time, but there is an upper bound to the total accuracy with which both can be understood. Exceed that boundary by measuring Position to an extreme level of accuracy, and you will lose insight into your Momentum. Conversely, knowing the Momentum to an excessive precision means you have no idea where the object itself exists. We go fast, but where the hell are we?
In managing software development
It struck me that there are parallels to be drawn to the experience of developing software or data products. A number of obvious complimentary variable pairs come to mind immediately: Visibility and Velocity, Team cohesion and Control, Predictability and Innovation, ...
The more management seeks to be precisely informed, demanding visibility into every step of the process, the more uncertain the velocity of the team becomes. The more management imposes rigid structure and processes, the more uncertain the emergence of innovation becomes. Et cetera. Connecting this is the proverb (or rather, truism) that the act of excessive control through measurement degrades its own value, or Goodhart's Law.
These observations seem to be valid both in terms of logics and in terms of lived experience, as any data or software engineer can tell you. Yet, workplace dysfunction exists, at scale. This begs the following questions:
- Why do organizations insist on tracking every morsel of work through JIRA-like ticketing systems despite diminishing returns?
- What drives the seemingly inevitable accretion of process overhead and bureaucracy?
- Why do these patterns replicate themselves across industries and decades, despite clear evidence of their dysfunction?
- Through what mysterious force do well-meaning organisations actively repel the very talent they need to succeed?
- Why can't we have nice things?
Now what?
Let's first examine the causative agent, the reason why excessive measurement paradoxically reduces actual velocity, cohesion, innovation and so on. Doing so will reveal that this is, in fact, not the fault of any single group of actors in the system. Framing this as 'management versus developers', or 'dumb corporates versus data artisans' is not the intention here. Instead, we wish to arrive at a more holistic answer to all the WHYs above, to see that systems in which software development takes place have a natural tendency to get stuck in a suboptimal state.
Understanding why this happens will then allow us to formulate defence mechanisms, or at least, maintain some level of professional sanity as either a PM, Agile practitioner, business analyst, individual contributor, or whatever other data-adjacent role you might have.
Information fog
Recently, HackerNews discussed that "Shipping is a social construct", and that there is an extremely vague distinction between shipping actual software and engaging in corporate politics to maintain the illusion that working software has, in fact, been delivered. This points to the existence of a reality distortion field, an information fog, that is pervasive in the industry and naturally emerges as a consequence of the uncertainty principle.
Consider the following manifestations, all which will probably resonate with anyone working in technology right now:
Status reports that are technically accurate without contributing to getting things done.
Hyper-visibility into 'has the widget been added to the dashboard yes/no' will create a climate in which no-one pauses to examine if the dashboard actually sparks joy, and if anyone cares about the widget at all.
Estimates padded with "buffer time" that's never explicitly labeled as such.
Strict control over time spent will create an atmosphere in which everyone secretly pads estimates to cover their asses, just in case. You don't even need to haggle with people over their estimates, the simple fact of putting every estimate under a microscope will cause ass-covering behavior. This stricter control solves nothing, the fat-tailed distribution of how long something takes means the tickets can STILL dramatically exceed their estimates.
Wanting the latest and greatest in AI innovation without changing the way you operate.
You can't simply go all-in on what LinkedIn tells you the world is doing, whilst refusing to address the years of technical debt in your data layer and your existing frontend. Or whilst treating innovation as just another IT cost center task, because this is how you have always done IT. You want to innovate, by neccessity, you must let go of how things were done in the past. Failure to recognize this leads to cargo cult innovation, where people don't blink at the procurement department telling you to fork over $25k a month for Azure OpenAI 'provisioned throughput instances'... to serve 100 daily API requests, that would have cost you 3 cent each with the default pay-per-use model.
My intuition here is that this fog isn't something arising from malicious deception. People are probably, on average, decent human beings. The information fog just shows up whenever incentives are weird and people hyper-focus on one thing. I refuse to believe that people are too inherently stupid to multiply 3 cents by a 100 daily requests by 30 days; there must be some information fog clouding their perceptions, disallowing a balanced view into the reality.
So. Excessive measurement or focus causes information fog. I even think that The System, once afflicted with the disease, literally cannot function without this fog -- it's the lubricant or the defense mechanism that allows people to keep functioning under excessive measurement pressure. How else can you avoid all critical thinking beyond 'procurement told us we need provisioned capacity because as a rule we buy highly available services and if I follow the rules I did a good job'. At a 278x cost increase.
Convergent devolution
The information fog initially benefits everyone. Employees gain breathing room to work effectively; middle managers maintain relevance and dignity by "translating" between layers; Executives preserve the comforting illusion of control; Organizations avoid paralysis from staring into the abyss of too much unfiltered truth about their SAP implementation. But.
- The measurement paradox intensifies -- as visibility (velocity, innovation, whatever) decreases due to the information fog, management responds by adding more measurement tools, which generates even less clarity and even more fog.
- Talent suffocation -- The best people, the ones who need autonomy like plants need sunlight, either mentally check out or physically bounce. What remains is people who thrive in bureaucratic hellscapes. The Gervais Principle in action.
- Increaing reality distortion -- The gap between reported progress and actual progress widens until the system literally cannot comprehend its own state. Teams are simultaneously "on track" and completely messed up.
- Repeat -- Where to now? Micromanagement will intensify until outcomes improve, so back to #1!
Eventually, but consistently, systems suffering from information fog converge upon their own destruction by entering a vicious, self-reinforcing cycle of overmeasurement, leading to ever higher uncertainty in whatever is not being measured. Once on the path of devolution, everything converges to misery. Star talent leaves due to the process of evaporative cooling (again a delicious analogy to physics where fast atoms escape out of a liquid, cooling off the remaining mass). Managers feel the wind of change and use the accumulated perceptions of shipped software to escape before the house of cards collapses. Organisations obsessively restructure every 9-18 months to maintain the illusion of control and forward motion.
Defense against the dark arts
Individuals and interactions over processes and tools. Is this what the creators of the Agile Manifesto had in mind? Fuck scheduled refinements, grooming, planning, retrospectives. You can't institutionalize and protocolize psychological safety. Putting 12 people in a room or in a call isn't a "good place to discuss what is really going on in a team". It's the tragedy of the commons, destroying all value there is in having a team by forcing people to open up on a schedule, thus only reinforcing the artificiality of the modern workplace and destroying any chance there was for genuine connection. Analogous lines of reasoning hold for the other ceremonial processes that exist. Honestly assess the last PI planning event you attended. Did you enjoy the experience, or was it the institutional equivalent of a Dementor attack?
The defense: Create your own micro-rituals with the people who actually get shit done. A quick coffee, a Slack channel without managers, a walk around the block. The real team isn't the org chart - it's the shadow network of people who know how to make things happen despite the system. And don't grow your teams too big lest you create a monster.
Working software over comprehensive documentation. Fuck sagas, epics, user stories, checklists, test plans, test executions. Work items simply move from the 'to do' column over to the 'done' column. That's it. Everyone has the right to change the software if they think they can improve it. If they break the software, they have the obligation to fix it.
The defense: Build a rapid feedback loop that makes change safe. Automated tests that actually tell you something useful, not 10,000 green checkmarks that validate nothing. Deploy small changes constantly and make rollbacks trivial. Your agility isn't measured by how many tickets you close but by how quickly you can pivot when shit inevitably hits the fan. Talk to eachother. Call out the architecture astronauts, the drive-by coders, the overconfident newbies, the slackers, with love and empathy. Everyone is delighted when things eventually work. That is something to rally around.
Customer collaboration over contract negotiation. Stop pretending that requirements document you spent 3 weeks writing will capture what users are actually going to do. Requirements are wishful thinking from people who haven't used your product in months. The real requirements emerge from watching real humans struggle with your garbage UI and nonsensical workflows. You don't need to add an AI chatbox to let a user turn a table into a bar chart because someone higher up told you to add an AI chatbox to let a user turn a table into a bar chart. People can still click buttons.
The defense: Put actual humans in front of your product regularly. Not formal usability tests where everyone's on their best behavior, but casual "hey, can you try this thing while I watch you get frustrated?" sessions. The grimace on someone's face tells you more than a 20-page PRD ever will.
Responding to change over following a plan. Plans are fantasy fiction written by people who think they can predict the future. Your roadmap is a collective hallucination that everyone pretends to believe in because admitting uncertainty is terrifying to organizations built on the illusion of control.
The defense: Continuously make sure there is a shared vision, even if it changes. People can accept change if they feel involved. The alternative is continuously "aligning people on your plans", which is directive and makes you look like a fool once the plan collides with reality. When the inevitable "everything is different now" moment arrives, you won't be the team having an existential crisis because your six-month plan just evaporated.
In closing
The people who wrote the Agile Manifesto were right all along.