When I look back to my understanding of agile and lean software development, when I first came in contact with these topics years ago, I do realize that I missed it totally at that time. Now, some years later, I do think I have a better understanding about it to a point where I feel comfortable to share with you some ground knowledge that I came to learn later and that made me see these topics from a totally different perspective.
Why do some people say that software development is an activity in the domain of complex systems or a complex adaptive process?
At first, you have to have an overall understanding of what complexity means in software development and where it comes from. The whole concept is far more closely related to a new way of thinking on how to do things than to a formal model that you can apply. But, don’t get me wrong on that because I’m not telling that this has nothing to do with science. Complexity is the subject of a whole research field in science. The act of programming a piece of software is something that can be simple and formal also. Thus, the complexity comes from all the other elements that are part of this process nowadays and the new way of thinking I’m referring to is mainly related on how to best handle all these interactions in order to extract optimal results out of them.
What I’ll share here is mostly empirical knowledge acquired during my journey – something based on my previous experiences. That said, let’s jump to the main characteristics of a complex system and understand how it influenced the current approaches we use nowadays in software development.
1 – Large number of inter-related elements
As far as I understand, a system is a set of inter-related elements that work together and depends on each other; no matter if simple or complex, every system will be composed this way. Complex systems have a larger number of inter-related elements when compared to simple systems. Also, complex
As well as the human body, modern software development is composed of multiple systems that only make sense when working together and in balance. The market, customer, company, stakeholders, managers, leadership, developers, office space, computer, weather, and many more components are parts in this system. It might sound strange to consider the weather or the office space as part of the system right? But in fact, it’s not. The layout of the office or the weather conditions can also play a decisive role in how effectively you are to navigate through a complex system. A decision to be made on a really hot day on an uncomfortable place can be influenced by such factors. There’s a lot of research proving that, and this is something that rarely we take into account, right?
When I see software development approaches insisting on treating people better, providing better conditions for them to work, create better work environments, I immediately connect such efforts to the fact that they’re as well part of the complex system that is supposed to build quality software and play an important role on final results.
2 – Non-linear
You might remember from your math lessons what linearity means but anyways I’d like to give a quick and dirty explanation. Linearity is an observed property of a relationship; you have to have at least two elements interacting with each other – usually, each element provide an input – in order to observe linearity. When measuring the combined result of this interaction, a direct, constant and predictable proportionality can be observed. The result of the interactions between these elements can be represented by a straight line on a graph. Thus, linearity is the idea that combining the input of two elements will yield the sum of the respective output . Increasing or decreasing one of the elements n times will result in also increasing or decreasing the result n times. Now that we understand what linearity is, is easier to understand what is non-linearity. Non-linear relationships are unpredictable and increasing or decreasing one of the elements n times will not affect the result in the same proportion. Have you heard about the Butterfly Effect? This is a very good example of non-linearity.
Modern software development is non-linear because even a minor change can product disproportionated consequences. I bet you have heard that a small bug caused was the cause of a big disaster, right? Did you remember what happened with NASA’s Mars climate orbiter that caused a $ 655 million machine to be useless due to a conversion issue? The size of the bug/issue is not even closely similar to the size of the impact that it caused. In day-to-day software development we made this same mistakes when we ignore for example the economics behind a decision to implement first a feature A instead of feature B, or when we ignore the impact that having a bad workplace can cause to your company’s results or even when we ignore the fact that not testing your software before shipping it to production is not a good idea nowadays if you want to iterate fast and build something sustainable by managing the technical debt you create and making things easier and safer to change later.
]? The size of the bug/issue is not even closely similar to the size of the impact that it caused. In day-to-day software development we made this same mistakes when we ignore for example the economics behind a decision to implement first a feature A instead of feature B, or when we ignore the impact that having a bad workplace can cause to your company’s results or even when we ignore the fact that not testing your software before shipping it to production is not a good idea nowadays if you want to iterate fast and build something sustainable by managing the technical debt you create and making things easier and safer to change later.
3 – Dynamic
The capacity of staying immutable is not something that appeals to complex systems because they change constantly and the outcomes cannot be predicted accurately. Usually, the outcome generated by a complex system is greater than the sum of its parts. In such systems, problems don’t have a unique solution. The challenge is to find one that can possibly solve temporarily the problem until the circumstances make it change again, forcing you to find a different solution for the “same problem”. Well, not exactly the same problem, although most people still believe it is. This phenomenon is called “emergence”. You might have heard already about evolutionary design right?
To understand better what dynamic means for complex systems, lets take a look in the car and the forest example. Imagine a car. Now imagine an experienced car engineer. Do you think that this engineer is capable of disassembling this car and reassemble it in a way that it stays exactly as it was initially? I do think so. Now think about the forest and all the elements that interact with it, such as: the weather, the wind, the animals – including us -, and everything else that affects the forest. Do you know any engineer capable of disassembling and assembling a forest? I don’t believe so. It is impossible to define the forest exclusively as the sum of its parts because forests are complex systems that constantly change and have relationship with lots os external systems that are also complex systems – winds, for instance -.
Modern software development is also dynamic. There is a principle in modern software development that says that the software design should emerge and not be defined upfront. Any clues why? Exactly! Is because while the circumstances changes – business scenario, user needs – the software design will also change in response to this “emergent” circumstance.
The best way to deal with this characteristic is to use an experimental approach that can be able to fast respond to emergent changes in the environment. In modern software development, we don’t impose solutions. Instead, we experiment in order to facilitate the path forward to be revealed. Thus, experimentation followed by sensing and concluded with a response is fundamental nowadays in software development companies.
4 – Evolutionary characteristics (co-evolution, adaptiveness)
Complex systems have a past that heavily influence the present. The decisions and happenings in the past shaped what the system is now, and this is something that cannot be reversed. This kind of systems evolves in response to environmental changes. The extent and depth of these changes are what makes the system reacts and change. This is a characteristic of evolutionary systems. Systems’ components and external elements evolve together on a series of small changes that leads to a certain momentary state.
Modern software development isn’t different. It’s heavily based on the premise that what we do should be a response to an external change. If the business context change, we should change. If the user needs change, we should change. If the market change, we should change. If the state of the art in technology change, we should change. This is something we know nowadays as “Adapt or die”.
We have seen many cases of big corporations that just disappeared because they were not able to change or adapt to modern times. Also, we constantly hear about the benefits of a start-up structure that due to its flexibility can easily change directions and explore quicker new opportunities or trends. In software development, we transitioned from the idea of specialists to the idea of generalists landing on a middle ground ofter referred to as “cross-functional” professionals. Such professionals have to expand their knowledge to other areas not directly connected to their specialty.
5 – Uncertainty
When you analyse a complex system by looking at its past, it’s somehow easy to reach the conclusion that you can predict how future events would be or that there’s a clearly defined co-relation between cause and effect that you can use for predicting future outcomes. Don’t fall into this pitfall. If you’re thinking like that, you’re forgetting that a complex system is unpredictable due to the constantly changing environments to which they are interacting with.
Modern software development is strongly dependant on the environment where it takes place. Oftentimes we refer to this environment as a “context”. I do believe that every context is different and modern software development practices should adapt to such specific contexts in order to bring any results. I’ve heard many stories on how people tried to use a certain approach that worked very well for company X but for their company it simply didn’t work. That’s because they missed to consider the context. Every process, practice, framework is created based on a certain context. If you don’t compare this context to yours when applying these ideas, you’ll certainly have trouble to make them work properly.
6 – Unordered nature
Unlike ordered systems, where cause and effect can be predicted with reasonable accuracy, complex systems can’t have such predictability. The complex interactions between its elements make cause and effect super hard
Modern software development evolved into a scenario where it became unordered. Apart from very small cases where studies have shown successfully anticipation of customer behaviours, most of the time companies they to assess the current context and experiment something to see if this something is the answer that best fits the environmental needs at that particular time. Modern approaches to software development became much more adaptive and responsive to external changes nowadays.
Studying and understanding what a complex system is and how we can work with it could help us understand why do we have all these different ways of developing products and why they are framed in certain ways that sometimes seems to be so uncontrolled, chaotic. The point that all these approaches are trying to make is to find a way to enrich interactions between system’s elements, best react to environmental changes as fast as possible, and increase the chances to find the right answer for that particular moment with less cost. Acknowledge that the work we do is complex and you’ll be on the right track. If you try to solve nowadays problems with old tools, that I’d say that you should urgently reconsider your options.
An open question for further discussion
If producing software is a complex adaptive task and in order to deal with such tasks we need to apply experimental and very flexible methods in order to find the most appropriate answer, how suitable are these large scale approaches to such scenarios? Aren’t they too prescriptive to survive in such environments or trying to control an environment that cannot be controlled? Aren’t they trying to create order or bring linearity to a system that is by nature non-ordered and non-linear?
Let me know what you think in the comments below.