I’m a dad to 3 kids, and lucky to share my life with a wife I love. We share a joint bank account for family expenses, as well as retaining our own accounts for personal spending. I spend from my account as I please, and so does she. We both spend from our shared joint account – this is where we collaborate together. I don’t need to ask her permission to buy the kids an ice-cream. But I wouldn’t book a holiday or buy a car without up-front discussion, planning, and agreement. Other spending is significant – but routine – like the electricity and water bill.
Perhaps you are a engineer who works on an inner source product. You share a joint bank account code repository with other teams, as well as retaining your own deployment repository to configure your own infrastructure. You change your deployment repository as you please, as other teams do theirs. All teams change the shared code repository – this is where you collaborate together. You don’t need to ask permission to – well, to what? What’s the same as buying an ice-cream? What’s the same as booking a holiday? What about the electricity bill?
Treating All Changes the Same
If we ignore our instinct that buying an ice-cream is different to booking a holiday, we can choose to treat all changes the same. “Buying anything from the joint account Rob? Ask me first!” says my wife. Changing anything in the shared code repository? Here’s the required steps. “The” steps? Yes – the same steps and approvals for all changes.
The result is either:
- It works brilliantly. Turns out that all your changes are similar enough to benefit from treating them all the same. And having 1 process is simpler than having 2 or 3 so there is less confusion. OR:
- Small, simple or routine changes are slow and users complain about the effort and bureaucracy required. You are treating all changes as significant and complex. Such changes require more upfront effort, planning, and agreement. You apply that diligence to every change – unnecessary for many of them. A backlog of changes is building up behind a planning or approval bottleneck. OR:
- Large, complex or risky changes are chaotic and encounter unexpected delays or operational failures. Your stakeholders complain about poor visibility and control. You are treating all changes as being small and simple. The lack of planning and agreement for complex change is causing delivery problems.
If you are at outcome 1 – fantastic, stop reading. However, the rest of us need to find a way to tell the difference between an ice-cream and a holiday.
The Triage Process
Lets stop treating all changes the same. Instead, we categorise changes into different types, and apply different processes to each type. But what categories? And how do we decide which category a change belongs to?
Triage (verb) : conduct a preliminary assessment of (patients or casualties) in order to determine the urgency of their need for treatment and the nature of treatment required.
Triage is a term often used to describe the process of categorising a change. The term comes from the medical profession, where a doctor must quickly assess the urgency and nature of a patient’s need for treatment. The doctor must do this quickly, and with limited information. The doctor must also be able to change their assessment as more information becomes available.
Note how it:
- requires expert judgement.
- is quick.
- adjusts over time as more becomes known.
I limit triage of a spend from our joint account to trusted experts – either my wife or I, not the kids. We both use a quick judgement call to decide if we need to discuss the spend. We reflect on our spending habits and adjust over time. “Rob, it looks like you are buying the kids an ice-cream every day. That’s too much – let’s make it once a week.”
For an inner source product, the experts are the maintainers – they’re the parents in this analogy. Start discussing with them the best categories of change, and how to quickly triage new changes into a category.
Major vs Minor
What categories of change do you need? The first step forward from treating all changes the same is to have 2 types:
- Major changes are significant, complex or risky.
- Minor changes are small or simple.
This simple categorisation is enough to make a big difference. For example:
- Triage assigns a change as major, minor, or rejected. Rejected simply means the change is unsuitable.
- Minor changes get submitted as a pull request and require one maintainer approval (e.g. the maintainer in the implementing team).
- Major changes require at least one maintainer in each division agree with a written design proposal. Implementation pull requests also require at least one maintainer approval in each division.
This example shows how the process for major and minor changes can be different. The goal is to deliver simple changes efficiently and focus the cross-divisional collaboration effort where it’s most valuable.
Routine Changes
Teams often build forward from a simple major/minor separation by examining their routine changes. A change that you do regularly is faster if you create a new change category and streamlined process for it. For regular changes, this is often worth the effort to do so.
For example, a common routine change is updating code library dependencies either pro-actively or to resolve a recently discovered security vulnerability. You might automate this using dependabot, and agree a lightweight process for reviewing and merging the automated pull requests.
My wife and I look after routine spend from our joint account via sponsorship. I’m responsible for the electricity & insurance bills. My wife is responsible for the water & council tax bills. We set up special processes for these bills – using pre-agreed direct debits. We only discuss these bills when they change significantly or are likely to cause a problem. We each review the bills we sponsor periodically to check we’re getting the best deal.
Emergency Changes
Emergency change to resolve an operational incident is another area that often requires a separate change category and process. These are rare, but when required are urgent, executed under pressure and outside normal working hours without the availability of the normal stakeholders.
The usual setup is to pre-agree an emergency process which allows the on-call team to make changes without the normal prior approvals. However, the on-call team must notify the normal stakeholders as soon as possible, and work with them to retrospectively review, adjust, and approve the change once they’re available.
I remember some uncomfortably large taxi and hotel bills when my wife ended up in hospital after a nasty accident on a cycle touring holiday together. I had to make emergency changes to our plans, and my wife was in no position to discuss them as usual. I made the best decisions I could, and let my wife know later once she was well enough.
Some teams use a “change freeze” process to reduce operational risk during a significant sporting event by avoiding making changes during an agreed period. For an inner source product, different teams may require this at different times. You can either agree a shared calendar which all teams respect, or apply the change freeze just to the team’s deployment repository. For example, if you use an open source product like MySQL, the MySQL community won’t respect your change freeze. But this isn’t a problem because you can choose not to deploy any MySQL updates until the end of your change freeze. You can treat an inner source product in the same way. The only difference is you need to consider how to deploy an emergency fix without pulling in changes that other teams have made during your change freeze.
Your Categories, Your Process
The change categories you need, and the processes you use for each, depends on your product, teams, and context. But if small, simple changes are slow – or large, complex changes are chaotic – you may need to find a way to separate them, and treat each type differently.
by: Rob Tuley
in:
tags: WoW Pull Requests
category: Academy