How do you know something works? You measure it. If you cannot measure it, you do not know if it’s working or not. But how do you measure something? Sometimes it is plain for all to see, but often this is not the case. Can you measure your health? Or the quality of your’s or somebody else’s work? Perhaps you can, but not directly. You need to represent something complex in a simple way. A number. It is amazing when it works, but it’s so hard to define one. This is why it is often wrong.
When a number becomes important
You’ve got a fitness tracker to measure your daily step count. Apparently the recommended amount of steps per day per person is 10 thousand. Where does this number come from? Like most people, you don’t care. What you care about is that every time you look at your wrist, you can see your step count. And the steps of your friends, of course. Because naturally, there is an app for that.
Suddenly, this number becomes important for you.
So important, you start to change your habits.
Morning, time to go to work. Bus or car? Bus is 15 min slower, but you add 3000 steps this way, so bus it is.
Going out for lunch or ordering pizza? Go out! 1500 bonus steps!
A month later you are doing 12K steps per day on average. You feel good, no walk can scare you now. You are among the top 3 of all your friends that are using the app. Your spouse and family also own a fitness tracker as well.
Now you are a slave to that number.
You are not asked to improve your health. You are asked to make more steps. And this is easy, so you do it. All is good.
Now when I look at it, I am currently at second position…
What if the number is asking you to do the wrong thing?
While ago I was part of a team in a large organisation. We were building software and release was approaching. As a way to ensure quality, process dictated that we pass a certain criteria before we can say our product is “Ready”. One of the criteria is number of bugs in our bug tracking system.
We had a lot of bugs. Release date was approaching, so management decided to do something about it. The team that will crush most bugs in the next week will get an reward!
This is how many big organisations work, nothing surprising. There is a process and the number of bugs is a checkbox there. Management said – crush bugs, so we crushed!
For two weeks we competed. Three teams got a reward.
Did it work? We checked that checkbox, so I guess it did.
The week after we were closer to our software release. Except one small thing – our automated test coverage was low. We have automated a lot of our testing, but now we were not hitting the number prescribed by our process.
Another number, another target. No rewards this time, they were not needed. After a while we hit that and some other metrics and finally our software was released.
What was the wrong thing?
Just like with the daily steps count, we changed our behaviour in order to meet our goals. Some changes were good. Some were not.
- We got focus.We focused only on fixing bugs. Focus is good. Doing one thing at a time is usually much more effective.
- We were discouraged to impact the number negatively. In other words, we were discouraged to open new bugs. Not that we spend a lot of time on bug-hunting, but still not good.
- We were focused on what is easy to do, not what have to be done.Why bother with more complex stuff, when you can show progress now? Progress is a good thing. Focusing on easy work too much however is damaging in long-term. Complex bugs and automation are left for last, but those are often much more valuable than the “easy-to-do” ones. You risk doing a lot of work with no real impact this way, so you have to be careful.
- Our metric targets were in conflict. Conflict is actually good – it leads to balance. Fixing bugs and test automation in our case were in slight conflict. To explain how are they related – some bugs blocked automation efforts and had to be resolved first. So to complete our second target we also had to fix some bugs with higher priority. What if we had these two complex issues that are blocking our automation? As they were too complex we saved them for last while fixing bugs. Now we will have to fix them anyway, but our test automation will be blocked and we cannot progress. Yes, this really happened 🙂 Balance can only be achieved if you actually let two opposing forces work against each other. Focus on one at a time too much and there you go.
- There were other ways to impact the number other than doing the work.You see, you do not have to fix the bug, you can only declare it as closed because of some other reason. This is a grey area, but if you are wondering if a bug is invalid or duplicate of another, now you are more likely to go and mark it as invalid. Certainly more bugs were marked like this while we were bug-hunting. As this is not an easy decision for sure it was OK for many of them, which is good. Many ended up as “false-negatives” and were reopened after.
Number too important?
Issue is not with the defined metrics. Problem, in this case? Prizes. This is when a number becomes important. Too important.
Like every parent knows too well, you should not give your kids too many rewards. They will start doing stuff only if they receive something in return. Why should grownups be different?
Prizes and quotas sound good in theory and are great motivators in the short term. Rely on them too much and you pay the price. And it is not only the people “doing the work”. Danger is also for the management to set metrics that could be completely off. Simplifying complex stuff to a simple number is hard, so its often wrong.
Want an example?
We had quota for the bugs we had to fix as well. As a senior software developer I had to fix one bug per day on average. Junior developers where expected to fix one per two days. Reasonable if you work in manufacturing and all work is the same size. I wish software bugs were always exactly the same. Alas, this is not the case. I guess what this quota system served is good way to predict our performance for the upper management. In a way they also got it. They asked for it, and this is what they received. Which brings me to my last point.
Whatever you expect, you are going to get
Nothing more, nothing less. 70 bugs fixed, 85% test automation. Important numbers. But do they lead to high quality, or people just meet the number given to them? How do you spot the difference?
Question numbers frequently. Search for ones that balance each other. Resist the urge to make it too simple. Do not rely on them too heavily.
But most important of all:
Be careful what you ask for.
Because you may get exactly that (and it will not be what you wanted in the first place).
Thank you for reading! If you like it, please subscribe below.