Flipping The Reversed Test Pyramid

Some time ago, my team was tasked to implement a form in an website. I do not remember exactly the functionality. What I do remember is how we approached the problem with our QA. Our organization invested in test automation and each team had automation QAs assigned to them.

As part of our process and to save time chasing future issues, we discussed the test cases. Our testers prepared them before this meeting.

Black-box approach

If this was a simple registration form (it wasn’t :)), this is how the tests would look like:

• When form with all mandatory fields filled is submitted, it will go to a confirmation page
• When form without first name is submitted it will display an missing required mandatory field error on submit.
• When form without address is submitted it will display an missing required mandatory field error on submit.
• etc…

I think you can imagine the rest 🙂

Those test cases were written from testers perspective and naturally described our system as a black box being tested.

We discussed them and our test automation approach. QA will then use Web Browser Automation tool like Selenium to automate them.

All these in Selenium? Hmm, well, but how will I, a developer, test this if I had to do this?

A not so “black” approach

Well, the difference between me and the QA is I knew the implementation details. This made me think differently about the problem. What I would do will be something like this:

• Form without an error will lead to a confirmation page
• Form with an error will display error upon submit
  • Forms with missing first names will produce missing first name error
  • Forms with incorrect birth date format will produce incorrect date format error
  • etc…

And how do I automate those?

We can of course still do it with Selenium, all the way. But that was a lot of slow tests to my liking. The first two tests I can test with Selenium. They described the flow of my application. I wanted to make sure the flow is OK. The rest I will unit test, focusing not on the whole browser, but just this single page code.

What did we chose?

I explained my solution, but got confused looks from our QA team. I was stepping on their turf. You see, while our organization really wanted high level of automation, it was the QAs that owned it. Most of the automated tests were just the steps of a manual execution, automated.

They said that by not focusing on the end-user bugs will go unnoticed.

This concern sounds reasonable, but I am not sure it was the only issue.

It was not only end-to-end tests, it was also many of our tests in general were integration and most treated our systems as a Black-box.

What about the programmers and unit testing? Well, we were certainly not discouraged to write them… But we were not encouraged to write them as well, so few bothered. Why waste effort, when somebody else will do it?

Problem with Black-box only testing

So what we got in the end was essentially Black-box only test automation. I knew from experience this does not end well.

It is like an reversed test pyramid:

Reversed, because instead of using unit tests as a foundation, with less integration / end-to-end on top, it is doing the exact opposite.

Not using unit tests as a foundation lead to some issues:

• Build was slow, it took time to understand if a fix worked/broke something else
• Tests were not isolated – whole series of tests may fail because they shared steps
• Tests were flaky – too many false positives
• Test failure analysis took a lot of time
• Even when there was a bug, it took time to find exactly where the issue is

Flipping the reversed test pyramid

Automation QA leads were aware of the issues with slow builds. Their solution was to plan and rely more on integration tests instead. While more reliable and faster, those were still fragile and required their QA to become proficient in programming.

We needed to flip the pyramid, but this required cultural changes inside the organization.

What I did was to try to promote unit testing at least in our team. I would often pair with another developer and test drive our code. Or work with the automation QAs on their integration tests and try to make them more focused and less flaky.

I wish this particular story ends well, but it didn’t – I left that job while ago. I still maintain contact with my former colleagues and from what I hear things are moving in the right direction, but perhaps not too quickly.

Is Black-box evil? No, you absolutely need it. Otherwise your team will have big issues with integration and configuration. Point is – they should not be your main focus.

Why bother?

Black-box only testing is just a symptom.

It is my belief, that the only way to achieve fast delivery and good software quality is when people on a team – programmers/testers/etc work together.

For years however I observe similar behaviors in both QA and programmers. QA will be “responsible for quality” and treat attempts to change this as “stepping on our turf”. Programmers are happy that somebody else will do the testing for them, so will not bother to write tests themselves. Testing is seen as something that “slows them down”.

Agile/Lean helps, but even when testers and programmers are part of the same team, their work and responsibilities are often divided. Each group will then optimize its own part of the software delivery.

Who will optimize the whole?

We are all software developers

Until now, I have purposefully used the terms QA, tester and programmer, but not the term developer in this post.

This is because developer for me is a person that develops software. It could be a programmer, a tester, business analyst or whatever else is found in a software organization.

Does it matter who is doing what in a team, as long as the end product is good? Shouldn’t we all be responsible for quality, quick and reliable delivery?

If are all responsible, how often we will have to to deal with low quality, slow builds and flaky tests?

We should be free to optimize the whole software delivery process, not just isolated parts of it.

It is OK then for programmers to write unit tests. Yeah, it is more work for them now, but it saves much more work in the future for everybody. It is OK for testers to trust others with overall quality of the product, because they are not the only ones “responsible” for it anymore.

Together, we develop software and help our fellow developers to do their best.

This is how pyramid can be flipped back to how it should be.

This is how everything can be flipped back to how it should be.

Thank you for reading.

If you would like to receive future post updates, please subscribe below. I post once per week and will not spam you.