As your application grows, so does the flow of tasks for QA. And if you don’t reduce that amount of work in time, it will begin to threaten your releases. And just resemble a sprawling bush that’s not being looked after. I will tell you how we reduced the regression testing time from 5 to 3 days in a couple of months maintaining quality and relieving the level of stress for the QA team.
We at FunCorp live in two-week release cycles: we perform tasks, test logic and negative scenarios. Once every two weeks, we merge the tasks into one branch and freeze it.
Until three months ago, the process after the freeze looked like this:
During the whole working week of the regression testing, QA were not engaged in tasks for the next release. This has become an issue for ourselves: the developers were far ahead of us, and the testing queue was growing. It was stressful and made us feel like we were always chasing something. In addition, it was harder for programmers to switch context from current tasks to bugs that we found in tasks they had done a week or two before.
It hindered business — we even had to postpone the release a couple of times. And the rest of the time we could not include some of the tasks in it, because we did not have time to check them (but, of course, the most important tasks always got in the release). Given that we test product hypotheses with each release, and product data still needs to be collected, obtaining A/B test results was delayed. This caused even more stress.
It seemed to us that the problem was becoming systemic, which meant it was time to stop and review the process.
We decided to stop redundant testing. If we want to test hypotheses quickly, we need to test the hypotheses, not everything around them. So, the focus of testing will shift to them.
The first thing we agreed to do was to inject some tasks into the branch with developer testing rather than QA involvement. For example, if we are talking about updating some libraries, and if something is not updated, we will see it in the next steps. In this way, we reduced the number of tasks in the check queue and removed some of the routine.
And then we accelerated the process after the freeze:
Bottom line: we’ve improved overall task planning, overhauled the testing process itself, increased developer involvement in tasks, and come to the point where A/B tests get to the production 30% faster. No QA or release was affected during the improvement process.