Testing Strategy: Planning of testing activities in the company Automation of the testing phase

The developer of the code is able to execute it and test it continuously in production-like environments, tracing any change in a version control system. However, the tests performed by the developers in the development phase represent only one of the testing categories to which the software is subjected by going through the pipeline. Only when the development phase is completed, in fact, the quality control team will perform a second testing phase. What often happens, however, is that this second phase is typically performed two or three times in a year, given the low frequency with which a developer releases his code. In this way there is a risk that the developers themselves will be notified of the errors after months, canceling the intrinsic advantages of continuous integration.

An automated testing strategy, on the other hand, aims to solve the problem, avoiding wasting a proportion of time proportional to the lines of code that are added during development. As soon as a developer commits, suites containing thousands of tests are automatically executed on the modified code. If all tests are passed, the code is ready for deployment in production. Starting in 2013, the use of automated testing and continuous integration allowed, in a large organization like Google, to coordinate thousands of small teams among them, all engaged simultaneously in development, code integration, testing and deployment in production. Their code is shared in a single repository, composed of billions of files, in which the code is continuously compiled and revised; about 50% of this code is changed in just one month. To give an example of how productive this approach can be, here are some of their statistics:

• 40 thousand commits a day
• 50 thousand daily builds
• 120 thousand suites used for automated testing and 75 million total tests
• More than 100,000 engineers who design new tools for continuous integration and software release automation, in order to increase the productivity of each individual developer.

The importance of following this approach is therefore evident, given the excellent results. Before going into the details of the continuous delivery pipeline, analyzing the flow of execution of the various test categories, we try to describe them individually, from the fastest to the slowest:

1. Unit tests: They go to test a single method, class or function, notifying the developer about the correctness of their code. The unit tests are written and managed exclusively by the developer within the development phase. They are the fastest test category because they do not interact with any other component, such as databases or filesystems, that require external calls. This implies, on the other hand, a little detailed testing which is limited to locally verifying the correct execution of the code. In the deployment of modern applications, which communicate with different databases or exchange data of different nature with external services, unit tests are not sufficient on their own to guarantee the total reliability of the software.
2. Functional acceptance tests: They answer the “How do I know if I’m finished” questions from the developers, and “I got what I wanted” from the user. Unlike the unit tests that go to validate a single and isolated part of the application, with functional tests it is shown that the application behaves as the client has requested and not that it simply works as the developer has planned.
3. Integration tests: They are the least used type of automated tests, given their high complexity and computational cost. Very often, in fact, after the correct execution of the functional tests, it is preferable to submit the application to the manual testing phase (eg exploratory tests, tests on the graphic interface). This is because not everything can be automated: aspects such as usability, appearance, perception, are difficult to automate and this is why manual tests must continue to be used, albeit in a decreasing percentage.

The ideal approach would be to first run the fastest tests, unit tests, and then those that take longer, respectively, acceptance tests and integration tests. The difference in terms of speed of execution between these categories is also reflected in their ability to send feedback to developers about errors detected during testing as soon as possible. In fact, if the developer quickly solves a bug encountered by the integration tests (or similarly by the acceptance tests), he will still have to wait a few hours before figuring out if it has been solved or not.

On the contrary, if the bug were detected by the unit tests, the feedback and the resolution of the errors would be almost immediate. It would be ideal, therefore, that most of the errors were detected by unit tests, but many times, the testing pyramid takes the inverse form. A possible countermeasure can be to create a new unit test case every time an error is discovered by the slowest test types; in this way, we try to enlarge the base of the non-ideal pyramid as much as possible, increasing the coverage of the fastest tests through a logic of error caching. In addition, in order to run the tests as quickly as possible, you might think about running them in parallel, potentially distributing them on different servers.