This is Part 2 of our automation series. Read Part 1 to understand of how to get started with automation.
Every automation tester faces this challenges when the test scripts that were written and executed perfectly on a local devices, fail when run on a server or alternative device. This is called test flakiness and everyone in automated testing faces it at some point or another. There are multiple reasons for test flakiness, and we will try to cover many of them in this series, so sit tight and relax as being calm and having patience will actually turn out to be one solution.
The most common reason is test synchronization, wherein your test script is not properly synchronized with the application under test (AUT). Your test script wants to perform an action on the AUT, but either the AUT is not ready or is still in process.
Let’s see how a little bit of patience helps make our tests more robust and reliant.
Patience Flow Diagram
For each test step that interacts with the AUT, you need to make sure that it follows the above diagram. Here we start with interacting with AUT at a spaced interval, for a predefined maximum period of time. This helps the AUT to process or load during the spaced intervals and our tests to be resilient while that happens. The maximum time per suite, or per script, is something usually determined by your service level agreements (SLA) or your Product Management Group who define what is the maximum time you can afford to wait for a particular action on the Product.
For example, if your application is a web app and it has a login page, then network connectivity and page load would trouble your test synchronization. Using the above logic let’s say we define a maximum time of 20 seconds, and within that, your test script should check the browser if the required input fields for login are available or not, and then query the same action in a spaced interval of 1 seconds for maximum time of 20 seconds. Similarly if your application is a mobile app, then scrolling or processing in the application will have an impact on your test synchronization.
This patience flow diagram helps PayPay run tests on their Staging and Production environments every minute, of every hour, of every day! We guarantee that our test code works all the time and to do that we had to build some mechanisms for our test code as well, which we would cover next month. Till then adios!
This is part of our monthly automation testing series, wherein we provide information, look at common mistakes made during testing, and how to scale up and quickly start your own continuous testing infrastructure.