PayPay is one of the most popular and fastest growing mobile payments services in Japan. At PayPay, we follow an agile mindset, wherein we have Continuous Development, Continuous Integration, Continuous Deployment, and Continuous Testing. This blog series would primarily focus on Continuous Testing.
Starting small, and keeping it simple is very important when building any software or development framework. This is especially true when it comes to creating test automation infrastructure. A common misconception is regarding where does the test automation team lie, is it something that belongs with the Development team? Or is it something that belongs with Product Management Groups? And that is where most of the issues start with, identity crisis.
However if your aim is to guarantee the quality of the finished product, then none of that should matter as in these agile times, you need to be fast in churning out your test scripts, nearly as fast as developers build their application or product managers think and execute their product features, it needs to be at par with them. If you build something that creates more problems for you then you have built it wrong. Automation testing should be so simple that anyone should be able to contribute, and yet it needs to be so flexible and robust that it can handle anything you throw at it.
Coming to the part of starting your own test automation infrastructure, there are going to be numerous issues which you would face, Here’s what we did at PayPay. All these learnings are based on the work that has been done at Paypay to build an agile testing suite.
Baby Steps
While creating your 1st test script or your 1000th, always keep these steps in mind, your goal should be to write a script that could execute at any time, in any place, on any infrastructure. This really helps create a robust test script, which could run constantly and guarantee that the application under test (AUT) is working well.
Your test script should start in a known state or known place, at a point which you know will not fail, otherwise you would end up either running wrong tests, or which take too much time to execute. If you know that the first step is done, then your testing point should be a very specific point and each test step should only do one thing at a time. Finally it is always a good practice to clean up, and end in a known state, one wherein you know that test has to stop, for the next test to begin.
For example, if your application has a login page, or sign up page, when your test case is not related to testing login, or sign up, always create separate automation scripts for those steps in a way that they never fail, like using application API’s and/or setting predefined cookies and always close the browser or mobile application after each run, that way your tests would always run in isolation and interdependent of each other.
Use Automation to help your Automation
These baby steps are something that helped PayPay scale from 0 automated tests to around 200 for Web, more than 1000 tests for both Android and iOS each, and around 2000 tests for APIs in less than a year! We are running these tests with less than 5% of flakiness, something we were able to achieve with our own nifty polling logic, which we would cover next month. Till then sayonara!
This is part of our monthly automation testing series, wherein we provide information, look at common mistakes made during testing, and how to scale up and quickly start your own continuous testing infrastructure.