This is Part 3 of our automation series. Read Part 1 to understand of how to get started with automation and part 2 for insight into reducing the flakiness of your automated tests.

On a warm summer night, my deep slumber was broken by an alert from one of our automated monitoring systems stating that “ABC system failed in Production”. I quickly got up and started verifying manually and it seems the ABC system was working fine.

To my dismay it seems there was some issue with the automated monitoring system itself and someone recently merged some change which was the real cause of this failure.

Sounds familiar? If so then you are not alone, and a lot of automation test engineers face this issue when any change that they make on their local machines, works fine on their local machines. 

However when the same change is merged and runs on a server it starts failing. The best way to solve this is by implementing continuous integration for your continuous testing framework.


The ✅ of clean code

Most of the modern version control systems have a mechanism wherein you could run some tests on the change that you intend to make before actually merging the change. Now the tricky part is, if your code does the only job of testing another piece of code, or application under test (AUT) then you need to be a bit crafty in your approach.

First identify the most stable part of the AUT, then target that with your CI test. So whenever someone wants to merge a new change to your codebase, you can run this CI test, if the test fails, it means either the change that you want to merge is toxic and would cause failures, or else the AUT itself is failing due to possibly a bug. In either case as an automation test engineer it is a WIN-WIN for you.

For example, if your AUT has a login page, or home page, then your CI test should target the loading of this page.

These checks help me have a good night sleep and also helps PayPay’s Continuous Testing Framework run flawlessly at all levels. Speaking of levels, there are multiple levels of End-to-End testing, which we would cover next month. Till then sayonara!