Being able to to run individual tests repeatedly and also being able to run them consecutively is a serious challenge when it comes to automation. I think all automation engineers want it, but some fall prey to not having it due to time constraints, setup and teardown complexity, or system access limitations.
What good is automation when it can’t be ran repeatedly, further more running them all at once? Seems like a dumb question, but I’ve seen plenty of non-repeatable tests or suites written by others, and occasionally have fallen prey myself.
How do I ensure repeatability?
- When planning the automation time budget, I allocate at least 15% for building infrastructure. Can’t get the time? Fight for it, as an automation engineer it’s your job. You have to sell the fact that if you don’t have the time to build in the infrastructure for repeatability your tests will be less useful, take longer to setup due to manual intervention, and will eventually crumble into uselessness over time because manual intervention is not something you can easily hand off to another automation engineer or developer.
- I start building in repeatability with test #1. This test can often take the longest to write because I’m often times building in the infrastructure needed to run all my tests repeatedly (reusable functions, system access functions (db connections, etc), setup and teardown functions).
- When the test is done being developed, I run the test, I run it again, I run it again. repeat and rinse until it passes every time.
How do I ensure I can run all my tests consecutively?
- I build each test so that it can be ran independently, which means that when I run them all from a list, no matter which order they are in, they will not interfere with each other. This is why extracting common code into methods, setup and teardown is instrumental; I will reuse it in almost every test.
- I try to run all tests after each new test is developed and completed to make sure the new test plays nice with the others. This isn’t always possible if you have long running tests, but if they aren’t, I definitely do this.
- I run them all at least once a day.