#test automation
The rise in popularity of agile methodologies and increasing adoption of DevOps has had a big impact on quality assurance (QA) and testing. The use of iterations in the Agile process, and DevOps’ reliance on continuous integration and delivery (CICD) have prompted a movement from manual testing towards automation testing. Software Development Engineers in Test (SDET), QAs and Testers focused on automation are far more commonplace in engineering teams than even a few years ago, and the framework options are growing on an almost daily basis.
So how do you decide which test automation framework is the right one?
In this article, we consider how you can make a decision and choose the right option for your product and your team. We’ve come up with 10 questions you need to ask in order to establish your next steps. Then we assess the 1) open source 2) building your own and 3) ‘paid-for’ approaches before diving deeper into one of those options.
Choosing the right test automation framework is no picnic
Let’s be under no illusions – choosing the right framework can be difficult. There are a lot of frameworks for implementing QA automation solutions out there, and companies need to take care when choosing which one to use.
A good place to start is by answering the following questions with your team. They’re designed to help you identify your objective and ultimately what you want to achieve from your automated testing.
- Which level of software testing is within our scope? (Unit testing, functional testing, performance testing, load testing – what else?)
- Where will the software testing be used? (Front-end/back-end)
- Which devices are within the product scope? (laptops, tablets, smartphones, what else?)
- Which browsers need to be supported?
- What skills does our team currently have?
- What skills do we need to add to our team?
- How many concurrent and real users do we have, or expect to have?
- What technologies and third-party dependencies need to be integrated?
- What is the ideal reporting output?
- What is our budget?
Discussing these questions with your team will help you identify your needs, understand the most important factors for making the decision, and ultimately choose the right framework.
Open Source, Build Your Own or ‘Paid for’ Solutions
Now it’s time to decide which option is best suited to your team and product: open-source, building your own, or a“paid for” solution. There’s more to this decision than budget
considerations, although for some teams with little or no budget, a “paid for” option might immediately be ruled out. In the table below, we’ve listed some of the pros and cons each solution.
As the table shows, there are compelling reasons to choose any of the three options. When it comes to making a decision, you’ll need to consider your objective, the features you require, your resources and the level of external support.
It’s worth remembering that the framework is not the only cost. Having chosen your solution (and depending on your product), you’ll also need to invest in a CICD platform for executing your tests. Having a built-in test solution means you can plan and coordinate your test solution across environments, quickly.
For companies starting out on the automation journey, open source frameworks are often a great place to start, as information is widely available and there is a willing community ready to provide advice. Don’t expect to find the best solution on your first attempt. Try a short proof of concept phase to analyse the tool and figure out if it’s the right one for the job. In the remainder of this article, we take a look at a few trends in the open source community and the most popular frameworks right now. Of course, this isn’t the right option for every team and product – the ten questions above will inform what’s right for you.
Open Source Framework Trends
As mentioned above, the features of a framework are not the only thing you need to consider in your decision-making process. When considering which opensource tool to use, you need to take a look at the community, maintenance, dependencies, forks, issues, starts, updates and the level of quality.
The folks over at NPM Trends have put together a great tool for looking at the popularity of opensource frameworks. Chart 1 shows the most popular frameworks for functional software testing over UI, while the second shows frameworks for unit testing over UI.
Chart 1: Open Source Framework Downloads for Functional Software Testing in the past 2 Years
Chart 2: Open Source Framework Downloads for Unit Testing in the past 2 Years
From the data, we found that WebdriverIO, Nightwatch, Jest & Mocha are some of the most downloaded frameworks over the last 2 years. Despite emerging relatively recently, Cypress is top of the list. This tendency could be reflective of a more mature QA culture and also the result of a new era of digital products.
The data also clearly highlights that opensource test frameworks are far more widely used today than they were two years ago. Both charts have a more positive tendency for functional frameworks when compared with components frameworks, but in both cases, the number of downloads increased month by month.
Note, a “New Year effect” is clearly visible in the chart where each December-January there’s a quick decrease in downloads, but it soon normalizes.
Conclusion
Choosing the right framework can be tricky. There are plenty of options out there, and it can be overwhelming. Understanding your objective from the outset, and recognizing your team’s strengths and weaknesses will help you to start the process on the right foot.
Whether you choose the ‘paid-for’, open source, or ‘build your own’ approach, it’s important to remain flexible and to remember you can always adapt your approach. It’s important to keep automation code agnostic where possible and to be able to change the code if, and as required. Don’t hesitate to replace deprecated or depreciated code, and remember, clear and modular automated code can be enriched with other pieces of code and scripts as you go.
And don’t wait until your product is ready to start testing, as the saying goes – “test early and test often”.
Sergio Cabrera Oria is a QA Automation Lead at Zartis.
Zartis empowers companies to build outstanding software. Our services include technology advisory, remote teams and software development. If you‘re in need of QA support, reach out to us today!
