Last year, due to the pandemic, businesses across all industries were forced to adapt and find new ways to operate remotely, exerting unprecedented pressure on web and mobile apps to stay reliable and perform flawlessly. The need for the highest quality apps put software testing into the spotlight.
Any business with mission-critical applications or technology systems requires testing. Testing is now becoming automated and there is a shift towards a more agile model to ensure the testing quality is improved and the risk of failure is minimised.
The software testing landscape has changed and evolved a great deal since the beginning of the COVID-19 pandemic. So, we analysed the current state of software testing and drew some interesting conclusions.
Current trends in today’s software testing landscape
Teams are focused on more automation and better coverage. Coverage, not just from test scenarios but also covering more platforms.
As social restrictions continue, it has driven a significant reliance on technology. The mobile and web applications must perform flawlessly, more than ever, across many different channels, platforms, locations and more, which has created a need for greater quality.
Despite its link to faster releases, many businesses seem to still automate fewer than 50% of their tests. But the trend is changing as many have recognised the importance of automation. Due to increased pressure and time to market, CIOs are now making automation a key priority.
Shifting left is ‘shifting’ into high gear
Software development is driven by the need for faster releases. Large organisations are leading the pack in shifting left. But you can only do this successfully if your automation is good, to begin with. When you have an unstable or unreliable automation in place, shifting left does not provide the much need benefit, so be wary.
The top performers are addressing the automation skills gap
With higher levels of automation within the typical release cycle these days, leading organisations are evaluating behaviour-driven development (BDD) and script-less solutions. Few organisations, realising their gap in the landscape, are mixing and matching technologies and frameworks to fit different skills in their teams to contribute to the quality and faster releases.
Leaders and practitioners are not on the same page
The C-suite and VPs believe their organisation’s testing outputs are more automated than specialists are reporting. Businesses should prioritise aligning their views when assessing their next two year investments.
DevOps Trends
- DevOps is often differentiated by the following criteria:
- Rapid release frequencyFully automated CI/CD pipeline
- Ability to promote/demote releases frequently
Democratising quality assurance across the entire organisation is key for DevOps success (shifting it left and right). If quality is not top-of-mind for every practitioner in the whole DevOps/software development/delivery teams, and everyone else, then DevOps will not work.
DevOps is being driven by growing practices helping shift-left, including:
- Use of features flags (now supported by Android APKs as well)
- A/B testing
- Cloud-based testing
The top software testing challenges affecting businesses in today’s landscape:
- Lack of resources for test creation
- Too many manual test cases, inability to fit in the schedule
- Not achieving the right test automation coverage (knowing what to automate on which platforms)
- Implausibility to automate advanced scenarios – this is often quite costly for businesses. But you must automate advanced scenarios as manual testing in complex scenarios leaves high risk. Businesses should use solutions such as machine learning (ML), diverse frameworks, mix of tools and take the approach of adding new skills to the team to address the situation. The complexity of tests is not an excuse.
- Testing is not prioritised properly and done too late in the cycle
- Deciphering why tests fail
- Matching tools to the right skillsets
- Test flakiness (elements, etc.)
- Test environment set up and test maintenance
- Frequent change in application requirements
- Justifying testing within the organisation (proving value/ROI and earning sufficient budget)
- Understanding product requirements/user stories
- Working with dev and agile/TDD methods
2021 and beyond…
Which frameworks are top of mind in 2021?
- Cypress – 40% (JavaScript framework)
- Flutter – 19% (Cross-platform for mobile and web development and testing)
- Codeless solutions – 44% (Expands the test automation coverage and addresses skill gap for coding)
- Playwright – 10%
- Others – 31% Which trends impacting testing are people looking to invest in this year?
- Progressive Web Applications – 21%
- Foldable phones – 13%
- 5G – 29%
- AI and ML in codeless testing – 61%
- Other – 14%
What kind of test documentation can testers create in 2021?
- Traditional test reports and dashboards
- High-level test plans – This is very specific for vertical industries and projects of large sizes
- Checklists and criteria documents
- Mind maps – This is a very powerful tool for testers, PMs and developers
- Test charters and team goals
- Bug reports
The current state of test automation in 2021 – Here are the top used frameworks for testers:
For cross-browser testing:
- Selenium
- Cypress
- Google, Firefox, IE, Safari
- Playwright
For mobile app testing:
- Appium
- Calabash
- Espresso
- UI Automator
- XCTest
For cross-platform testing:
- Cucumber
- Flutter
Why are teams building, or not building, custom test automation frameworks today?
Reasons for building own test automation frameworks:
- A desire to customise the framework for test demands – 64%
- Affordable cost – 37%
- There are sufficient skills and resources to do so – 37%
- Previous use of own framework since test automation began – 30%
Reasons for not building own test automation frameworks:
- Lack of time – 51%
- Other tools have all the capabilities for test automation – 48%
- Lack of sufficient skills – 43%
- Other tools are more affordable – 37%
- Building new frameworks are too expensive – 29%
Release Cadence and Test Automation Fitness
What does a typical release cycle look like?
- Daily – 9%
- Weekly – 39% – It is important to ask, what is being included here? Remember that releasing weekly is good, but many businesses doing this are still not covering everything, or in some cases, the quality of testing is being compromised to keep up this frequency which is potentially counter-productive.
- Monthly – 33%
- Quarterly – 14%
- Once or twice per year – 4%
Where does testing fit into a software development lifecycle?
- At the beginning of each software sprint – 42%
- After a new build is ready – 71% – This is the basis of shift-left, to include as many of your automated test cases in the build acceptance testing, then it will be an improvement.
- Upon each code change – 54%
- Unable to keep up inside the sprint and so it is added to the backlog – 12%
The current test automation maturity
- 33% of the industry currently automates between 25% and 50% of their tests
- It is still rare for people to be automating more than 75% of their tests
- But a significant majority automate between 10% and 75%
Ask yourself, if you are automating 50% of test cases and releasing weekly, where is the other 50% going?
To be in a DevOps, shift-left, CI/CD reality, you need to be up to 75% or 80% automation of all test cases, not just the easy ones.
Testing as part of the CI/CD Process
Where are businesses currently in the DevOps transformation?
- 16% Waterfall
- 31% Agile but still throwing code over to QA and Ops
- 35% recently started making strides towards DevOps
- 18% DevOps
The testing role within CI/CD
- 46% said testing is an active part of the CI/CD process
- 26% said they do not have CI/CD or testing is not part of the process
- 8% said their testing team gets reports without actionable outcomes
- 20% said testing teams use CI/CD reports as ongoing feedback
Sources of testing information – How do you build your test cases?
If you have good user stories coming from your product owners, you will have valuable, clear information to know where to do your testing.
- 79% User stories
- 70% Past experience in other testing projects
- 65% Bugs from customers
- 62% Formal requirements
- 54% Brainstorming sessions within the team
- 53% Work based on exploratory testing and heuristics
- 36% Interviewing customers and/or product owners
- 20% Production logs and/or monitoring services
- 13% Application modelling
- 6% Formal process
- 2% Other
What should you do this year and going into next year to meet the trends?
Try to understand the different personas and roles within your organisation, be clear on their objectives, and identify the strengths of their skillsets to find the best technology for them to use.
Select and match software testing automation technology to the organisation’s structure.
Each person’s coding skillset must align currently with the test robustness they are working with. For example:
- Developers – They want speed, and they want to accelerate feedback, execution, and so on. Some examples for this persona will include Cypress, Espresso and XCUITest.
- SDETs – These will typically need a bit more time, so they want enhanced test automation, coverage via coding, etc. Examples here would be Selenium, Appium, and similar tools.
- Business testers – Here you’ll want to move from manual to automation with no code to make the whole process easier. This could involve Tricentis, Cucumber and similar.
We recommend you match the tools and frameworks to the individual's skillsets and give what they need to help the organisation as a whole move towards achieving its overall quality/transformation goals.
See where your skills lie within the whole team and supplement that, or fill any gaps, with the right tools and systems for automation.
Upcoming technology trends to keep an eye on…
- Apple AppClips
- Foldable smartphones
- PWAs
- Android APKs – These are coming into play from June as a mandatory requirement
- Smarter testing and big data analytics
- Standardisation of automation frameworks
Closing Note
To discover how Testhouse can transform your business, by providing deep insights into the quality of your software products, contact us for a discussion and we will be happy to assist you.
Also published on: https://www.linkedin.com/pulse/state-software-testing-industry-2021-anish-roy/