Testing Has a Type: Smoke, Sanity, Regression, Re-Test

What is it All About?

As a testing engineer, you've probably heard of such types of testing as smoke, sanity, re-test and regression testing. It is quite possible that many of these types are used by you on a daily basis.

In this article, I would like to clarify and explain the difference between these types of testing and try to figure out, draw boundaries (albeit conditional) where one type of testing ends and another begins.

For beginners in testing (and even experienced testers), separating these concepts can be difficult. And in fact, how to distinguish where sanity testing begins and smoke ends? How much do we need to limit the verification of a part of the functionality of the system or its components to call it "smoke" testing?

Is entering a login/password into the user login form on the site a smoke test, or is the very fact of its appearance on the site page already a passed test?

Strictly speaking, you will still be able to conduct testing, even though you won't be able to tell exactly what the difference is. You don't even have to think about the distinction of what kind of testing you are currently engaged in. But still, in order to grow above yourself in a professional sense, you need to know what you are doing, why, and how well you are doing it.

Details

Below are brief definitions of the types of testing that we are comparing today:

• Smoke tests: are performed every time we receive a new build (version) of a project (system) for testing, while considering it relatively unstable. We need to make sure that the critical AUT (Application Under Test) functions are working as expected.

  The idea of this type of testing is to identify serious problems as early as possible, and reject this build (return it for revision) at an early stage of testing, so as not to delve into long and complex tests, thereby not wasting time on obviously defective software.

• Sanitary testing: Used every time we get a relatively stable software build to determine the operability in detail. In other words, it validates that important parts of the system functionality work according to the requirements at a low level.

Both of these types of testing are aimed at avoiding the loss of time and effort in order to quickly identify software flaws and their criticality, as well as whether it deserves to go into a phase of more in-depth and thorough testing or not.

• Re-test: Performed if the feature/functionality already had defects, and these defects were recently fixed.

• Regression tests: actually, what takes the lion's share of time and what testing automation is for. Regression testing of AUT is carried out when it is necessary to make sure that new (added) application functions / fixed defects did not affect the current, already existing functionality that worked (and tested) earlier.

For a better understanding, a comparative table of these concepts and areas of application is presented below:

Smoke

• They are executed in order to verify that the critical functional parts of the AUT are working as expected.
• The goal is to check the "stability" of the system as a whole in order to give the green light for more thorough testing.
• Rechecking defects is not the purpose of Smoke.
• Smoke testing is performed before regression testing.
• It can be performed automatically or manually.
• Is a subset of regression testing.
• Test cases are part of regression test cases, but covering extremely critical functionality.

Sanity

• It is aimed at establishing the fact that certain parts of the AUT still work as expected after minor changes or bug fixes. The goal is to check the overall state of the system in detail in order to proceed with more thorough testing.
• Rechecking defects is not the purpose of Sanity.
• Sanitary testing is performed before regression and after smoke tests.
• More often performed manually.
• A subset of acceptance testing.
• Sanitary can be performed without test cases, but knowledge of the system under test is mandatory.

Regression

• Confirm that recent changes in the code or the application as a whole have not had a negative impact on the already existing functionality/set of functions.
• The goal is to make sure that recent changes in the code have not had side effects on the established working functionality.
• Rechecking defects is not the purpose of Regression.
• It is carried out based on the requirements of the project and the availability of resources (closed by autotests), "regression" can be carried out in parallel with re-tests.
• The best reason to automate this type of testing, because manual testing can be extremely resource- or time-consuming.
• It is executed with any modification or modification in an already existing project.
• Regression testing test cases can be derived from functional requirements or specifications, user manuals, and are conducted regardless of what the developers have fixed.

Re-Test

• Rechecks and confirms the fact that previously littered test cases pass after the defects are corrected.
• The re-test verifies that the defect is fixed.
• The fact that the defect has been fixed is confirmed by Re-Test.
  • Re-test is performed before sanity testing.
  • Also, the priority of the re-test is higher than regression checks, so it should be performed before them.
• Not amenable to automation.
• The re-test is performed on a corrected assembly using the same data, on the same environment, but with a different set of input data.
• The same test case that revealed the defect is used.

But Essentially?

I will give an example of the differentiation of concepts on my current project. Example: We have a web service with a user interface and a RESTful API. As testers, we know:

• That it has 10 entry points, for simplicity, in our case located on the same IP
• We know they all accept a GET request for input, returning some data in json format.

Then you can make a number of statements about what types of tests you need to use at what point in time:

• By executing one simple GET request to one of these entry points, and receiving a response in json format, we are already convinced that the smoke testing has passed.

  If one of these entry points also returns data from the database, while the first one does not, you need to additionally make another request to make sure that the application processes requests to the database correctly. And this is the end of the "smoky" test.That is, we executed the request — a response came from the service, and it did not "smoke", that is, it did not return a 4xx or 5xx error, and something inarticulate, instead of json. On this we can say that the "smoky" test has been passed. To check that the UI works the same way, just open the page in the browser once.

• Sanitary testing in this case will consist of executing a request to all 10 api entry points, verifying the received json with the expected one, as well as the presence of the required data in it.

• Regression tests will consist of smoke + sanity + UI running together in one heap. Purpose: to check that adding the 11th entry point did not break, for example, password recovery.

• Re-test in this example is a point check that, for example, a broken api entry point in the next build works out as intended.

At the same time, if this api also accepts post requests, then it is obvious that it is necessary to include these requests in another set of sanity tests. By analogy with the UI, we will check all the pages of the application.

Let's Summarize

I hope that after reading this article, you will have clarity in determining which type of testing you are using at what stage, and what is the difference between these types of testing.

As mentioned at the beginning, the boundary between these concepts is very conditional and remains at your discretion within the framework of the project.

Thank you for your time, that is all.

Ruslan Muradov.