Testing is the most important part of the software testing and development lifecycle, but people usually don’t give it serious attention, thinking that it is time consuming.
However, this can’t be far from the truth. Testing is not time consuming; instead, finding bugs and resolving them might consume significant testing time.
Any well-developed can be deployed in the market for the general public use only after being successfully tested with almost all the bugs identified and removed.
We have said it is almost impossible to identify every bug during software testing and remove it.
But what exactly is software testing?
The ANSI/IEEE 1059 standard defines software testing as a process that evaluates the software application features and analyses and detects the difference between required and existing conditions in the software application features.
Depending on the stakeholders and other project leads in a small or large organization, a single tester or an entire testing team is responsible for evaluating the software application developed by the developers.
You might have heard the terms Software Tester, QA Analyst, Software Quality Assurance Engineer, etc., as some designations in different organizations are given to people who perform testing.
Generally, the following professionals are involved in software testing:
- Software Developer (mostly in Unit testing)
- Software Tester
- Project Lead/Manager
- End Users
In layman’s terms, software testing is the process in which a tester needs to evaluate a system for possible bug fix and verify and validate it against the specified requirements.
To sum up, Testing is done to identify errors, defects, gaps, or missing requirements in the existing system when compared with the actual requirements.
Few confusing terms
Before moving on with our software testing guide, let us first throw some light on the most confusing terms in software testing.
1. Testing vs. Debugging
Many of us might have heard these two terms, testing and debugging, and often think they are the same.
But they aren’t. In fact, there are major key differences between them.
|1||Identifying bugs, error, or defect in the software but not correcting it is called testing.||It is a part of White Box Testing. Unlike testing, debugging identifies, isolates, and fixes the bugs.|
|2||QA background professionals are responsible for testing.||Since it is done as unit testing, developers who code that specific section of the application are responsible for debugging it as well if they encounter some error.|
|3||Testing is done in the testing phase.||Debugging is done in the development phase during unit testing or in subsequent phases involving the fixing of reported bugs.|
2. Verification vs. Validation
Most often, people get confused about verification and validation and make the mistake of considering it as the same term.
However, there are few key differences between them both, and hence they should not be used interchangeably.
|1||Validation is a subjective process and requires subjective decisions related to the working of the software.||Verification is an objective process without any need for a subjective decision.|
|2||Dynamic activities such as the execution of software against its requirements are included.||Static activities such as walkthroughs, collecting reviews, and inspections are included.|
|3||Ensuring that the functionalities have the intended behavior is a part of validation.||Ensuring that the software meets the functionalities is part of verification.|
|4||It is done by testers.||It is done by developers.|
|5||Validation is done after verification and includes checking of the overall software.||Verification is done first and includes checking of code, documentation, etc.|
|6||Answers the question: “Are you building the right thing?”||Answers the question: “Are you building it right?”|
3. Audit vs. Inspection
In the verification process, audit and inspection activities are carried out as part of the static activities involved in verifying any software.
Since these terms often seem familiar, let’s understand how they are different.
|1.||An audit is a systematic process that includes an independent examination of actual testing processes that is done as part of the software testing within any organization or team.||Inspection is a formal verification technique in which an artifacts’ formal or informal technical reviews are conducted to identify any error or gap.|
|2.||According to IEEE, audit is a review of documented processes that are implemented and followed in any organization.||According to IEEE, inspection of software requirements, codes, or designs is a formal review technique that examines faults, development standard violations, and other problems. It is done by a person or a group that excludes the author.|
|3.||Legal Compliance Audit, System Audit, and Internal Audit are the types of audit.||Processes included in a formal inspection meeting are:PlanningOverview PreparationInspection MeetingReworkFollow-up|
4. QA vs. QC vs. Testing
These are some other confusing terms involved with software testing.
Quality Assurance (QA), Quality Control (QC), and Testing are interrelated terms, but few key points differentiate them and set them apart.
|S. No.||Quality Assurance (QA)||Quality Control (QC)||Testing|
|1.||It includes process-oriented activities.||It includes product-oriented activities.||It includes product-oriented activities.|
|2.||Process-oriented activities such as ensuring implementation of processes, standards, and procedures during verification of developed software are included in QA.||Activities ensuring verification of developed software against documented requirements are included in QC.||Activities identifying bugs, errors, or defects in the software are included in testing.|
|3.||QA is a subset of STLC (Software Testing Life Cycle).||QC is a subset of QA.||Testing is a subset of QC.|
|4.||QA is a preventive process.||QC is a corrective process.||Testing is a preventive process.|
|5.||Instead of conducting actual testing, QA focuses on processes and procedures in the actual testing of the system.||By implementing procedures and processes and executing software to identify bugs or defects, QC focuses on actual testing.||Testing focuses on actual testing.|
Standards for Testing
International Organization for Standardization is the most renowned organization worldwide that develops and implements different standards for the improvement of software quality.
Below are some ISO/IEC (International Organization for Standardization/International Electrotechnical Commission) standards that are mostly used and associated with QA and testing.
Part 11 of ISO/IEC 9241 standard specifies the extent that a product is used by its users such that it remains effective, efficient, and satisfactory during usage.
Instead of focusing on clients’ production processes, ISO/IEC 12119 deals with software packages delivered to the client.
Its primary focus is on:
- Providing instructions for the testing of software packages with clients’ specified requirements.
- Providing requirements for the delivered software packages.
ISO/IEC 9126 standard sets appropriate measures to determine the quality and attributes of that software application quality.
- Quality model
- Internal metrics
- External metrics
- Quality in use metrics
SQuaRE or Software Quality Requirements and Evaluation guidelines are provided by ISO/IEC 25000:2005.
As the name suggests, this standard helps enhance and organize software quality requirements processes and evaluations.
There are the following main aspects of SQuaRE:
- Reference Models
- Requirement Engineering Standards such as planning, evaluation process, specification, and measurement
- Individual Division Guides
- Terms and Definitions
- General Guide
The two old ISO standards – ISO 9126 and ISO 14598 can be replaced by ISO – 25000 standard.
The subparts of SQuaRE are:
- ISO 2500n – Quality Management Division
- ISO 2501n – Quality Model Division
- ISO 2502n – Quality Measurement Division
- ISO 2503n – Quality Requirements Division
- ISO 2504n – Quality Evaluation Division
Testing: Start and Stop
After being aware of a few basics related to testing, let us understand the start and stop testing conditions.
An early bird catches the worm, or early testing catches the bug sounds about right when testing.
The earlier we start the testing process in any project, the more budget and time friendly testing will be. Early testing significantly reduces the cost of producing bug-free software and the time involved in ramifying those bugs.
In the SDLC or Software Development Life Cycle, we can start testing and go on until the deployment phase once the requirement phase is completed.
However, the testing start phases might differ according to the development model used. In some development models, such as the Waterfall model, testing takes place in the testing phase. In contrast, in other models, such as incremental models, testing is performed at every iteration and then again once the system is developed.
Different testing done in different phases include:
- Requirement phase – Requirement analysis and verification testing to match the requirement document with the stakeholder.
- Design phase – Design review testing to improve software design.
- After the coding phase – Testing is done by the developer.
Since testing is a never-ending process, it can never be claimed that testing is 100% complete.
Thus, the following aspects might prove helpful when trying to stop the testing process –
- Completion of testing test case execution
- Management Decision
- Completion of functional code coverage to a satisfactory point
- Testing deadlines
- The decline in bug rate and identification of high priority bugs.
Different Types of Testing
All the testing types are done to identify bugs and any other anomalies in the software.
Several aspects, such as test planning, test cases, and test scenarios, ensure that the testing is completed.
At a general level, there are mainly two software testing types user:
- Manual testing
- Automation testing
1. Manual Testing
As the name suggests, when any software testing is performed manually without any tool or script, it is called manual testing.
Manual software testing involves testing the system for any bug or unexpected behavior by assuming the role of the end user.
Different stages involved in manual software testing are:
You can find the various manual testing tutorial over the web to enhance your knowledge about testing manual testing.
2. Automation Testing
In Test Automation or Automation Testing, the tester uses another software to test the product by writing a script.
It is better than manual testing because it automates the manual testing process, and it quickly re-runs the manually tested test scenarios repeatedly.
It improves accuracy, increases test coverage, and saves money, and is less time consuming when compared to manual testing.
Since automation testing is done by software, thus there are many automation tools available in the market for automation testing, such as:
- HP Quick Test Professional
- Visual Studio Test Professional
- Testing Anywhere
- IBM Rational Functional Tester
Using any of the above automation tool, one can easily do automation testing.
Computer languages such as VB scripting can be used along with these tools. Many script writing tools are available in the market.
Following processes are used to automate testing:
- Identification of bugs and performance issues
- Writing test scripts
- Executing test scripts
- Identification of software components for automation
- Creating result reports
- Developing test suits
- Selecting an appropriate test automation tool
Now that you know how to do automation testing, we should understand when we can do automation.
The following aspects should be considered before automating any software:
- Projects requiring testing of the same areas frequently.
- Manual testing has proved software to be stable.
- Adequate time is available.
- The project is critical and large.
- Many virtual users will access the software, and thus load and performance testing are to be done.
- Project requirements do not change frequently.
However, not every component of the software can be automated.
Only the components that deal with user-based transactions (login and registration forms), GUI items, field validations, database connections, or those where many users interact with the software can be automated.
Levels of Testing
Different methodologies are included in levels of testing when we conduct software testing.
We are providing a brief description of each level of testing.
Two main levels of testing are:
- Functional testing
- Non-functional testing
1. Functional Testing
Functional testing involves evaluating the system’s compliance with the specified requirements by testing a completely integrated system.
Functional testing, a type of black-box testing in which testers provide input to the system and then examine the obtained results against the expected results from that particular function or component.
The following five steps are included in functional testing checks to ensure that every organization maintains strict software quality standards:
- Determining the functionality of the intended application.
- Creating test data based on the specifications.
- Creating expected outputs based on test data and requirement specifications.
- Executing test cases by writing different test scenarios.
- Comparing the obtained and expected results.
2. Non-Functional Testing
Non-Functional testing tests the non-functional attributes of any system specified in the requirement document.
Testing such as performance, user interface, security, etc., are non-functional testing.
Some commonly used types of non-functional testing are:
a) Performance Testing
It is used to identify performance issues in any system.
These are bottlenecks to a system’s performance and are not related to bugs present in the system.
Performance testing is crucial because it tests the system’s speed, capacity, stability, and scalability aspects.
Various causes contribute to poor performance, such as:
- Client-side processing
- Data rendering
- Network delay
- Load balancing in servers
- Database transaction processing
This testing can be qualitative or quantitative and is divided into two further testings:
- Load testing
- Stress testing
(i) Load Testing
No, not the physical load, but the data load or software accessing and data manipulating load is part of load testing.
In load testing, the testers apply a maximum load on the system in terms of a large amount of data with either normal or peak load conditions.
Load testing is automated testing in which virtual users using automated testing tools and scripts are defined.
Load testing tests the maximum capacity that can be handled by any system at peak time by increasing or decreasing the number of virtual users concurrently or incrementally.
Various automated tools are used for load testing, such as:
- IBM Rational Performance Tester
- Apache JMeter
- Silk Performer
- Visual Studio Load Test
(ii) Stress Testing
There are abnormal conditions that might occur in any system, and the way the system copes with those conditions is called stress testing.
Stress testing is used to apply load on a system, sometimes beyond the actual load limit, and exhaust all system resources to identify the system’s breaking point.
Different scenarios that can be tested using stress testing are:
- Turning database on and off
- Shutting down and restarting network ports randomly.
- Running various processes consuming CPU, server, memory, etc.
3. Alpha Testing
The combination of Unit testing, Integration testing, and acceptance testing is called alpha testing.
The developer or QA team generally performs alpha testing as part of the initial phase of testing in which the following aspects of the system are tested:
- Broken links
- Load times and latency
- Spelling mistakes
- Cloudy directions
4. Unit Testing
Unit testing, a type of manual testing, means testing individual isolated components of the system and ensuring that they are working correctly and according to the requirements.
Performed by developers, unit testing implies testing the source code’s individual units before formally passing on the software to the testing team.
Developers test their respective code units using test data, but this data is different from that used by the QA team.
However, it should be remembered that it’s impossible to identify and resolve every bug in the application because there is a limit as to how many test scenarios and test data can be created.
5. Integration Testing
After unit testing, integration testing is conducted.
The project moves on to the testing team, which integrates different system components to test if they work correctly.
Integration testing is also a type of manual testing.
Multiple tests are conducted until the whole manual testing process is concluded, and the system is integrated.
There are three approaches to integration testing:
- Bottom-up approach – In this, the testers combine smaller modules and go progressively towards higher-level combinations.
- Top-down approach – In this, the testers test higher-level units first and then progressively integrate and test lower-level modules.
- Bi-directional approach – When we work in a comprehensive software development environment, the bottom-up approach comes first, followed by a top-down approach.
6. System Testing
Once all the units are integrated, and a system is formed, the whole system is tested rigorously for more bugs and anomalies in its working using manual testing.
Performed by a specialized testing system testing team, this testing is completed once the system meets the specific Quality Standards.
Importance of system testing:
Functional and technical specifications are tested for the system.
The business requirements and application architecture can be tested, verified, and validated.
In the Software Development Life Cycle, System testing is the first instance where the entire system is tested as a whole.
The testing environment is created such that it mimics the actual production environment of the application.
7. Beta Testing
After alpha testing, beta testing or pre-release testing is conducted to test the system from the user’s perspective.
We all have once in our life been a part of or heard of the beta programs or beta versions of the applications that we use.
That is beta testing that requires human testers to test an application and find whether it is error free or not.
For example, many applications have beta versions available for which users can register on the Google Play store.
These beta versions are not stable and are deployed to users to check the application before its official release.
Users test the following processes in beta testing:
- Installation and Execution of the application, after which users send their feedback to the developers.
- This feedback is helpful to the developers to fix the issues.
- End User tests system crashes, typographical errors, and confusing application flow.
Using beta testing, a greater number of problems a user might face can be identified, and timely resolved to allow the project team to release a higher quality application to the general public.
8. Acceptance Testing
As the QA team performs the acceptance testing, QA testing makes it the most important type of testing and is done manually.
Also, there are legal and contractual requirements linked to acceptance testing.
The team has a set of pre-defined test cases and scenarios in which the application will be tested to gauge whether the system satisfies all the clients’ specifications and requirements.
This gives the project team a chance to test the application better for its accuracy and get an idea about how the system will perform in the end user environment.
Acceptance testing tests bugs or major issues that might cause system failure, cosmetic errors, spelling mistakes, and user interface gaps.
9. Regression Testing
Whenever we change any aspect of the software, it has a ripple effect on other software aspects.
Thus, with regressions testing, testers test the system for any new bugs or business rule violations caused by the old bug’s resolution.
Importance of regression testing:
- New changes are tested, and their effects on other areas of application are verified.
- The product can be marketed speedily.
- Regression testing does not compromise the timeline and increases the test coverage.
- Gaps in testing created due to new changes can be minimized using regression testing.
- Risk mitigation is possible when regression testing is performed.
10. Usability Testing
Usability testing, a black box testing technique, observes the users’ usage and operation, and identifies errors and improvements.
Standards such as IEEE std. 610.12, ISO-9241-11, ISO-9126, ISO-13407, etc., define the quality and usability standards for a system.
There are various definitions of usability, such as:
- According to Nigel and Macleod, usability is a quality requirement and can be measured using the outcomes generated by the computer system’s interactions.
The end user will be satisfied when the quality requirements are fulfilled, and the intended goals are achieved using the software.
Five factors define usability:
- Efficiency to use
- Learn the ability
- Memory ability
- Nielsen gave these five factors. He also said that if any product possesses the above factors, then its usability is good.
- In 2000, Molich gave another definition of usability in which he stated that every user-friendly system must fulfill the following goals:
- Easy to learn
- Easy to remember
- Efficient to use
- Satisfactory to use
- Easy to understand
It can be considered one of the subparts of system testing. It also tests entire software from a usage perspective in different environments such as browsers, operating systems, and computer hardware.
Portability testing tests the reusability and movability of the software using the following strategies:
- Building an executable file (.exe) so that the application can run on different platforms.
- Transferring installed software between computer systems.
- There are certain pre-requisites for portability testing:
- There should be an established test environment.
- The associated components should be unit tested.
- The portability requirements of the software must be considered while coding it.
- The system should be integrated, i.e., integration testing should have been performed.
In software testing, there are mainly three classifications of testing methods. They are:
- Black Box Testing
- White Box Testing
- Grey Box Testing
Before we understand these in detail, let us first understand the difference between them.
|#||Black box testing||White box testing||Grey box testing|
|1.||No Knowledge about the internal workings of the application is known to the tester.||Full knowledge about the internal workings of the application is known to the tester.||Limited knowledge about the internal workings of the application is known to the tester.|
|2.||It is done by trial and error.||Testing of data domains and internal boundaries is better.||Testing of data domains and internal boundaries is possible if its known.|
|3.||End users or testers and developers assuming the role of end users perform this testing.||Only testers and developers normally do this testing.||End users or testers and developers perform this testing.|
|4.||This method consumes the least amount of time and is exhaustive.||This method consumes the most amount of time and is very exhaustive.||This method consumes only partial amount of time compared to white box testing and is exhaustive.|
|5.||It is done based on external expectations without knowing anything about system’s internal behavior.||Test data can be designed according to the internal workings of the software.||Using high-level database and data flow diagrams, testing can be done.|
|6.||It is not suitable for algorithm testing.||It is suitable for algorithm testing.||It is not suitable for algorithm testing.|
|7.||Other names: Data driven testing, closed box testing, functional testing.||Other names: structural testing, clear box testing, code-based testing.||Other names: translucent testing.|
1. Black Box Testing
Unlike the black box stored in aircraft and other vehicles, this black box testing actually means blindly going for testing.
We know this is a layman way of defining it, but when a tester tests the software without gaining any prior knowledge about the software’s internal working, it is called black-box testing.
Being oblivious to the system architecture and having no access to code, a black box tester interacts only with the system’s user interface.
The tester provides inputs to the system and then examines the generated outputs cross-referencing them with expected outputs without knowing how that output is being generated.
- There is no requirement to access the code.
- Moderately skilled testers can easily test the application without understanding the implementation of the software or the source code or Operating System.
- Black box testing is best suited for segments of large code.
- It separates the user’s and developer’s perspective using visibly defined roles.
- Since the tester has limited knowledge about the systems’ working, the testing is inefficient.
- Test case designing is difficult.
- Since there are a limited number of scenarios, the test coverage is also limited.
- As the tester cannot target error-prone code segments, black-box testing gives only blind coverage.
2. White Box Testing
Contrary to black-box testing, white box testing, also known as glass box testing or open box testing, is more focused on investigating the code’s internal logic and structure.
The tester can identify which code segment is not behaving properly by looking inside the source code.
- White box testing optimizes the code.
- It provides maximum coverage as the tester is aware of the source code.
- Due to code availability, it is easy to determine appropriate input data to test the system properly.
- Hidden defects can be brought out by removing unnecessary lines of code.
- Specialized tools such as code analyzers and debuggers are required in white box testing, making it difficult to maintain.
- Since a skilled tester is required in this testing, the overall cost of testing is increased.
- Although white-box testing has extensive coverage, it is sometimes impossible to check every nook and corner of the code for possible errors.
3. Grey Box Testing
Yes, you guessed it right, white + black = grey.
The grey box is nothing but a mix of black and white testing in which a tester has limited knowledge about the system’s internal working.
Unlike black-box testing and white box testing, in grey box testing, the tester can access the information required for designing the database and documents for the system.
With this limited knowledge, a tester can prepare better test data and better test scenarios during test planning.
- Grey box testers rely on interface definitions and functional specifications instead of relying on the limited information available to them.
- The testing is done from the users’ perspective, not the developers’ perspective.
- It provides the best of the black box and white box testing.
- Excellent test scenarios can be designed around data type handling and communication protocols with limited information.
- If the test cases have already been run, tests can be redundant.
- It is unreasonable and unrealistic to test every input stream possible, due to which many program paths might go untested.
- There is limited ability to go over the code as the source code access is limited.
Estimating the software testing efforts is one of the Software Development Life Cycle’s major tasks as it helps in maximum coverage, proper management, and allocation procedures.
Some of the techniques that are useful for estimation are:
1. Test Point Analysis
For the function point analysis of acceptance or black-box testing, Test Point Analysis is used.
Important aspects of Test Point analysis are:
2. Functional Point Analysis
Analyzing functional user requirements in the software is called Functional Point Analysis.
There are the following categories:
- Internal files
- External files
3. Mark-II Method
Measuring and analyzing estimations from an end users functional perspective is done using Mark-II Method.
Its procedure is:
- Determine the user’s viewpoint
- Identify the purpose and the type of count.
- Define the boundary of the count
- Identify logical transactions
- Identify data entity types and categorize them.
- Count input data element
- Count functional size
At the end of this software testing guide, we bring you the most important topic without which any software testing process is incomplete – Documentation.
Having proper documentation of every testing artifact developed before or during software testing is essential.
Documentation helps the testers and other team members to estimate the testing efforts, coverage of the test, requirements fulfilled, etc.
There are four basic documents involved in software testing:
- Test Plan
- Test Scenario
- Test Case
- Traceability Matrix
The test plan document outlines the testing strategy, resources to be used, environment to be created, scheduled testing, and limitations of testing the application.
Since it is essential, it is handled by the QA team.
The following are included in a test plan:
- Introduction to Test Plan Document
- Schedule for tasks and milestones
- List of features that are to be tested
- Resources that will be allocated to the application for testing
- Assumptions made while testing
- Risks involved with the testing process
- List of deliverables that are to be tested
- List of test cases
- What approach to be used for software testing
Test scenarios are single-line statements that ensure that the testing of all process flows is done end to end. They notify which will be tested.
Depending on magnitude and complexity, any area in the application can have any number of test scenarios, i.e., it can be one, two, and up to a few hundred.
Sometimes the terms’ test cases’ and ‘test scenario’ are used interchangeably, but they have a basic difference.
A test case has one step, while a test scenario has several steps and comprises various test cases.
Also, test cases are independent of each other, while test scenarios need to be executed sequentially.
Test cases are a set of steps, inputs, or conditions used to perform testing tasks whose sole intent is to determine the failure or success of any system and keep track of the system’s testing coverage.
Test cases can be functional, error, physical test cases, negative, logical test cases, UI test cases, etc.
Sometimes testers create multiple test cases for a single software, in which case they are called test suites.
The following are the components included in every test case:
- Test case ID
- Product version
- Product module
- Expected outcome
- Actual outcome
- Revision history
Requirement Traceability Matrix has three main goals:
- Making sure that the software is developed following the mentioned requirements.
- Tracing the documents developed in various SDLC phases.
- Finding root causes for bugs.
In Traceability Matrix, every requirement is linked with its associated test case along with a Bug ID so that the corresponding requirement can be tested and bugs identified and mapped. It generates a comprehensive bug report.
It can be done in two ways:
- Forward tracing – Moving from Requirements to Design phase or Coding phase
- Backward tracing – Moving form Coding phase to Requirements
Testing is crucial to every software development process and does not require a fully developed product.
It is a very common myth that testing is expensive or time consuming, which is not true.
Many facts related to testing bust these common myths and have proven that testing is actually not done in every phase of SDLC.
Remember that not everyone can successfully test software, and no software can ever be completely tested and deemed bug-free.
For delivering the best quality product to your client, make sure that your organization adheres to the quality guidelines and requirements document because delivering quality is the responsibility of all, not just the testers.
Even though you might be interested in automation testing manual testing, is also very important in real life as every application needs not to automate everything.
You can find manual testing tutorial and software testing tutorial to enhance your manual testing concepts.
At last, we’d say that as a tester, your job is not limited to finding bugs. You are the domain expert of that software.