Being effective means we focus on writing the right tests. Software testing is all about trade-offs. Testers want to maximize the number of bugs they find while minimizing the effort required to find the bugs. How do we achieve this? By knowing what to test.
Being systematic means that for a given piece of code, any developer should come up with the same test suite. Testing often happens in an ad hoc manner. Developers engineer the test cases that come to mind. It is common to see two developers developing different test suites for the same program. We should be able to systematize our processes to reduce the dependency on the developer who is doing the job.
Boundary testing: testing for edge cases.
Example-based testing: testing with a single example input.
Exhaustive testing is impossible. Testing all possible situations in a software system might be impossible even if we had unlimited resources. The life of a tester involves making trade-offs about how much testing is needed. We need to use effective tests.
Testing shows the presence of defects, it does not show the absence of defects. Not test team can achieve 100% defect detection effectiveness.
Test early. This let the tests guide the design, get feedback as early as posible, find bugs when they are cheapest to fix, find bugs when they have caused least damage, etc. The cost involved in fixing such defects in early stages is very less when compared to those that are found during the later stages of testing.
Knowing when to stop testing. Our goal should always be to maximize the number of bugs found while minimizing the resources we spend on finding those bugs.
Variability is important (the pesticide paradox). There is no single testing technique that you can always apply to find all possible bugs. Different testing techniques help reveal different bugs.
Pesticide paradox: every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual. Testers must use different testing strategies to minimize the number of bugs left in the software. Re-running same test suite again and again on changing program gives false sense of security. To overcome this, the test cases need to be regularly reviewed & revised, adding new & different test cases to help find more defects. Testers cannot simply depend on existing test techniques. He must look out continually to improve the existing methods to make testing more effective.
Bugs happen in some places more than others. Note that bugs are not uniformly distributed (defect clustering).
Testing will never be perfect or enough. While we may find more bugs by simply testing more, our test suites, however large they may be, will never ensure that the soft-ware system is 100% bug-free. They will only ensure that the cases we test for behave as expected. “You cannot test everything”.
Testing is context-dependent. The context plays an important role in how we devise test cases. For example, devising test cases for a mobile app is very different from devising test cases for a web application or software used in a rocket. You might use a different approach, methodologies, techniques, and types of testing depending upon the application type.
Verification is not validation. Note that a software system that works flawlessly but is of no use to its users is not a good software system. “Coverage of code is easy to measure; coverage of requirements is another matter.” Software testers face this absence-of-errors fallacy when they focus solely on verification and not on validation. “Verification is about having the system right; validation is about having the right system.”
It is possible that software which is 99% bug-free is still unusable. This can be the case if the system is tested thoroughly for the wrong requirement. Software testing is not mere finding defects, but also to check that software addresses the business needs. The absence of Error is a Fallacy i.e. Finding and fixing defects does not help if the system build is unusable and does not fulfil the user’s needs & requirements.
<aside> ℹ️ Defect clustering means that a small number of features have caused the majority of quality issues in an application. Basically, the defects are not distributed uniformly across the entire application, rather defects are concentrated or centralized across two or three functionalities. This is the application of the Pareto Principle to software testing: approximately 80% of the problems are found in 20% of the modules. If the same tests are repeated over and over again, eventually the same test cases will no longer find new bugs.
</aside>