Software Quality Assurance (QA) has grown wildly complex and specialized, while at the same time becoming increasingly ineffective. It's time to question a couple key assumptions and to establish and simple, clean, unifying framework to get the results we want from SQA: "get it right" and "keep it right."
Software QA Today
There's a huge amount going on with software QA. For example, the American Society for Quality has a Certification for Software Quality Engineer.The test has about 160 subject areas. Here is a sample of just four of them:
If you accept the assumptions about software development that underly most modern QA, it doesn't seem completely insane. However, I challenge several of the core assumptions.
It is a historical fact that much of modern software development theory, including QA, is based on the notion that building software is somehow like a manufacturing process. There are a couple edge cases in which this assumption is useful; but for the general case, it's just wrong and should be discarded.
It is similarly clear that most modern software development theory, including QA, is based on the notion that the root fact of software, the fact from which everything else grows, is a set of requirements. There are important cases in which this assumption is useful (a large number of them are algorithms, think Knuth); but for most software, it causes more trouble than anything else, and should be forgotten like a bad dream.
Finally, much software is developed with the notion of predictability as central. This is a choice organizations make. If they want predictability, let them have it; it's a valid (if mostly brain-dead) choice. But for most of the software that makes a difference, speed is more important than predictability.
Instead of these assumptions, it is more fruitful in most cases to assume the following:
- Software is primarily a design process
- The requirements of the software evolve as you build it
- Speed is more important than predictability.
Core Software QA Function: Keep it Right
"Keep it Right" is one of the two major categories of software QA. It has its own distinct methods. It can and should be largely automated.
The core, transformative observation here is that your software need not (yet) be correct. When you alter the software, the main thing you want is to avoid changing its behavior (except as intended). The killer in all regression testing software is that you have to go function by function and write scripts to make sure the "right" thing happened. So your work is proportional to the size and complexity of your program.
Once you understand the role of "Keep it Right," you see that all you need to do is compare what the software did before with what the changed version does. There are always way to accomplish this. The key observation here is that writing comparisons (think diff) is something you do just once, for the point at which the differences in the software are being compared (for example, the UI or the DBMS). So your work is independent of the size and complexity of your program.
The key method of "keep it right" is to replace "correctness testing" with "comparison testing." This can be completely automated; it reduces programmer overhead; and it does what users do, viz., ask the question "what's different?"
Core Software QA Function: Get it right
"Get it Right" is one of the two major categories of software QA. It has its own distinct methods. It cannot be automated to any significant extent.
Scripts or automation of any kind are irrelevant to "get it right." Instead of having product people write requirements that someone down the line eventually turns into test plans and test cases, the people who would normally write requirements watch the software's behavior as it grows in capability and guide its development interactively. They see whether this iteration of the software does what they expected, and they also watch their own reactions as they use it, learning from those reactions. Outside people may also play a part in this process.
Meanwhile, of course, "keep it right" is running all the time, assuring that only forward steps are taken, and that unintended side-effects are caught so they can be corrected. Once the new round of changes have been "gotten right," they become part of the base set that is "kept right" using the comparison-based test-for-change method.
Keep it Right and Get it Right alter the organization, methods, work and results not just of QA, but of the entire software-producing organization. They change who does what and how they do it. When done reasonably well, they align quality efforts with customer expectations, which nearly always are based on the hopeful assumption that at least you will avoid breaking what used to work. At least as important, your product managers and programmers can "goal seek" their way rapidly to an effective solution with minimal overhead.