Work

Software Quality Assurance

Home > Work > Software Quality Assurance

When focus is quality, quality rises and costs fall

When focus is cost, costs rise and quality falls

This post introduces some high-level quality objectives common to most IT projects. It defines quality, and presents common quality objectives (Wiegers, 2003) as user stories. These can (and should) be used by project teams to derive specific quality requirements applicable to their projects.

Introduction to Quality

The International Organisation for Standardisation (specifically ISO 9000:2015) defines quality as the degree to which a set of inherent characteristics fulfils requirements. In this definition the word degree implies that quality exists on a continuum – starting at zero and moving toward, perhaps, infinity. This inference is ambiguous in that the quantifications of “good” and “bad” are not qualified.

Another popular definition of quality, as defined by Joseph Juran, is fitness for use, with fitness and use being crucial to the understanding of quality. Consumer and provider interpretations of these two terms often conflict, and so cannot be left to interpretation. Organisations often bound these two terms with specifications for their products and services.

Specifications may be explicit or implicit. Explicit specifications are prepared by the provider, and are given to the consumer. Implicit specifications are not defined, but are understood to be necessary. Examples include safety, security and fault tolerance requirements.

Quality Assurance User Stories

Availability

As a		user
I want		the system to be at least n1 percent available between [start time] and [end time] on [days of week] and at least n2 percent available on [days of week] local time
		I can complete my assigned tasks during normal working hours, and occasionally use the system out of hours.
Measure:		% Availability = MTTF / (MTTF + MTTR) * 100 Where MTTF is mean time to failure, and MTTR is mean time to repair

Efficiency

As a		User
I want		at least n percent of the processor capacity and RAM available to the application to be unused during planned peak load conditions
So that		unanticipated demand spikes and future growth do not impair acceptable performance
Measure		How well a system utilizes processor capacity, storage space, memory, or communication bandwidth (Davis, 1993).

Flexibility

As a		business
I want		to introduce x feature into the system, including code , modifications and testing, with no more than n hours of labor.
So that		the system can evolve through a series of successive releases or by evolutionary prototyping
Measure:		How easy it is to add new capabilities to a product, to change existing capabilities, or to remove redundant or obsolete capabilities.

Integrity

As a		business, user or administrator
I want		to block unauthorized access to system functions, prevent information loss, and protect the system from tampering
So that		I can protect the interests of [business], and the privacy and safety of [entities].
Measure:		Integrity is a constraining business requirement, and as such has no tolerance for error. Ensure that a threat model exists (I personally like STRIDE), and includes mitigations to reduce probability, and contingencies to reduce impact.

Interoperability

As a		user or administrator
I want		to submit data to, and access data from [location(s), external system(s) and/or service(s)], and import and export data in [format(s)]
So that		I can easily exchange data and services with other systems.
Measure:		How easy it is to share or access data or services with or from other systems.

Reliability

As a		user
I want		n percent of operations to complete correctly within x [time unit], and for the system to operate for a period of y [time unit] before failing
So that		BAU is maintained, and the impact of system failures is minimized.
Measure:		The period of time that software executes without failure (Musa, Iannino, and Okumoto, 1987); the percentage of operations that complete correctly.

Robustness

As a		user
I want		the system to continue to function properly when confronted with invalid inputs, defects in connected software or hardware components, or unexpected operating conditions
So that		the system can recover gracefully from problem situations and is forgiving of user mistakes.
Measure:		The degree to which the system continues to function with confronted with invalid inputs, defects in connected software or hardware components, or unexpected operating conditions.

Usability

As a		user
I want		to spend x amount of effort in preparing input for, operating and interpreting the output of the system
So that		functions are easy to complete, and complete within acceptable timeframes.
Measure:		The effort required to prepare input for, operate and interpret the output of the system (Constantine and Lockwood, 1999).

Developer Stories

Maintainability

As a		developer
I want		an average time of n1 [time unit] to fix a problem, and to fix n2 percentage of problems correctly
So that		I am encouraged to take actions to the intent, and not the letter, of the goal.
Measure:		The average time required to fix a problem, and the percentage of fixes that are made correctly.

Portability

As a			developer
I want			to select design and coding approaches that enhance the system’s portability
So that			the system isn’t tied to a single technology stack or vendor’s products.
Measure:			The effort required to migrate a piece of software from one operating environment to another.

Reusability

As a		developer
I want		to develop modular, well-documented software that is independent of a specific application and operating environment, and somewhat generic in capability
So that		the relative effort required to convert a software component for use in other applications is minimal.
Measure:		Reusability is difficult to quantify. Identify which components are suitable to use in other applications (typically components that address cross-cutting concerns such as monitoring, logging, communication, and authentication/authorisation), and estimate the relative effort required to convert those components for use in other applications.

Testability

As a		developer
I want		to write software modules in which the maximum cyclomatic complexity does not exceed n
So that		the number of logic branches and loops are kept within a range that makes them easier to test, to understand and to maintain.
Measure:		Count cyclomatic complexity. Do not measure automated test coverage (remember that the objective is quality, and not coverage).

References

Chemuturi, Murali. 2011. Mastering Software Quality Assurance. Best Practices, Tools and Techniques for Software Developers. J. Ross Publishing, FL.
Constantine, Larry L., and Lockwood, Lucy A. D. 1999. Software for Use: A Practical Guide to the Models and Methods of Usage-Centred Design. Reading, MA: Addison-Wesley.
Davis, Alan M. 1993. Software Requirements: Objects, Functions, and States. Englewood Cliffs, NJ: Prentice Hall PTR.
Deming, William E. 1986. Out of the Crisis. MIT Centre for Advanced Engineering Study.
Musa, John D., Iannino, Anthony, and Okumoto, Kazahira. 1987. Software Reliability: Measurement, Prediction, Application. NY: McGraw-Hill.
Wiegers, Carl E. 2003. Software Requirements (Second Edition). Microsoft Press, WA.
Ecker, Robert. 2017. The Problem With High Test Coverage.

< Back | ^ Back to top