Since 1976, (scientific) minds have been arguing about a conceptual model around the topic of software product quality. A number of authors proposed different approaches [astrotech.io+2022], until 1991 the ISO organization took over and began publishing its vendor- and product neutral standards ISO-9126. In 2011, that was replaced by the ISO-25010, which is still in active use.
Too bad that active use does not correspond to practical or usable…
This article presents a few quality models and comes to a (potentially) surprising conclusion…
Why do we need a Quality Model
Quality consists of many characteristics. Therefore, quality is usually captured in a model or terminology that contains these characteristics and their relationships. Such quality models show what people consider important when talking about quality [Jamwal+2009].
Quality models define a common language for terms related to product, system and software quality.
A Bit of History
In 1976, [Boehm+1976] published their quality model, including inner qualities like structuredness and legibility. Take a look at the original paper, it is a nice example of historical typesetting, some graphs look like drawn with a ball pen :-).
[McCall+1977] suggested to model quality as a hierarchy of terms, whose first level consists of Operation, Revision and Transition. To revision, he counted e.g. Maintainability, Flexibility and Testability.
Next [Grady+1992] at Hewlett-Packard published FURPS and its extension FURPS+, the latter found proper acceptance in practice due to its integration into the (once popular) IBM Rational Unified Process [Eeles+2005].
The ISO organization then took up the issue, publishing vendor- and product-neutral standards since 1991, starting with ISO-9126, which “lasted” for 20 years, and was superseded in 2011 by ISO-25010, which is still in effect today. Both were considered the conceptual reference for software quality, and have gained considerable currency in practice. In my humble opinion, this is largely due to the fact that the ISO bodies define the terms used quite properly. Therefore, they can be forgiven for simply ignoring (or forgetting?) some really relevant quality characteristics. I also liked quality features of the VOLERE Requirements Template, which, however, have found less acceptance than the (omnipresent?) ISO model.
Quick-Check of Software Quality Models
| Authors / Name | Year | Summary | Nr of Quality attributes |
|---|---|---|---|
| Boehm | 1976 | Hierarchical model, 3 levels. Top-level qualities: utility, maintainability, portability. See below for more info. | 23 |
| McCall | 1977 | Hierarchical model, 2 levels. Top-level areas: operation, revision, transition. Predecessor of current quality models. | 11 |
| ISO-9126 | 1991 | Hierarchical model, 2 levels, 6 top-level qualities. No safety, security underrated, disputable terminology | 27 |
| R. Grady, FURPS | 1992 | Single level with functionality, usability, reliability, performance, supportability. Lacks operational qualities and safety | 6 |
| IBM FURPS+ | 1999 | Add lots of sub-characteristics to FURPS, addressing requirements in general. Part of the (overly complex) Rational-Unified process | 30 |
| VOLERE | 1999 | Integrated in sophisticated template for requirements. Combines qualities and constraints | 8 |
| ISO-25010 | 2011 | Supersedes ISO-9126. Hierarchical model with 8 top-level qualities. Adds security as top-level. Widespread, AFAIK. Disputable definitions of terms. | 32 |
| Bass et al. | 2022 | One level, see this article. Practical, with a few rough edges. | 10 |
| ISO-25010, draft 2022 | 2022 | Proposal to add safety and change a number of terms and definitions. Still disputable terminology, overly complex for day-to-day use. | 39 |
Boehm Software Quality Model ([Boehm+1976], [Boehm+1978])
Barry Boehm supposedly was the first to propose a hierarchical model of software quality, with three top-level qualities, that are refined on two further levels. When looking into the details of his model, one will notice that several attributes from level-2 and level-3 are referenced multiple times, making this model more of a graph than a tree. In my opinion, this is perfectly realistic, but the later ISO-standards got rid of this pragmatic feature.
Please consider this model within the historical perspective: No private person owned a computer, as the personal computer hadn’t been invented. Only very few companies used them, and programming computers was a black art that only a few magicians could apply.
With this perspective, the Boehm model was incredibly farsighted, as it included inner quality attributes like legibility and structuredness (which got lost in modern models like ISO)
FURPS+ for Classifying Requirements ([Grady+1992], [Eeles+2005])
Robert Grady and Peter Eeles (from Hewlett-Packard) proposed the following schema for classifying requirements:
- Functionality
- Usability is concerned with characteristics such as aesthetics and consistency in the user interface.
- Reliability is concerned with characteristics such as availability (the amount of system “uptime”), accuracy of system calculations, and the system’s ability to recover from failure.
- Performance is concerned with characteristics such as throughput, response time, recovery time, start-up time, and shutdown time.
- Supportability is concerned with characteristics such as testability, adaptability, maintainability, compatibility, configurability, installability, scalability, and localizability.
The “+” represented several additional categories, such as design-, implementation-, interface- and physical- requirements.
Despite the lack of security, safety, resource-consumption and operational qualities, this model was (and likely still is) intensively used in practice.
The ISO Standards 9126 and 25010 ([ISO+25010])
Design-by-Committee is regarded as a suboptimal approach to development - and in my opinion that happened to the ISO-standards for software quality. Instead of just unifying definitions from Boehm and FURPS, the invented a kind of metamodel, distinguishing between five different areas of quality (see figure below). Divisions? Model and Management? Measurement and Evaluation? These distinctions seem overly complicated from my practitioners’ viewpoint.
I really wonder if anybody working in development projects (except me) ever took the time to read through all those standards. (Ok, I exaggerated, I read only 25010, but that was hard-enough work).
Apart from being in wide use, these models really lack practical applicability, although the (still unofficial draft) update from November 2022 somewhat improves the situation). I covered a few downsides in my article on ISO-25010 shortcomings. To summarize:
- Overly many terms, with a lot of overlap.
- Critical qualities (e.g. safety, scalability, operational properties) missing from the official version
- Despite proposing 30+ attributes, code- and architectural qualities are missing
- No examples of how the ISO-model might be applied to real-world problems
But the standards would not have survived as long if they didn’t contain some goodies: The ISO defines all terms contained within the standard, and these definitions are available free-of-charge. Although some of these definitions are quite academic, they provide a nice starting point, and help to avoid conflict when discussing with different stakeholders.
SEI Quality Model
A quality attribute (QA) is a measurable or testable property of a system that is used to indicate how well the system satisfies the needs of its stakeholders beyond the basic function of the system. You can think of a quality attribute as measuring the “utility” of a product along some dimension of interest to a stakeholder.
For Bass and his colleagues, quality consists of 10 major properties, depicted in the following overview.
Wow, what a difference - only 10 areas instead of 40 terms in the ISO 25010 standard. They differentiate these qualities into two categories:
We will focus on two categories of quality attributes. The first category includes those attributes that describe some property of the system at runtime, such as availability, performance, or usability. The second category includes those that describe some property of the development of the system, such as modifiability, testability, or deployability.
For their work, a huge “thank you” goes to Len Bass and his colleagues, for providing an alternative to the ISO-25010 approach. They did a great job, and their book is definitely recommended to read in full (please use the fourth edition, as it was heavily updated and modernized!)
Now comes my “but”: Their approach can IMHO be further improved. Let me consider the SEI-Model step-by-step:
- Availability is surely an important goal for many systems, but I know of many systems (or services) that need to work only on certain occasions, and can be turned off the rest of the time. Therefore, I suggest making “reliable” the top-level goal, and availability is part of that.
- Deployability is great for online- and mobile systems, that are built in highly automated continuous integration workflows. Real-time and embedded systems are (even in 2023) deployed less often, and with less automation. Next thing is, that once the system is deployed, we definitely need to administer, configure and monitor it. Therefore, I suggest making “operable” a top-level quality, and consider “deployable” a sub-goal of that.
- Energy-efficiency is in many people’s mind, partially due to the incredible increase in energy-consumption by IT infrastructure worldwide. But energy is just one critical resource: What about water, and carbon-dioxide? Again, I prefer a slightly more general term, “resource-efficient”. That even makes “Performance” redundant.
- Modifiability sounds important, but here again I consider additional aspects: Sometimes I don’t want to modify a system, just install it on another operating system. Or configure a new database for it. In my opinion, the general term is “flexible”. That even allows for removing “Integrability” from the SEI wishlist.
arc42 Quality Model
Trying to learn from its predecessors (or, as others have called it “it’s easy to stand high on the shoulders of giants”), arc42 proposes a simple, efficient and practical model.