ISO/IEC 14756: Measurement and rating of performance of computer-based software systems
This international standard specifies methods for measuring and evaluating the user-oriented performance of computer-based software systems (CBSS). It provides a comprehensive framework for specifying performance metrics, test procedures, and evaluation methods to assess how well software systems perform under various conditions.
The standard focuses on measuring performance from the user’s perspective, emphasizing response times and throughput rates that directly impact user experience. It introduces the concept of “user-oriented performance” which considers the system’s behavior as experienced by end users rather than just technical metrics.
Quality Attributes Required or Emphasized
The standard primarily focuses on performance-related quality attributes, but also addresses several related aspects:
| Attribute | Relevance in ISO/IEC 14756 |
|---|---|
| Performance Efficiency | Core concern. Provides frameworks for evaluating overall system performance under various loads. |
| Time Behaviour | Central focus. Defines methods for measuring response times and processing durations. |
| Resource Utilization | Addresses how efficiently the system uses hardware resources during operation. |
| Throughput | Specifically measures the rate at which the system can process transactions or data. |
| Capacity | Covers the system’s ability to handle increasing workloads while maintaining acceptable performance. |
| Reliability | Indirectly addressed through consistent performance under specified conditions over time. |
| Latency | Focuses on measuring delays in system response to user actions. |
Key Concepts
The standard introduces several important concepts:
- User-oriented performance measurement: Focuses on the user’s experience rather than just technical metrics
- Representative operation profiles: Defines realistic usage patterns for testing
- Statistical evaluation methods: Provides frameworks for analyzing performance data
- Reproducible test environments: Emphasizes controlled testing conditions for reliable results
References
Official Standards Documents
- ISO/IEC 14756:1999 - Measurement and rating of performance of computer-based software systems
Related Standards
- ISO/IEC 25010 - Systems and software Quality Requirements and Evaluation (SQuaRE)
- ISO/IEC 25023 - Systems and software Quality Requirements and Evaluation (SQuaRE) — Measurement of system and software product quality
Additional Resources
- Smith, C. U., & Williams, L. G. (2002). Performance Solutions: A Practical Guide to Creating Responsive, Scalable Software. Addison-Wesley.
- Jain, R. (1991). The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling. Wiley.