A measurement based software quality framework

In this report we propose a solution to problem of the dependency on the experience of the software project quality assurance personnel by providing a transparent, objective and measurement based quality framework. The framework helps the quality assurance experts making objective and comparable decisions in software projects by defining and assessing measurable quality goals and thresholds, directly relating these to an escalation mechanism. First results of applying the proposed measurement based software quality framework in a real life case study are also addressed in this report.

💡 Research Summary

The paper addresses the well‑known problem that software quality assurance (QA) decisions in many projects rely heavily on the personal experience of QA personnel, leading to inconsistency, lack of transparency, and difficulty in meeting industry standards. To overcome this, the authors propose a Measurement‑Based Software Quality Framework (MSQF) that combines a conceptual model (MSQC) with an operational process (MSQP). The framework’s core idea is to replace intuition‑driven go/no‑go decisions with objective, repeatable, and organization‑wide measurements.

The MSQC concept defines a set of criteria that the framework must satisfy: (1) provide objective quality assurance throughout the project lifecycle, and (2) comply with relevant industry standards such as Automotive SPICE (MAN.6) and CMMI‑DEV (Measurement and Analysis). These high‑level criteria are broken down into concrete sub‑criteria (C1.1‑C1.5 for objectivity, C2.1‑C2.2 for standards compliance).

The MSQP operationalizes the concept through five sequential activities: (1) define/refine measurement goals, (2) define/refine metrics and thresholds, (3) plan measurement activities, (4) measure and analyze data, and (5) act on the results. A distinctive element is the notion of “abstract milestones” – predefined checkpoints (e.g., design review, release candidate) where metric values are sampled and compared against thresholds. Deviations trigger a three‑level escalation mechanism (levels 0‑2) whose severity is linked to the percentage deviation from the target. The process also mandates regular review and adjustment of thresholds, and mandates sharing of measurement results across the organization.

To validate the framework, the authors implemented it at ThyssenKrupp Presta Hungary (TKPH), an automotive supplier developing electronic steering systems. They tailored the MSQP to TKPH’s context by: (a) defining a unified set of metrics for each Automotive SPICE process area, (b) establishing a common measurement database and visualization tool (DataDrill Express), (c) creating new work products (quality assurance strategy, project quality plan, release audit report, training material), and (d) assigning specific roles (Project Quality Leader, Quality Department Leader, etc.) with clear escalation responsibilities.

In a confidential “Project A”, the framework was applied from project start to delivery. At the beginning, a wide tolerance (±30 %) was allowed (Escalation Level 2). As the project progressed, tolerances were tightened to ±20 % (Level 1) and finally to ±10 % (Level 0) at the final milestone. When metric values exceeded the defined thresholds, the appropriate escalation level was automatically triggered, notifying the responsible manager and prompting root‑cause analysis and corrective actions. The case study demonstrated that the MSQF could objectively assess quality, provide early warning of potential issues, and enforce consistent decision‑making across the project.

The authors evaluate the framework against the predefined criteria and find full compliance: all objective quality criteria (C1.1‑C1.5) are satisfied, and the framework aligns with Automotive SPICE MAN.6 and CMMI‑DEV Measurement and Analysis requirements (C2.1‑C2.2). They acknowledge limitations: the validation is limited to a single organization and a single project, and metric selection may still involve subjective judgments. Nonetheless, the study shows that a structured, measurement‑driven approach can significantly reduce reliance on individual experience, increase transparency, and support industry‑standard compliance.

In conclusion, the MSQF offers a practical, repeatable method for organizations seeking to institutionalize objective software quality assurance. Future work should explore broader industrial adoption, scalability to larger portfolios, and deeper automation of data collection and analysis to further enhance the framework’s efficiency and impact.