Traceability Score™ – An Empirical Way to Reduce the Risk of Late Requirements
|
In this blog, learn how a Traceability Score™ can act as an empirical way to reduce the risk of late requirements.
Traceability Score™ – An Empirical Way to Reduce the Risk of Late Requirements
Executive Summary
One of the main causes of rework, delays, and cost overruns in product development is the creation of new requirements late in the process. This is a well-known risk in product development, but what management practices can empirically be shown to reduce this known risk?
Using our proprietary database of metadata from over 50,000 complex product development projects, we were able to determine that the Traceability Score™ is an empirical method to reduce late requirements. In fact, teams that maintain a high Traceability Score reduce the burden late requirements have on their project by 67% compared to teams with low traceability scores.
- With this knowledge, our recommendation is that practitioners measure and monitor the Traceability Score™ of their projects to resolve issues early and ensure that the risk of late requirements is kept to a minimum.
Dataset Background
Jama Software® has the world’s largest, live dataset of engineering process performance with over 50,000 engineering projects updated and growing continuously. Leveraging this dataset, it is now possible to determine empirically which management practices improve the performance of the product development process. To learn more about our benchmarking, please review our Traceability Benchmarking Report.
The Empirical Questions
In this analysis we will explore three key questions:
- What are late requirements?
- How do late requirements negatively impact projects?
- Does maintaining a high Traceability Score reduce the risk of late requirements?
What are late requirements?
For the purpose of this analysis, we define “late requirements” as those requirements created after the completion of a project’s requirement decomposition phase which we estimate as spanning the middle 50% of all requirement creation activity (creation and refinement). To illustrate what late requirements look like, we show two actual projects below with requirement activity plotted over time.
Requirement Creation Over Time
In the Timely Project, requirement creation occurs in a defined requirement decomposition phase to form a necessary and sufficient set of requirements, with very few requirements being added after the fact (e.g. in fig (a), only 1.3% of requirements created late). In the Late Project’, requirement creation bleeds into future phases of the project, leading to a significant amount of late requirements (e.g. in fig (b), 9.2% of requirements are created late).
RELATED: Requirements Traceability Benchmark
How do late requirements negatively impact projects?
We can measure the outsized burden late requirements have on project teams, which we have illustrated for our two projects below. We define late requirement burden as the total number of requirement activities (creation and refinement) attributed to late requirements as a percentage of all requirement activity.
Impact of Late Requirements on Project Team Activity Burden
In the Timely Project, minimal late requirements enable better forecasting of project completion, and limits the rework and cost brought on by late requirements (e.g. in fig (c), late requirements only create an additional 8% burden).
In the Late Project, the high volume of late requirements makes it much harder to forecast project completion as the scope of the project is constantly changing, and project teams need to accommodate the late requirements (e.g. in fig (d), late requirements contribute an additional 31% burden).
Unsurprisingly, this additional burden of late requirements has an impact during testing for requirement validation. In our actual project examples, the Late Project has a test failure rate over 3x that of the Timely Project.
RELATED: Unlocking The Power of Live Traceability with Jama Connect®
Does maintaining a high Traceability Score reduce the risk of late requirements?
A core theorem of Systems Engineering is that maintaining high requirement traceability from the start of a project reduces the risk of late requirements and negative product outcomes. With our project dataset we can now test this theorem empirically. We define traceability as a measure of a project’s ‘expected’ traceability that has actually been established and calculate the Traceability Score as follows:(1)
For our example projects, the Timely Project achieved a Traceability Score over 6X that of the Late Project; suggesting that maintaining a high Traceability Score throughout the project reduces the risk of late requirements.
To further determine if Traceability Score correlates to late requirements, we divided our dataset of projects into quartiles based on their Traceability Scores (Quartile 1 = bottom 25% traceability score, Quartile 4 = top 25% traceability score) and then compared the distribution of ‘Late Requirements Burden’ across these quartile groups. What we found is that projects within the bottom traceability quartile had a median Late Requirements Burden 3x greater than those in the top traceability quartile. In other words, the evidence supports that projects managed with higher traceability generally experience less risk from late requirements.
Recommendation
Our analysis has shown that late requirements negatively impact projects and that managing projects through a Traceability Score is the only empirical way to reduce the risk of late requirements. Below you can see how one can measure the Traceability Score over time as a project progresses to ensure system engineering best practices are being followed. A low or falling Traceability Score can quickly identify areas to address to reduce the risk of late requirements.
Here you can see how managing the Traceability Score directly as the project is underway would have identified the risk early in the Late Project.
To learn more about achieving Live Traceability™ on your projects, please reach out for a consultation.
Interested in learning more? Download the entire Research Notes: Traceability Score™ datasheet HERE.
CEO at Jama Software
As CEO of Jama Software, Marc is passionate about client success with our products and nurturing a growth mindset among the entire team. Marc has a proven track record of leading high-performance teams to deliver outsized results for clients, employees and investors. He understands the key challenges that exist for companies building complex products and undergoing digital transformation. Most recently, he was CEO at Aberdeen and a COO for Lionbridge Technologies driving organic growth from $70 million to $140 million in four years and a successful go private transaction. Marc began his career at McKinsey and is a graduate of MIT Sloan and Brown University.
Latest posts by Marc Osofsky (see all)
- Traceable Agile™ – Speed AND Quality Are Possible for Software Factories in Safety-critical Industries - October 3, 2023
- How to Plan for Large Language Model (LLM) Adoption Within Your Engineering Organization - July 26, 2023
- How to Achieve Higher Levels of the Capability Maturity Model Integration (CMMI): Part 2 - July 25, 2023