Introduction

When starting in system engineering there is usually the continuous improvement aspect that needs to be addressed. In order to continuously improve the process, you need to be able to measure the effectiveness of the current effort.

This paper will present a methodology linking system engineering to the capability maturity model developed by Carnegie Mellon University. Furthermore, this approach can be used to evaluate the system engineering maturity of an organisation prior to implementing any of the system Engineering processes. 

If you would like to find out more about how HKRP can assist your organisation in understanding and improving your Systems Engineering maturity to support your operations, please contact us via our website - hkrp.com.au.

Systems Engineering

Systems engineering covers the development of systems, which may or may not include software. Systems engineers focus on transforming customer needs, expectations, and constraints into product solutions and supporting these product solutions throughout the life of the product.

This paper will present the Capability Maturity Model Integration Systems Engineering CMMI SE approach to assessing the maturity of an organisation / team in System Engineering in accordance with the CMMI levels.

The CMMI Model

In CMMI there are five maturity levels, each a layer in the foundation for ongoing process improvement, designated by the numbers 1 through 5:

1. Initial
2. Managed
3. Defined
4. Quantitatively Managed
5. Optimising.

Each level builds on top of each other and as such, each level needs to be fully implemented prior to the maturity level being raised. Alternatively, failure to implement one of the lower-level processes reduces the maturity level of the organisation even if other processes of a higher maturity level are implemented.

Maturity Level 1: Initial

At maturity level 1, processes are usually ad hoc and sometimes chaotic. The organisation usually does not provide a stable environment. Success in these organisations depends on the competence of the people in the organisation and not on the use of proven processes. Despite this ad hoc chaotic environment, maturity level 1 organisations often produce products and services that work however, they frequently exceed budget and schedule expectations.

Maturity level 1 organisations are characterised by a tendency to over commit, abandon processes in the time of crisis, and not be able to repeat their past successes.

Maturity Level 2: Managed

At maturity level 2, an organisation has achieved all the specific and generic goals of the maturity level 2 process areas. In other words, the projects of the organisation have ensured requirements are managed and that processes are planned, performed, measured, and controlled.

The process discipline reflected by maturity level 2 helps to ensure existing practices are retained during times of stress. When these practices are in place, projects are performed and managed according to their documented plans.

Maturity Level 3: Defined

At maturity level 3, an organisation has achieved all the specific and generic goals of the process areas assigned to maturity levels 2 and 3. At maturity level 3, processes are well characterised and understood, and are described in standards, procedures, tools, and methods.

The organisation’s set of standard processes, which is the basis for maturity level 3, is established and improved over time. These standard processes are used to establish consistency across the organisation. Projects establish their defined processes by tailoring the organisation’s set of standard processes according to tailoring guidelines.

In comparison to maturity level 2, maturity level 3 is typically described in more detail and more rigorously than at maturity level 2. Furthermore, at maturity level 3, the standard processes descriptions and procedures for projects are tailored from the organisation set of standard processes to suit a particular project organisational unit. As a result, the processes performed across the organisation are consistent except for the differences allowed by tailoring guidelines.

Maturity Level 4: Quantitatively Managed

At maturity level 4, an organisation has achieved all the specific goals of the process areas assigned to maturity levels 2, 3 and 4 and the generic goals assigned to maturity levels 2 and 3. Subprocesses are selected that significantly contribute to overall process performance. These selected subprocesses are controlled using statistical and other quantitative techniques.

Quantitative objectives for quality and process performance are established and used as criteria in managing processes. Quantitative objectives are based on the needs of the customer, end users, organisation, and process implementers. Quality and process performance are understood in statistical terms and are managed throughout the life of the processes. For these processes, detailed measures of process performance are collected and statistically analysed.

Maturity Level 5: Optimising

At maturity level 5, an organisation has achieved all the specific goals of the process areas assigned to maturity levels 2, 3, 4 and 5 and the generic goals assigned to maturity levels 2, 3 and 4. Processes are continually improved based on a quantitative understanding of the common causes of variation inherent in processes.

Maturity level 5 focuses on continually improving process performance through both incremental and innovative technological improvements. Quantitative process-improvement objectives for the organisation are established, continually revised to reflect changing business objectives and used as criteria in managing process improvement. The effects of deployed process improvements are measured and evaluated against the quantitative process-improvement objectives. Both the defined processes and the organisation’s set of standard processes are targets of measurable improvement activities.

Moving through the maturity levels

Organisations can achieve progressive improvements in their organisational maturity by first achieving stability at the project level and continuing to the most advanced-level, organisation-wide continuous process improvement, using both quantitative and qualitative data to make decisions.

The organisational maturity describes the expected result that can be achieved by an organisation for their system engineering endeavour. It is one method of predicting the most likely outcome for the next project the organisation undertakes. For instance, an organisation performing at maturity level 1 will have a different expected outcome than an organisation operating at maturity Level 3. It is more likely for the organisation operating at maturity level 3 to have a more successful outcome then for the one operating at maturity level 1.

The CMMI Levels and Processes

The following levels and their respective processes are as defined in CMMI-SE (v1.1).

Level 1 - Initial

Any organisation, as long as it is functional and is delivering products and/or services, is considered a Level 1 organisation. Despite processes being considered ad-hoc and chaotic at this level, those organisations can function despite not following standardised approaches. Projects tends to be delivered over budget and/or schedule and past success are difficult to repeat.

Level 1 is the default level for any organisation that is using System Engineering.

Level 2 - Managed

The following list of processes represent the maturity level 2 process area:

• Requirements Management
• Project Planning
• Project Monitoring and Control
• Supplier Agreement Management
• Measurement and Analysis
• Process and Product Quality Assurance
• Configuration Management.

Requirements Management 

The purpose of Requirements Management is to manage the requirements of the project's products and product components and to identify inconsistencies between those requirements and the project's plans and work products.

The following processes should be implemented for a maturity level 2 rating, and are required and expected for a maturity level 3 rating in Requirements Management:

• Manage Requirements
• Institutionalise a Managed Process.

Project Planning

The purpose of Project Planning is to establish and maintain plans that define project activities.

The following processes should be implemented for a maturity level 2 rating, and are required and expected for a maturity level 3 rating and above in Project Planning:

• Establish Estimates
• Develop a Project Plan
• Obtain Commitment to the Plan
• Institutionalise a Managed Process.

Project Monitoring and Control

The purpose of Project Monitoring and Control is to provide an understanding of the project’s progress so appropriate corrective actions can be taken when the project’s performance deviates significantly from the plan.

The following processes should be implemented for a maturity level 2 rating, and are required and expected for a maturity level 3 rating and above:

• Monitor Project Against Plan
• Manage Corrective Action to Closure
• Institutionalise a Managed Process.

Supplier Agreement Management 

The purpose of Supplier Agreement Management is to manage the acquisition of products from suppliers for which there exists a formal agreement.

The following processes should be implemented for a maturity level 2 rating, and are required and expected for a maturity level 3 rating and above:

• Establish Supplier Agreements
• Satisfy Supplier Agreements
• Institutionalise a Managed Process.

Measurement and Analysis

The purpose of Measurement and Analysis is to develop and sustain a measurement capability that is used to support management information needs.

The following processes should be implemented for a maturity level 2 rating, and are required and expected for a maturity level 3 rating and above:

• Align Measurement and Analysis Activities
• Provide Measurement Results
• Institutionalise a Managed Process.

Process and Product Quality Assurance

The purpose of Process and Product Quality Assurance is to provide staff and management with objective insight into processes and associated work products.

The following processes should be implemented for a maturity level 2 rating, and are required and expected for a maturity level 3 rating and above:

• Objectively Evaluate Processes and Work Products
• Provide Objective Insight
• Institutionalise a Managed Process.

Configuration Management

The purpose of Configuration Management is to establish and maintain the integrity of work products using configuration identification, configuration control, configuration status accounting, and configuration audits.

The following processes should be implemented for a maturity level 2 rating, and are required and expected for a maturity level 3 rating and above:

• Establish Baselines
• Track and Control Changes
• Establish Integrity
• Institutionalise a Managed Process.

Level 3 - Defined

The following list of processes represent the maturity level 3 process area:

• Requirements Development
• Technical Solution
• Product Integration
• Verification
• Validation
• Organisational Process Focus
• Organisational Process Definition
• Organisational Training
• Integrated Project Management
• Risk Management
• Decision Analysis and Resolution.

Requirements Development

The purpose of Requirements Development is to produce and analyse customer, product, and product component requirements.

The following processes should be implemented for a maturity level 3 rating, and are required and expected for a maturity level 4 rating and above:

• Develop Customer Requirements
• Develop Product Requirements
• Analyse and Validate Requirements
• Institutionalise a Defined Process.

Technical Solution

The purpose of Technical Solution is to design, develop, and implement solutions to requirements. Solutions, designs, and implementations encompass products, product components, and product related lifecycle processes either singly or in combinations as appropriate.

The following processes should be implemented for a maturity level 3 rating, and are required and expected for a maturity level 4 rating and above:

• Select Product Component Solutions
• Develop the Design
• Implement the Product Design
• Institutionalise a Defined Process.

Product Integration

The purpose of Product Integration is to assemble the product from the product components, ensure the product, as integrated, functions properly, and delivers the product.

The following processes should be implemented for a maturity level 3 rating, and are required and expected for a maturity level 4 rating and above:

• Prepare for Product Integration
• Ensure Interface Compatibility
• Assemble Product Components and Deliver the Product
• Institutionalise a Defined Process.

Verification

The purpose of Verification is to ensure selected work products meet their specified requirements.

The following processes should be implemented for a maturity level 3 rating, and are required and expected for a maturity level 4 rating and above:

• Prepare for Verification
• Perform Peer Reviews
• Verify Selected Work Products
• Institutionalise a Defined Process.

Validation

The purpose of Validation is to demonstrate a product or product component fulfills its intended use when placed in its intended environment.

The following processes should be implemented for a maturity level 3 rating, and are required and expected for a maturity level 4 rating and above:

• Prepare for Validation
• Validate Product or Product Components
• Institutionalise a Defined Process.

Organisational Process Focus

The purpose of Organisational Process Focus is to plan and implement organisational process improvement based on a thorough understanding of the current strengths and weaknesses of the organisation’s processes and process assets.

The following processes should be implemented for a maturity level 3 rating, and are required and expected for a maturity level 4 rating and above:

• Determine Process-Improvement Opportunities
• Plan and Implement Process-Improvement Activities
• Institutionalise a Defined Process.

Organisational Process Definition

The purpose of Organisational Process Definition is to establish and maintain a usable set of organisational process assets.

The following processes should be implemented for a maturity level 3 rating, and are required and expected for a maturity level 4 rating and above:

• Establish Organisational Process Assets
• Institutionalise a Defined Process.

Organisational Training

The purpose of Organisational Training is to develop the skills and knowledge of people so they can perform their roles effectively and efficiently.

The following processes should be implemented for a maturity level 3 rating, and are required and expected for a maturity level 4 rating and above:

• Establish an Organisational Training Capability
• Provide Necessary Training
• Institutionalise a Defined Process.

Integrated Project Management

The purpose of Integrated Project Management is to establish and manage the project and the involvement of the relevant stakeholders according to an integrated and defined process that is tailored from the organisation's set of standard processes.

The following processes should be implemented for a maturity level 3 rating, and are required and expected for a maturity level 4 rating and above:

• Use the Project’s Defined Process
• Coordinate and Collaborate with Relevant Stakeholders
• Institutionalise a Defined Process.

Risk Management

The purpose of Risk Management is to identify potential problems before they occur, so that risk-handling activities may be planned and invoked as needed across the life of the product or project to mitigate adverse impacts on achieving objectives.

The following processes should be implemented for a maturity level 3 rating, and are required and expected for a maturity level 4 rating and above:

• Prepare for Risk Management
• Identify and Analyse Risks
• Mitigate Risks
• Institutionalise a Defined Process.

Decision Analysis and Resolution

The purpose of Decision Analysis and Resolution is to analyse possible decisions using a formal evaluation process that evaluates identified alternatives against established criteria.

The following processes should be implemented for a maturity level 3 rating, and are required and expected for a maturity level 4 rating and above:

• Evaluate Alternatives
• Institutionalise a Defined Process.

Level 4 - Quantitatively Managed

The following list of processes represent the maturity level 4 process area:

• Organisational Process Performance
• Quantitative Project Management.

Organisational Process Performance

The purpose of Organisational Process Performance is to establish and maintain a quantitative understanding of the performance of the organisation’s set of standard processes in support of quality and process-performance objectives, and to provide the process performance data, baselines and models to quantitatively manage the organisation’s projects.

The following processes should be implemented for a maturity level 4 rating, and are required and expected for a maturity level 5 rating and above in Project Planning:

• Establish Performance Baselines and Models
• Institutionalise a Defined Process.

Quantitative Project Management

The purpose of the Quantitative Project Management process area is to quantitatively manage the project’s defined process to achieve the project’s established quality and process-performance objectives.

The following processes should be implemented for a maturity level 4 rating, and are required and expected for a maturity level 5 rating and above in Project Planning:

• Quantitatively Manage the Project
• Statistically Manage Subprocess Performance
• Institutionalise a Defined Process.

Level 5 - Optimised

The following list of processes represent the maturity level 5 process area:

• Organisational Innovation and Deployment
• Causal Analysis and Resolution.

Organisational Innovation and Deployment

The purpose of Organisational Innovation and Deployment is to select and deploy incremental and innovative improvements that measurably improve the organisation's processes and technologies. The improvements support the organisation's quality and process performance objectives as derived from the organisation's business objectives.

The following processes should be implemented and continuously improved for a maturity level 5 rating:

• Select Improvements
• Deploy Improvements
• Institutionalise a Defined Process. 

Causal Analysis and Resolution

The purpose of Causal Analysis and Resolution is to identify causes of defects and other problems and take action to prevent them from occurring in the future.

The following processes should be implemented and continuously improved for a maturity level 5 rating:

• Determine Causes of Defects
• Address Causes of Defects
• Institutionalise a Defined Process.

Conclusion

Despite the CMMI model being defined in the above section it is important to note there is still some work that needs to be performed to link ISO 15288 and its lifecycle processes to the existing improvement processes and objectives defined in CMMI-SE.

Furthermore, in order to obtain repeatable and consistent evaluation method, an evaluation sheet with defined metrics needs to be created to evaluate the performance of an organisation against the goals of each process.

If you would like to find out more about how HKRP can assist your organisation in understanding and improving your Systems Engineering maturity to support your operations, please contact us via our website - hkrp.com.au.