Industry Insights on Digital Engineering and Defense Acquisition

Photo by Staff Sgt. Mikayla Gibbs

Digital engineering (DE), as defined by the U.S. Department of Defense, is an “integrated digital approach that uses authoritative sources of system data and models as a continuum across disciplines to support life cycle activities from concept through disposal.”^[1] DE can be seen as automated support to the systems engineering process.

In 2018, the Air Force announced a new strategy for using DE in its acquisition life cycle process. The Digital Engineering Strategy (which was updated in December 2023) emphasizes the importance of DE in improving the efficiency, effectiveness, cost, and quality of Department of the Air Force (DAF) systems. It also highlights the need for a common digital thread that connects all aspects of the acquisition process and life cycle management of weapon systems. DAF programs are now being encouraged to develop DE implementation plans at their inception.

To date, however, most DE efforts are in early stages of implementation, so there has been no empirical analysis of the extent to which DE is yielding these benefits for industry and the DAF.

To close this knowledge gap, recent work for the Air Force Cost Analysis Agency builds on previous RAND research by investigating how DE is being implemented in defense acquisition and whether it is achieving these claimed benefits.

As part of this work, the RAND research team conducted a survey that was distributed to more than 300 people across the aerospace industry who were associated with DE. Approximately 70 responses were received, the majority of which were from large firms with more than 1,000 employees. Questions covered a variety of topics on DE, such as goals of DE implementation, activities undertaken to accomplish DE, costs and benefits of DE implementation, challenges that firms are facing with implementing DE, and metrics to measure DE success. This brief describes industry insights and experience with DE drawn from those survey results.

Digital Engineering Implementation Goals and Activities

About one-half of industry respondents reported that reasons for implementing DE were similar across defense-related programs or contracts. The most-reported reasons for engaging in the implementation of DE were to attain the intended system quality or performance, to contain cost, and to adhere to schedules. Firms are also expecting to receive internal benefits from implementing DE.

Survey results indicate that the goal of enabling specific program objectives should be kept in mind when selecting DE implementation activities and that these activities should be designed to generate the largest return on investment. An example would be creating digital twins only for specific components that will inform a limited set of decisions rather than creating many digital twins for a broader array of systems that could end up being underused.

Most firms are actively investing in DE activities; according to more than one-half of survey respondents, these investments are leading to changes to their business models. Changes are being made in such areas as increasing efficiency and system performance, modernizing technology, changing policies and procedures, and investing in new technology. But it is important to recognize that DE activities can look different depending on the type of system or platform, the maturity of the system design, and the age of the system or acquisition in its life cycle. Consequently, DE implementation must be tailored to the specific needs and characteristics of each system or platform.

Coordination and collaboration across all stakeholders are key to realizing the benefits and mitigating the costs of DE. Failure to collaborate could not only limit DE benefits but also create problems with defense acquisition. Subtier suppliers that work on multiple defense projects, for example, could have difficulty managing different sets of project demands with potential for inefficiencies and rework.

Survey respondents suggested leveraging a common ontology of consistent terminology and basic concepts that would enable coordination across the supply chain. A clearly defined supply chain with a communicated flow of requirements from prime to subcontractor is one way to encourage coordinated implementation of DE.

Costs and Benefits

DE implementation in defense acquisition does entail costs. The biggest drivers of DE implementation costs are information technology (IT) and related costs, notably (1) IT infrastructure and data storage and (2) IT and information security (Figure 1). Software, models, and tools are also identified as major costs to DE implementation. The categories of data management and governance and of hiring, training, and retaining a qualified workforce also ranked as cost drivers by respondents but were identified as slightly less prohibitive than other categories. Firms noted that without effective management practices, it is difficult to contain DE implementation costs.

Category	Prohibitive	Moderate	No Response
IT Infrastructure and Data Storage	14	12	7
IT and Information Security	18	8	7
Software, Models, and Tools	13	15	5
Data Management and Governance	8	16	7
Hiring, Training, and Retaining Qualified Workforce	12	10	10

SOURCE: RAND analysis of industry survey results. Responses provided to a prescribed list of items.

Although the implementation of DE entails costs, it is also expected to lead to a variety of benefits, according to survey respondents (Figure 2). Survey responses support industry’s anticipation that DE will contribute to containing costs and meeting schedule goals prior to development. Most respondents said that the largest benefits of DE are expected during the sustainment phase.

Improvements to cost, schedule, and performance are expected to occur as a result of more-authoritative data management and decisionmaking that is informed by modeling and data analytics. Improvements in systems engineering and increases in automation are also potential benefits anticipated by the majority of survey respondents; another expected improvement is reduced uncertainty in operations and sustainment. Other benefits include increases in innovation during development and production, establishing infrastructure for collaboration, and a reduction in the need for physical tests. Data on the cost and schedule impacts of DE are limited, but performance-related benefits are already coming to fruition.

• Authoritative data management and decisionmaking informed by models and data (85%)
• Improving systems and engineering and increasing automation (80%)
• Reducing uncertainty in operations and sustainment (65%)
• Technological innovation in development and production
• Establishing infrastructure for collaboration
• Reducing physical tests

SOURCE: Features information from industry survey data.

NOTE: Numbers in parentheses indicate the percentage of firms reporting that benefit. The researchers provide percentages only for the top three responses, which yielded results of 65 percent or higher.

Survey results also indicate that the DAF needs to implement DE across acquisition programs if it wants to leverage lessons learned, data, insights, and other benefits. Even potential enterprise investments—such as IT infrastructure and architecture; personnel recruitment; and software, models, and tools—need shared requirements across programs to maximize benefits from initial investments.

Implementation Challenges

The top three challenges resulting from DE implementation, as reported by more than 60 percent of firms in the industry survey, are data availability, quality, and management (80 percent); software, model, and tool availability, quality, and interoperability (70 percent); and meeting information security needs (60–65 percent). These issues are being reflected in the need for better tools and IT infrastructure, the difficulty in integrating legacy systems, and the need for better collaboration and communication among stakeholders.

Workforce issues pose another major challenge to DE implementation. Firms responding to the industry survey identified three separate workforce challenges: transforming the workforce culture (around 65 percent); hiring, training, and retaining a qualified workforce (around 55 percent); and obtaining security clearances for their workforce (around 30 percent). Firms also indicate that only a portion of the workforce has a desire to learn and adapt to DE workflows.

Metrics for Measuring Success

One quality that is important to leaning into the DE space is the ability to measure the impact of using DE approaches to ensure that they are working as intended. Yet survey respondents highlighted difficulties in measuring DE implementation. In part, measurement difficulties stem from the fact that programs implementing DE have not had the runtime necessary to identify costs and benefits. This is especially true when impacts span the lives of these programs, which may be several decades, and programs are in the early part of the life cycles. Large programs and programs with many parties involved, particularly in design and development, also can make it difficult to capture metrics.

When asked to discuss what kind of metrics their firms already use to assess the benefits of DE, more than one-half of the respondents preferred not to say or did not know. This lack of awareness can make it difficult to develop standardized metrics for measuring DE success across the industry. Respondents who did provide details most often cited scheduling metrics: on-time product deliveries, hours per drawing sheet, or product development cycle time reduction estimates. Other cited metrics were the number of model-based systems, engineering programs, and financial or cost metrics.

Although most respondents did not know or preferred not to say what metrics their firms collected, they also seemed to have an appetite for metrics. When asked whether specific metrics would help firms assess the benefits of DE for defense contracts, the majority of respondents agreed. The top-ranked metrics were physical test reduction (86 percent of respondents), performance improvements of technical requirements (such as mass, function, and noise; 85 percent), and cost reduction of changes or rework (83 percent).

To have value, metrics need to measure benefit, be consistent, and be consistently applied. Most benefits to DE are anecdotal, with firms expecting to see gains in performance, schedule, and cost. Although these benefits are promising, there is a need for approaches to measuring DE success that are more systematic and standardized. By developing and using clear and consistent metrics, stakeholders can better understand the impact of DE on defense acquisition and can identify areas for improvement.

DE is worth exploring and implementing in a structured way because it has already realized a variety of benefits to defense acquisition beyond the potential for cost savings and schedule reduction. DE has enabled increased complexity in system design and expanded trade space of options during system design downselect. By supporting more-accurate and more-comprehensive modeling and simulation, DE can improve the quality and performance of defense systems. It can also improve collaboration and communication among stakeholders, leading to more effective decisionmaking and reduced risk.

Overall, although benefits might not be immediately forthcoming in terms of cost savings or schedule reduction, DE nonetheless has the potential to provide significant long-term benefits to defense acquisition.

Photo by Senior Airman Maria Umanzor Guzman

Actions for the DAF

The insights from the survey, along with inputs from other experts and open-source literature, suggest actions that the DAF should consider when implementing DE for defense acquisition programs:

• Develop goal-oriented plans of action. To ensure successful implementation of DE for defense programs, it is essential to develop goal-oriented plans of action that are tailored to the unique characteristics and objectives of each program.
• Manage the lack of standardization and interoperability. Defense programs must confront the lack of standardization and interoperability among government stakeholders and industry partners. Tools, software packages, and models that vary across these entities need to be identified and effectively managed.
• Adopt strategies for hiring and retaining skilled personnel. The shortage of personnel with the necessary expertise in DE and related fields can hinder the ability of DAF programs to thrive in DE environments.
• Promote cultural change. Without purposeful change management and a drastic shift in culture, successful implementation of DE within the DAF—with a focus on collaboration, communication, and innovation—is not likely to materialize.
• Collect cost and schedule data at a low level. Impacts from DE may not be apparent at the platform level. Therefore, programs should aim to track data at the engineering activity or system component levels to assess the impact of DE implementation.
• Track qualitative benefits. Although the DAF is still collecting data on the cost and schedule impacts of DE, performance-related benefits are already being realized and should be documented.
• Investigate sustainment impacts. DE benefits are expected to have the most significant impact during a weapon system’s sustainment phase. The DAF should investigate the transition of digital artifacts from development and production into sustainment. Additionally, cost and schedule data should be tracked to assess impact.

Note

• [1] Office of the Deputy Assistant Secretary of Defense for Systems Engineering, Digital Engineering Strategy, U.S. Department of Defense, June 2018, p. 3.

Available for Download

Download PDF

Topics

• Cyber and Data Sciences
• Military Acquisition and Procurement
• Military Budgets and Defense Spending