Performance-Based Earned Value


​​​​​NDAA for FY 2016, Sec. 809, directed establishment of an advisory panel with a view toward streamlining and improving the efficiency and effectiveness of the defense acquisition process and to make recommendations for the amendment or repeal of regulations. The panel issued its Volume 1 report (809 report) on Jan. 31, 2018 which included a recommendation to eliminate the mandate for EVM for software programs using Agile methods. Section 4 concerned EVM.

The report cited shortcomings of EVM. These included:

Another substantial shortcoming of EVM is that it does not measure product quality. A program could  perform ahead of schedule and under cost according to EVM metrics, but deliver a capability that is  unusable by the customer...Traditional measurement using EVM provides less value to a program than an Agile process in which the end user continuously verifies that the  product meets the requirement. 

The following recommendations were submitted to the Sec. 809 Panel, as of April 9, 2018 :
Challenges
The 809 Report only partially addressed challenges related to the Earned Value Management System (EVMS). Challenges endure that were identified since 1996.
Events
1. 1996 DoD Vision
The intended purpose of an EVMS was announced when DoD accepted industry guidelines for EVMS to replace DoD criteria. DoD encouraged industry to develop a widely accepted industry or international standard. The announcement included DoD’s vision:
·   Adopt…commercial practices in lieu of government-unique practices.
·   Rely on our contractors to maintain management control systems that protect the public interest
·   Support the ‘insight, not oversight' philosophy underlying DoD acquisition reform initiatives.
2. 1998 EVMS Standard
ANSI/EIA-748 was issued as the U.S industry standard.
3. 2009 DoD Report to HASC/SASC in Response to a WSARA directive:
·   The ”utility of EVM has declined to a level where it does not serve its intended purpose.”
·   Contractors “keep EVM metrics favorable and problems hidden.”
·   If good technical performance measurements (TPM) are not used, programs could report 100% of EV even though behind schedule in validating requirements, completing the preliminary design, meeting the weight targets, or delivering software.
·   Integrate systems engineering with EVM.
Challenges

  • ANSI/EIA-748 is not widely accepted. All 32 guidelines are not utilized by companies unless there is a government mandate.
  • Develop project monitoring and control methods for all EMD programs, which may include EVM, that provide faith in the quality of data and support insight into schedule, cost, and estimate at completion.


Evidence
Per the 2016 Grant Thornton Government Contractor Industry Survey of participants with contracts that require EVMS:
      • Only 37% believe that EVM is a cost-effective management tool.
      • Only 25% of companies required to report under EVMS would adopt that system if they were not required to do so by  he contract.


Sen. Collins said in the WSARA conference report: “GAO observed that contractor reporting on EVM often lacks consistency, leading to inaccurate data and faulty application of the EVM metric. In other words, garbage in, garbage out.”
Excerpt from 809 Report:
“Another substantial shortcoming of EVM is that it does not measure product quality. A program could perform ahead of schedule and under cost according to EVM metrics but deliver a capability that is unusable by the customer.
Traditional measurement using EVM provides less value to a program than an Agile process in which the end user continuously verifies that the product meets the requirement.” 

Recommendations
Please consider two alternatives for changes to DFARS and DoDI 5000.02.
1. Replace requirement to comply with EVMS guidelines with requirement to comply with PMI Project Management Body of Knowledge (PMBOK® Guide) or
2. Eliminate requirement to adhere to 12 of 32 EVMS guidelines and add requirement to modify the language of 3 guidelines with regard to contractor compliance and compliance reviews:

  • Require use of technical performance measures (TPM)
  • Add “product scope” (technical baseline) to the authorized baseline


Alternative 1 (preferred)
The rationale is included in letters to:
1. OMB Director Mulvaney letter, 1/17/18 ,​​​​ Subj: Project Management of High Risk DoD Acquisitions

2. HASC Chairman Thornberry, 1/28/18, Subject: Updated NDAA Defense Acquisition Reform - Earned Value
Links to and excerpts from the letters are at http://www.pb-ev.com/acquisition-reform.html .


Additional information is in two articles in Defense AT&L: http://www.pb-ev.com/articles-and-tutorial.html
· "Earned Value Management Acquisition Reform", Nov. 2010
· "A Contract Requirement Rule for Program Managers," Nov. 2015
   An excerpt from the 2015 article follows:
EVM, based on EIA-748, is used primarily by federal contractors when contractually required. A more powerful tool is the ANSI standard that voluntarily is used worldwide because it works, not because it is imposed by federal acquisition regulations. It is the PMBOK® Guide.
The needs of the PM that are covered by the PMBOK® Guide but absent in EIA-748 include:
·  technical or product baseline
·  requirements management and traceability 
·  risk management


PMBOK® Guide also covers flow down of the above needs to contractors.  


The PMBOK® Guide contains many artifacts and tools that have no counterpart in EIA-748, including a differentiation between the “project scope” or work that needs to be accomplished and the “product scope,” a description documenting the characteristics of the product that the project will create.

The PMBOK® Guide also covers EVM topics such as scheduling (including network diagrams), Performance Measurement Baseline, control accounts, work packages, earned value, variance analysis, estimate at completion, and management reserve.

Pertinent excerpts from PMBOK® Guide and a comparison of EIA-748 with PMBOK® Guide for consistency with DoDI 5000.02 needs are at www.pb-ev.com/pmbok-excerpts.htm .

Alternative 2
The rationale for and implementing details are included in:
·  Thornberry letter, 11/17/13, Subj: Expanded NDAA Defense Acquisition Reform - EV
·  Skelton letter, 3/28/10, Subj: Panel on Defense Acquisition Reform 

My recommendations to implement Alternative 2 were developed as a consultant to PARCA and incorporated into CrossTalk, the Journal of Defense Software Engineering;
"Basing Earned Value on Technical Performance," Jan. 2013. http://www.pb-ev.com/articles-and-tutorial.html 

Current Regulation and Policy: DFARS Subpart 234.201, DoDI 5000.02 Table 8