Defense ATL Magazine, May-June 2011, published by Defense Acquisition University, includes the article "Path to Earned Value Management Acquisition Reform." Link: Path to EVM Acquisition Reform 

Topics:

• R
  eview of DoD acquisition guidance with regard to technical performance and EVM
• Recommended changes to OMB policy, FAR, and DFARS to close the Quality Gap in ANSI/EIA-748
• Why DoD can’t implement its own policy requirements and legislative requirements without EVM acquisition reform
• Why compliance with some EVMS guidelines is non-value added

Defense ATL Magazine, Nov.-Dec. 2010, includes the article, "Earned Value Management Acquisition Reform." 

Link: AT&L: EVM Acquisition Reform

Key points of the article:

1. ANSI/EIA-748, the government-required EVMS standard, is intrinsically flawed.

a. It ignores the technical baseline (product requirements) and focuses on measuring the quantity of work completed, not quality (Quality Gap). EVMS Quality Gap

b. It lacks guidance on risk management.

c. Consequently, it does not provide a framework to integrate cost, schedule, and technical performance or to integrate risk.

 

2. Contractors may be compliant with ANSI/EIA-748 even if their reported progress is not based on technical performance measures (TPM). The Quality Gap enables contractors to:

   a. Report EV based on the quantity of drawings or code completed, regardless of their failure to meet technical objectives or planned functionality.

   b. Overstate EV when cumulative EV is not adjusted to account for rework.

   c. Establish misleading PMB and schedule when tests and rework are planned in management reserve instead of in the initial PMB.

3. DoD has reported that EVM, based on ANSI/EIA-748, no longer serves its intended purpose. DoD EVM Implementation

4. Guidance to base EV on TPMs is provided in DoD and GAO guides but federal regulations (Federal Acquisition Regulation (FAR) and Defense FAR Supplement (DFARS)) do not require TPMs. 

5. Both Congress and the Executive Office are addressing acquisition reform regarding EVM in general and Information Technology (IT) projects, in particular. The National Defense Authorization Act (NDAA) includes a provision regarding technical performance/quality  and EVMS. 

6. Most commercial companies use the Project Management Institute (PMI) standard, Guide to the Project Management Body of Knowledge (PMBOK® Guide),  not ANSI/EIA-748.

7. If you are measuring the wrong things or not measuring the right way,  then EVM may be more costly to administer and may provide less management value.

8. Industry has not corrected the flaws in ANSI/EIA-748. Contractors prefer the status quo. Reform is up to Government. 

9. Commercial processes and best practices and systems engineering (SE) standards should be considered for EVM acquisition reform.

    a. Replace ANSI/EIA-748 or augment it with components of the PMBOK® Guide and Systems Engineering standards.

    b. Revise Office of Management and Budget (OMB), FAR and DFARS to require TPM integration, using provisions in existing DoD and GAO voluntary guidance.   

 

The Measurable News, published by the PMI College of Performance Management, includes the article, "Integrating TPM with EVM." The authors state Industry's position that acquisition reform is not needed (Measurable News, 2010, Issue 4 ,page 6) . A rebuttal of the article is in the email to the House and Senate Armed Services Committees ( Email to House and Senate Armed Services Committees ).  

Federal legislation:

DoD and the Congress have recognized that EVM is not working. Legislation which may lead to EVM acquisition reform was signed by Pres. Obama on Jan. 7, 2011 (See H.R. 6523, Ike Skelton National Defense Authorization Act for FY 2011 (NDAA), (below). 

DoD policy memoes and reports cite the following deficiencies:  

            DoD Report to Congress (WSARA)

• Other DoD policy memoes and reports:
• Use of EVM in program management, department-wide, is insufficient

• Unfavorable audit findings indicate EVM is not serving its intended function in the internal control process
•  Many instances of inappropriate changes

  o  Arbitrarily changing past variances

  o  Moving budgets to mask overruns

  o Making changes that were not properly authorized

• Navy: "Utility of EVM has declined to a level where it does not serve its intended purpose."

• Broad deficiencies in EVM compliance
• Intentional masking of cost and schedule variances
• Inadequate reporting of Estimates at Complete (EAC)

• Lack of integration across the cost, schedule, and work authorization systems (technical)
• DCMA Lockheed Martin EVMS Compliance Report, 2007
• Using Management Reserve (MR) to alter internal and subcontractor performance levels and   overruns

• Inappropriate EV techniques used for assessment of material, subcontracts, and rework

• Cost and schedule integration problems undermine validity of data

Congress:

The broad deficiencies in EVM compliance have triggered a Congressional oversight reaction. Sen. Collins, a sponsor of WSARA, stated in the conference report that:

• GAO observed that contractor reporting on EVM often lacks consistency, leading to inaccurate data and faulty application of the EVM metric.
• “In other words, garbage in, garbage out.”
• “With improved EVM data quality, both the government and the contractor will be able to improve program oversight, leading to better acquisition outcomes.”

WSARA directed DoD to provide recommendations to improve EVM and it implementation. A summary of the DoD EVM Report to Congress and  a link to the Report are at DoD EVM Implementation .

1/7/2011: H.R. 6523 , the ‘‘Ike Skelton National Defense Authorization Act for Fiscal Year 2011 (NDAA),’’  was signed by Pres. Obama. It contains key provisions that may lead to EVM acquisition reform and eventually hold contractors accountable to link technical performance with earned value.

  Section 864(a) requires DoD to review acquisition guidance, including DoD Instruction (DoDI) 5000.02 Operation of the Defense Acquisition System, and to consider:

  Sec. 864(b)(4):

• the extent to which earned value management should be required on acquisitions not involving the acquisition of weapon systems and
• whether measures of quality and technical performance should be included in any earned value management system

  Sec. 864(c) requires that the Secretary of Defense submit a report to the House and Senate Armed Services Committees detailing any changes in the DoD acquisition guidance identified during the review and any actions taken, or planned to be taken, to implement such changes.

Deficiency in DoDI 5000.02 (addresses Congressional request in NDAA):

A review of DoD acquisition guidance reveals that DoD Guides cover integration of technical performance measures (TPM) with EVM but DoD acquisition policy, DoDI 5000.02, does not discuss integration of quality and TPM with EVM. See: DOD Guide Table . In DoDI 5000.02, the term "technical performance" is only found once in the Enclosure on Test and Evaluation. The term "quality" is found once in the section on acquisition of services and once in the section on Habitability ("quality of life"). Conclusion: DoDI 5000.02 and DFARS are silent on the measurement of technical performance as a basis of earned value. Recommended revisions to DoDI 5000.02 are at Proposed revs to DoDI5000.02 .

There is a “Quality Gap” in the EVMS Standard which permits contractors to ignore technical performance. Read more about it at

            EVMS Quality Gap and in articles at Advanced EV: PBEV :   

• Improving the Quality of EVM Information
• Path to EVM Acquisition Reform
• Earned Value Management Acquisition Reform
• Applying EVM to Software Intensive Programs

• Integrating Systems Engineering with EVM 

Need to change Federal Acquisition Policy and Regulations:

Any changes to DoD acquisition instructions and guidance that may result from H.R. 6523 will not fix the problems because these documents are not applicable to contractors. Revisions to OMB policy, FAR, and DFARS are necessary to require contractors to measure and report earned value that is based on technical performance. Proposed revisions are detailed at the following tab: Link:OMB policy, FAR, DFARS ........

Characteristics of Inaccurate EV Information

The Quality Gap in ANSI/EIA-748, the EVMS Standard, permits contractors to report inaccurate EV information to the Government: 

• Contractors may overstate EV (and CPI) when they base EV only on the “quantity of work" completed and fail to report true schedule variances with regard to achieving planned technical performance (Quality).
• Contractors may budget an unrealistic Performance Measurement Baseline (PMB) by assuming unachievable efficiency targets (such as hours per drawing or lines of code (LOC), reduced number and quality of tests, no rework and the quantity of drawings, LOC, and tests that are necessary to achieve technical objectives). The assumptions are not justified based on past performance and process improvements. 
• Later, after the Preliminary Design Review (PDR) or the Critical Design Review (CDR) when the design fails to meet requirements or other success criteria of the Integrated Master Plan (IMP), MR is used to budget "new" rework that should have been completed to meet those requirements. MR is sometimes similarly used when test objectives are not achieved.
• DoD reported to Congress that contractors misused MR to mask true performance and cost overruns .
• If MR  is used to budget future rework (that should have been planned and completed earlier), then EV is earned in new work packages. If the previous work packages had been linked to IMP success criteria, they would still be open and behind schedule.

The following table includes conditions that may result from using EV that is not linked to technical performance.

Path to EVM Acquisition Reform: A recommended Path to EVM Acquisition Reform is in the letter to Mr. Gary Bliss, Director, PARCA, Office of the Asst. Secretary of Defense. Link: Letter to PARCA Director Gary Bliss . 

It is also recommend that the GAO and DCMA review the accuracy of contractors' EVM for consistency the technical performance. Technical performance is provided in the DoD Director of Test and Evaluation Annual Report for FY 2010. See the bottom of the Technical Performance:  DOT&E .

Interim Remedy

Until there is EVM acquisition reform, DoD (or any Customer who depends on the validity of EV reports) should consider the following to ensure accurate EVM reports:

• Exercise due diligence to define a technical baseline that can be implemented within realistic cost and schedule objectives.
• Verify that the Contractor has integrated EVM, program management, and systems engineering procedures and is compliant with them.
• Establish success criteria for major technical reviews (System Functional Review, PDR, CDR, System Verification Review) in the IMP.
• Technical reviews and criteria are defined in Systems and Software Engineering Defense Acquisition Program Support Methodology (DAPS). See Dept. of Defense tab.

• Use the Integrated Baseline Review (IBR) to verify that criteria and milestones exist for completion of the technical baselines (Functional, Allocated, Product Baseline) and that EV is linked to :
• TPMs
• Success criteria for major technical reviews
• Completion of Systems Engineering work products (trade studies, validated requirements baseline, requirements traceability matrix) 
• Use the IBR to verify that assumptions and estimates for efficiency and rework are realistic and that the PMB is achievable
• Constantly review variances, the program's "most likely" EAC, and use of MR
• Verify that schedule variances and impacts are consistent between the IMS, EV, and TPM reports