Waxman letter, June 22, 2007, Subject: Award Fees and Contract Oversight
Award fee criteria must be revised to ensure that fee shall be paid based on the contractor meeting cost, schedule, and performance goals.
Waxman response, dated July 10, 2007
Waxman letter, March 10, 2008, Subject: Award Fees, Contract Oversight and Lockheed Martin
- Both DoD and the U.S, Navy issued policy memoranda which disclose that EVM is not working for acquisition management. The DoD memorandum states that:
1. EVM is not serving its intended function in the internal control process
2. DoD is committed to resolve systemic, DoD-wide weaknesses
- The Navy memo is explicit regarding contractor issues. It states that reviews have found broad deficiencies across multiple contractor sites including intentionally masking cost and schedule variances and failure to document and manage changes to the baseline
Zients OMB letter, Dec.13, 2009, Subj: Section 302 of the Weapon System Acquisition Reform Act (WSARA) (Earned Value Management)
Please consider acquisition reform to improve the effectiveness of EVM for the acquisition of capital assets.
DoD Report to Congress excerpts:
- Contractors may circumvent proper EVM practices to keep EVM metrics favorable and problems hidden
- Engineering community should establish technical performance measures that enable objective confirmation that tasks are complete
- Various subsystems that make up many contractors' EVMS are not integrated, resulting in inconsistent portrayals of status
Skelton letter, 3/28/10, Subject: Panel on Defense Acquisition Reform
- Neither the DFARS EVMS clause 252.234-7002 nor its cited EVMS guidelines in EIA-748 require that contractors report progress toward achieving quality or technical goals that are specific and measurable.
- In fact, ANSI/EIA-748 states that earned value is a “measurement of only the quantity of work” and that “quality and technical content of work performed are controlled by other means” (Section 3.8). Guideline 2.2.b describes the use of technical performance goals to measure progress as an option, not a requirement.
- DoD’s Report to Congress, DoD Earned Value Management: Performance, Oversight, and Governance, stated that contractors “keep EVM metrics favorable and problems hidden.” The deficiency in DFARS and EIA-748 enables contractors to report metrics that are more favorable than actual conditions and to defer reporting of real problems.
- Today, neither the acquisition managers nor the PARCA office can be assured that a contractor’s performance metrics are valid or accurate. To my knowledge, neither the GAO nor any other agency ever validated that EVMS truly integrates cost, schedule and quality/technical performance or that it provides accurate status and Estimate at Completion. There is a need to transform EVMS into a more valuable acquisition management tool that will provide early warning of performance problems on a consistent basis.
Excerpts:
Currently, the FAR and DFARS fail to require contractors to measure and report technical performance in their monthly reports. The EVM reporting requirements are costly to administer. However, government program managers often receive flawed reports that lack credibility, transparency,
and accountability.
1. EIA-748, the government-required EVMS standard, is intrinsically flawed.
a. It ignores the technical baseline (product requirements) and measures only quantity of work completed, not quality (EVM Quality Gap)
b. It lacks guidance on risk management.
c. Consequently, it does not provide a framework to integrate cost, schedule, and technical performance or to integrate risk.
2. Contractors may be compliant with ANSI/EIA-748 even if their reported progress is not based on technical performance measures (TPM). The Quality Gap enables contractors to:
a. Report EV based on the quantity of drawings or code completed, regardless of their failure to meet technical objectives or planned functionality.
b. Overstate EV when cumulative EV is not adjusted to account for rework.
c. Establish misleading PMB and schedule when tests and rework are planned in
management reserve instead of in the initial PMB.
6. Most commercial companies use the Project Management Institute standard, Guide to the Project Management Body of Knowledge (PMBOK® Guide), not EIA-748.
7. If you are measuring the wrong things or not measuring the right way, then EVM may be more costly to administer and provide less management value.
8. Industry has not corrected the flaws in EIA-748. Contractors prefer the status quo. Reform is up to Government.
9. Commercial processes and best practices and systems engineering (SE) standards should be considered for EVM acquisition reform.
a. Replace EIA-748 or augment it with components of the PMBOK® Guide and Systems Engineering standards.
b. Revise FAR and DFARS to require that earned value be integrated with technical performance and risk, using provisions in existing DoD and GAO guidance.
McKeon letter, 9/13/11, Subject: Defense Acquisition Reform
- Contractors must submit monthly Contract Performance Reports (CPR) that show cost and schedule performance based on EVMS guidelines in the industry standard, EIA-748. This standard, maintained by the NDIA, has loopholes and deficiencies that enable contractors to submit flawed, inaccurate data to the military. Even if a DCMA compliance review determines that a contractor is compliant with the guidelines, the loopholes enable a contractor to overstate progress and understate final costs. Consequently, a DCMA compliance review can provide false assurance to the Program Manager. This is like relying on Standard & Poor’s “no risk” ratings of mortgage-backed securities before the financial collapse.
- A key deficiency is the lack of a requirement for contractors to link earned value to technical performance or quality. The FAR, DFARS, and EIA-748 enable contractors to report earned value that is based only on the quantity of work performed and to ignore technical performance or quality (Quality Gap). For example, earned value is often based on the quantity of drawings completed or software code written. Although technical performance measures (TPM) are used, such as weight, payload, or software functionality, contractors often ignore these TPMs when calculating earned value. As a result, the CPRs fail to provide early warnings of cost overruns and behind schedule conditions. More accountability and transparency is needed.
Sen. Collins said the following about EVM in the WSARA conference report: “GAO observed that contractor reporting on EVM often lacks consistency, leading to inaccurate data and faulty application of the EVM metric. In other words, garbage in, garbage out.”
As required by WSARA and the 2009 NDAA, DoD submitted a report to Congress in September 2009. It concluded the "Utility of EVM has declined to a level where it does not serve its intended purpose." It stated that:
- Contractors may circumvent proper EVM practices to keep EVM metrics favorable and problems hidden.
- Engineering community should establish technical performance measures (TPM) that enable objective confirmation that tasks are complete.
- Systems Engineering and EVM should be integrated, not stove-piped.
The Skelton provisions in NDAA for FY 2011Sections 864 a, b, and c) require DoD to:
- Review acquisition guidance, including DoD Instruction 5000.02, to “consider whether measures of quality and technical performance should be included in any EVM system.
- Submit a report to the Armed Services Committees.
McCain letter, 10/25/11, Cost Controls on the F-35 and Need for Acquisition Reform of EVM
The acquisition regulations and process enable contractors to submit invalid, misleading information to the Government on all capital asset acquisitions, not just weapon systems. We need to improve transparency and accountability when contractors use a contractually-required Earned Value Management System (EVMS) on cost-reimbursement contracts.
The EVMS guidelines are similar to Generally Accepted Accounting Principles (GAAP). However, GAAP protects investors but EVMS often fails to protect taxpayers. The standard has ambiguities and loopholes that should be removed.
The two most important deficiencies are:
1. Contractors are able to overstate progress and understate final costs (called Estimate at Completion or EAC) by basing earned value on the quantity of work performed, not on technical performance or quality.
2. Contractors routinely use “Management Reserve” (MR) as a slush fund for additional budget to compensate for poor planning, to reduce reported cost overruns, and to provide additional budget for tests and rework to designs that fail to meet technical performance or quality objectives.
McCain letter, 12/20/11, Proposed Amendment to NDAA for 2012 for
EVM Acquisition Reform
The current DFARS enables contractors to submit flawed, monthly Contract Performance Reports that overstate cost and schedule performance and understate the final costs (called Estimate at Completion or EAC).
The proposed amendment would require the Secretary of Defense to modify DFARS to force contractors to base earned value on technical performance or quality, instead of just the quantity of work performed. The new amendment would be a follow-up to NDAA for 2011, Section 864. That provision addressed including measures of quality and technical performance in any EVM system.
Specifically, please propose an amendment that would add requirements to DFARS 234.201 and 252.2347002.
The specific DFARS revisions should state that, in addition to compliance with the guidelines in EIA-748, the contractor shall be in compliance with the following augmentations to those guidelines:
Guideline 2.1(a): Add "product scope" to work scope.
Guideline 2.2(b): Require mandatory, not optional, identification of "technical performance goals" and measurement of technical.
Thornberry letter, 11/17/13, Expanded NDAA Defense Acquisition Reform - Earned Value
Compliance with the EVMS guidelines that are required by DFARS is costly to the programs and the taxpayers. However, compliance does not ensure that Contract Performance Reports provide timely, reliable, or verifiable information, as required by DFARS 252.234-7002. Nor does compliance ensure that performance-based acquisition management meet the OMB objective to measure progress towards milestones, cost, capability to meet specified requirements, timeliness, and quality (OMB Circular No. A-11, Sec. 300-5).
Recommendations:
- Tailor 3 (now 4) EVMS guidelines to meet OMB objectives to measure progress towards capability to meet specified requirements and quality.
- Remove compliance with 12 guidelines to reduce costs for contractor labor, DCMA oversight, training, consultants, and software.
In my opinion, the cost savings by eliminating compliance with eight guidelines will offset any cost increases that may be incurred because of the tailored guidelines.
McCaskill letter, 4/2/14, Request for Earned Value Management Acquisition Reform
McCain letter, 1/21/15, Cost Controls on the F-35 and Need for Acquisition
Reform of EVM
Letter of Appreciation from Sen. John McCain, March 5, 2015, for continuing efforts to improve acquisition reform.
McCain letter, 11/5/16, Cost Overruns and Delays on the F-35 Program and Need for Acquisition Reform
- DoD failed to implement improvements that were discussed in its 2010 DoD EVM Report to Congress that was required by NDAA and WSARA.
- Failures of DoD to implement the objective of its 2004 Policy for Systems Engineering (SE Policy).
- DCMA assessment that contractors are not integrating Technical Performance Measurement (TPM) with EVM.
Trump letter, 11/13/16, DoD Acquisition Reform - Under Budget and Ahead of Schedule
Unfortunately for taxpayers, contractors often exploit permissive EVMS guidelines and submit monthly contract performance reports (CPR) that overstate cost and schedule performance and understate the estimated completion costs and schedule. The CPRs often fail to provide an early warning of pending funding shortfalls and delays to providing working weapons to the warfighters.
The biggest loophole in the EVMS guidelines allows contractors to report progress in terms of percent complete that is based on flawed, misleading metrics. In construction terms, contractors may report cost and schedule performance based on the number of actual vs. planned architectural drawings, or floors built, even if the rising structure will not meet building codes, seismic risks etc. Contractors are not required by the EVMS guidelines to assess and report progress that is based on achieved vs. planned technical performance or quality (Quality Gap).
McCain letter, 1/4/18, Fraud on the F-35 Program and Need for Acquisition Reform
Mulvaney letter, 1/17/18 , Project Management of High Risk DoD Acquisitions
The Program Management Improvement and Accountability Act of 2015 (PMIAA) calls for adoption of government-wide standards, policies, and guidelines for program and project management for executive agencies. Mr. Gene Dodaro, in discussing the GAO’s 2017 high risk list, stated that the statute, if implemented effectively, will help foster progress on high-risk issues government-wide.
However, PMIAA is not applicable to DoD “to the extent that the provisions…are substantially similar to or duplicative of policy, guidance, or instruction of the Department related to program management.” In my opinion, DoD does not have similar or duplicative policy, guidance, or instruction. Consequently, DoD’s high risk acquisitions are off the hook. They will continue to incur cost overruns and schedule slips. Please consider proposing remedial legislation to require that PMIAA be applicable to DoD.
The primary Program Management (PM) standard used by DoD is the Earned Value Management System Standard (EVMS), EIA-748. The most widely accepted standard for project management, planning, and delivery is the “PMI Project Management Body of Knowledge” (PMBOK Guide). In comparing the two standards, it is obvious that EVMS has fatal deficiencies. The characteristics of EVMS that fail to meet the needs of a PM are:
1. EVMS addresses only the work scope but is silent on product.
2. The use of technical performance measures (TPM) is optional, not mandatory.
3. Earned value is a direct measurement of the quantity of work accomplished. The quality and technical content of work performed is controlled by other processes. I call this the “Quality Gap.”
In April 2016, DCMA made a presentation to the National Defense Industrial Association (NDIA) PM Systems Committee. It reported a common, EVM finding of a lack of objective measures to assess performance, including “Measurement does not indicate technical accomplishment.”
A PM needs a project management tool that integrates cost, schedule, and technical performance. However, the primary DoD PM instruction, “DoD Instruction 5000.02, Operation of the Defense Acquisition System,” does not fix the EVMS standard’s failures and close the Quality Gap. Its discussion of TPMs and metrics, in terms of progress against established plans, fails to discuss any linkage with EVM.
Please read, my article in the Nov. 2015 issue of Defense AT&L Magazine, “A Contract Requirement Rule for PMs.”I proposed that the EVMS standard be replaced by PMBOK Guide. This acquisition reform would enable any PM to identify and pinpoint emerging problems on a timely basis and act as early as possible to resolve problems.
Thornberry letter, 1/28/18 , Subject: Updated NDAA Defense Acquisition Reform - Earned Value
The Program Management Improvement and Accountability Act of 2015 (PMIAA) calls for adoption of government-wide standards, policies, and guidelines for program and project management for executive agencies. Mr. Gene Dodaro, in discussing the GAO’s 2017 high risk list, stated that the statute, if implemented effectively, will help foster progress on high-risk issues government-wide.
However, PMIAA is not applicable to DoD “to the extent that the provisions…are substantially similar to or duplicative of policy, guidance, or instruction of the Department related to program management.” In my opinion, DoD does not have similar or duplicative policy, guidance, or instruction. Consequently, DoD’s high risk acquisitions are off the hook. They will continue to incur cost overruns and schedule slips. Please consider proposing remedial legislation to require that PMIAA be applicable to DoD.
The primary Program Management (PM) standard used by DoD is the Earned Value Management System Standard (EVMS), EIA 748. The most widely accepted standard for project management, planning, and delivery is the “PMI Project Management Body of Knowledge” (PMBOK Guide). In comparing the two standards, it is obvious that EVMS has fatal deficiencies. The characteristics of EVMS that fail to meet the needs of a PM are:
1. EVMS addresses only the work scope but is silent on product.
2. The use of technical performance measures (TPM) is optional, not mandatory.
3. Earned value is a direct measurement of the quantity of work accomplished. The quality and technical content of work performed is controlled by other processes. I call this the “Quality Gap.”
In April 2016, DCMA made a presentation to the National Defense Industrial Association (NDIA) PM Systems Committee. It reported a common, EVM finding of a lack of objective measures to assess performance, including “Measurement does not indicate technical accomplishment.”
A PM needs a project management tool that integrates cost, schedule, and technical performance. However, the primary DoD PM instruction, “DoD Instruction 5000.02, Operation of the Defense Acquisition System,” does not fix the EVMS standard’s failures and close the Quality Gap. Its discussion of TPMs and metrics, in terms of progress against established plans, fails to discuss any linkage with EVM.
Revise DFARS to Replace EIA-748 with a Project Management Standard (PMIAA)
An alternative to implementing the recommendations in my 2013 letter is to abandon the requirement for compliance with EIA-748. I now recommend that DoD implement the Program Management Improvement and Accountability Act of 2015 (PMIAA) instead of having a waiver. The Act calls for adoption of government-wide (excluding DoD) standards, policies, and guidelines for program and project management for executive agencies. The Project Management Institute (PMI) Project Management Body of Knowledge (PMBOK Guide) is the only standard that meets PMIAA criteria.
Email to Undersecretary of Defense Ellen Lord, 2/3/18:
Subject:Section 809 Panel Report on Streamlining Acquisition Confirms shortcomings of EVM
Ms. Lord,
The Section 809 Panel Report on Streamlining and Codifying Acquisition Regulations, issued January 31, 2018, confirms what I have been asserting: the limited value of Earned Value Management. EVM does not measure product quality.
Excerpt from Sec. 809 Report:
“Limited Value of EVM
Another substantial shortcoming of EVM is that it does not measure product quality. A program could perform ahead of schedule and under cost according to EVM metrics, but deliver a capability that is unusable by the customer...Traditional measurement using EVM provides less value to a program than an Agile process in which the end user
continuously verifies that the product meets the requirement.”
The Report’s Recommendation 19 is to “Eliminate the EVM mandate for software programs using Agile methods.”
Previous Findings and Recommendations
In my email to you, dated Jan. 14, I cited an April 14, 2014 letter to Sen. McCaskill which stated that “Contractors are permitted to report progress in terms of the percent of work completed and may ignore reporting progress towards meeting the technical performance requirements or quality.”
The quality gap was reported in the May, 2004 article in Defense AT&L, “Integrating Systems Engineering with EVM.”
Excerpt from 2004 article:
“EVM Limitations
With regard to a PM’s needs, there are several limitations of EVMS that can be overcome by integrating EVM with robust systems engineering.
EVMS states that EV is a measure of the quantity, not quality, of the work accomplished. A PM should also ensure that EV also measures the quality and technical maturity of technical work products instead of just the quantity of work. Robust systems engineering processes should provide TPM and exit criteria for assessing technical maturity that are quantitatively linked to EV.”
The May 2011 article in Defense AT&L, "Path to EVM Acquisition Reform," reported that DoDI 5000.02 provides no guidance to link measures of quality and technical performance with EVM.” The article includes recommended regulatory changes that would require that EV be linked to quality. The article also cites the NDAA for FY 2011 which addresses this issue in Ike Skelton’s markup provision.
My 2012 consulting report and slides for PARCA/NAVAIR included this issue:
Again, as stated portrayed in yesterday’s email, from the 2004 article:
Unfortunately for taxpayers, contractors often exploit permissive EVMS guidelines and submit monthly contract performance reports (CPR) that overstate cost and schedule performance and understate the estimated completion costs and schedule. The CPRs often fail to provide an early warning of pending funding shortfalls and delays to providing working weapons to the warfighters.
Then Why Use EVM on non-Software Programs?
The Section 809 Panel recommends revising DFARS Subpart 234.201, DoDI 5000.02 Table 8, and OMB Circular A‐11 to eliminate the EVM mandate for specified software programs. So why not do the same for all EMD programs for which compliance with the EVMS Standard is required?
My PARCA white paper and many articles and tutorials on my website provide specific acquisition reforms and recommendations for implementing the needed fixes.
Incidentally, several articles on the website specifically address software development, including Agile projects and the use of EVM for commercial IT development at Samsung IT.
Letter to Sen. Ernst, 6/4/18, Subj: DoD Acquisition Reform; From Earned Value Management System (EVMS) to a Project Management Standard
Letter to Ms. Weichert (OMB) and Mr. Fahey (DoD), 7/26/18
Subj: Successful Implementation of PMIAA by all Agencies, including DoD
Letter to Ms. Weichert (OMB), 9/8/19
Subj: Recommended GAO Report Scope: High-Risk Series, Progress in Implementing PMIAA PDF letter
GAO Report, GAO-19-157SP, “High-Risk Series, Substantial Efforts Needed to Achieve Greater Progress on High-Risk Areas,” includes a discussion of the PMIAA. The report stated that DOD can benefit from improving Program/Project Management (P/PM).
The report states “In December 2019, we will report on OMB’s progress in implementing PMIAA.” I recommend that the scope of the GAO report include an assessment of DoD’s policy, guidance, or instruction related to P/PM to determine:
1. If the PMIAA provisions are substantially similar to or duplicative of DoD’s policy, guidance, or instruction.
2. If they are consistent with widely accepted standards for P/PM planning and delivery.
3. If they cover the P/PM technical competencies cited above.
Letter to Ms. Weichert (OMB), 12/4/19
Subj: Improve Training for Program and Project Management and New Voluntary Consensus Standard for Earned Value Management PDF letter
Please consider recommendations for more effective program and project management (P/PM). Both recommendations require replacing the current Voluntary Consensus Standard (VCS) for Earned Value Management System (EVMS) with the new VCS from the Project Management Institute (PMI), The Standard for Earned Value Management (PMI EVM Standard). The PMI EVM Standard should be used in concert with A Guide to the Project Management Body of Knowledge, (PMBOK® Guide).
Implementation of the recommendations will support your memo, dated April 5, 2019, which defined “P/PM competencies to select, assess, and train program and project management talent for the 21st century.” Implementation of the recommendation to include the PMI documents in the Capital Programming Guide Supplement to OMB Circular No. A–11 (Capital Programming Guide) will ensure that those managers receive valid, useful EVM data from contractors. EIA-748 does not provide such assurance.
Letter to K. Fahey, DOD, 12/9/19
Subj: New PMI Standard for EVM: Comparison with EIA-748 and Recommendations to Reduce Costs of DCMA EVMS Compliance Reviews
This letter augments the above letter to OMB Deputy Director Weichert. It provides a recommended template for future DCMA reviews of contractor compliance with the new PMI Standard for EVM when it is used in concert with the PMBOK® Guide. The template includes only those elements of the PMI documents for which no guidance is provided by EIA-748. This letter also includes opportunities to reduce DCMA costs, even if OMB policy is not changed and DOD retains the use of EIA-748.
Letter to OMB Deputy Director M. Weichert, 12/14/19
Subj: Additional Justification to Replace EVMS Standard in OMB Capital Programming Guide
New GAO report cites PMI standards, not EIA-748.
Letter to OMB Deputy Director M. Weichert, 12/16/19
Subj: Recommendations to Improve Program Management and Earned Value Management
My previous letter cited the report, GAO-20-44 Improving Program Management, to justify replacing EIA-748 with the PMI Standard for Earned Value Management (ANSI/PMI 19-006-2019) in the OMB Capital Programming Guide. This letter addresses two findings in that report.
The common thread is the proposal for OMB to adopt program and project management (P/PM) standards from the PMI.
Letter to HASC Chairman Adam Smith, 12/19/19
Subj: Request for Defense Acquisition Reform and GAO Investigation
Excerpts:
GAO report, GAO-20-44 Improving Program Management, dated December 2019, corroborates my assertion that PMI documents are consistent with widely accepted standards for program and project management (P/PM) planning and delivery.
o PMI documents are widely accepted standards for P/PM
o Utilized worldwide
o Generally recognized as leading practices for P/PM
o Approved by the American National Standards Institute (ANSI).
A new PMI standard, Standard for Earned Value Management (ANSI/PMI 19-006-2019), when used in concert with PMBOK® Guide, should be used by DOD instead of EIA-748. These documents meet PMIAA criteria and OMB criteria for a VCS.
o EIA-748 was approved by the Society of Automotive Engineers (SAE)
o It is not ANSI-approved.
o Think about the SAE grade of your motor oil. Capital acquisitions that cost over $100 M should be governed by a higher standard. Taxpayers deserve a higher standard.
There is an urgent need to transform EVM into a cost-effective acquisition management tool that provides reliable information and early warning of performance problems on a consistent basis.
Recommendations It is recommended that you:
1. Request GAO to verify my assertions that:
a. The provisions of PMIAA are not substantially similar to or duplicative of…policy, guidance, or instruction of DOD related to program management.
b. EIA-748 is no longer a VCS per PMIAA and OMB criteria. It is Ineffective, impractical, and obsolete.
Letter to Sen. Harris, 9/22/19
Subj: Support of Bogus Bonus Act and Related Acquisition Reforms
Letter to HASC Chairman Adam Smith, 2/6/20
Subj: F-35 Software Deliveries Using Agile Methods; Schedule Slips and “Technical Debt” per DOT&E
Excerpts:
• The program recently updated its software release schedule to reflect a delivery process termed “agile.” This process culminates in the delivery of a “Minimum Viable Product” (MVP) to the Services every 6 months.
• Software changes, intended to introduce new capabilities or fix deficiencies, often introduced stability problems and adversely affected other functionality.
• This “technical debt,” especially the most significant deficiencies, should be addressed by the program to ensure the SDD baseline configuration of software and hardware is stable, prior to introducing a large number of new capabilities to the software in the new hardware configuration associated with Block 4.
• DOT&E assesses the MVP and “agile” process as high risk due to limited time to evaluate representative integrated developmental test/operational test (IDT/OT) data before fielding the software. Testing will not be able to fully assess fielding configuration of the integrated aircraft, software, weapons, mission data, and ALIS capabilities prior to fielding.
Letter to HASC Chairman Adam Smith, 2/16/20
Subj: Award/Incentive Fees for Software Deliveries Using Agile Methods
Excerpts:
Revise DoDI 5000.02, “Operation of the Adaptive Acquisition Framework,” that would add award/incentive fee criteria that are dependent on:
a. Achieving the goals and features of each software iteration and increment in the Product Roadmap on a timely basis and
b. Reducing the total open deficiencies in the Technical Debt, especially Type I or critical deficiencies.
Letter to HASC Chairman Adam Smith, 5/12/20
Subj: Request for Supplemental GAO Investigation, F-35 Modernization
smith hasc letter may 2020.pdf
Addresses chronic problems on the F-35 JSF program with a request for House Armed Services Committee oversight. Excerpts follow.
The GAO just published its latest report, “F-35 JOINT STRIKE FIGHTER Actions Needed to Address Manufacturing and Modernization Risk on the F-35 Program,” GAO-20-339…Please consider initiating supplemental oversight actions that address the GAO findings. In my opinion, the current findings are symptoms of chronic Lockheed Martin F-35 failures to perform and to report the truth.
Two problems cited by GAO provide evidence that performance and reporting deficiencies, which I have been disclosing for years, have not been fixed, as follows:
- Unrealistic Cost Estimates at Completion (EAC)
- Deferred Functionality
Subject: HR 6395 EH, Sec. 1710N, Agile Program and Project Management; Maximum Viable Profit for Minimum Viable Product?
Letter to DOD Undersecretary of Defense Ellen Lord, 7/17/20
Subj: NASA OIG Report and Reform Implications for the DOD Adaptive Acquisition Framework Lord letter July 2020 pdf
Letter to NDIA IPMD Interim Chair Vaughn Schlegel, 8/20/20
Subj: Recommendations to NDIA IPMD: the Good, the Productive, the Counterproductive letter to NDIA.pdf
i. It is not a Voluntary Consensus Standard per OMB criteria.
ii. It is not a widely accepted standard for P/PM planning and delivery.
iii. It is Ineffective, impractical, and obsolete.
It is recommended that the NDIA IPMD’s Strategic Plan be modified to develop a path to abandon EIA748 and to incorporate the recommendations in EVMS-lite. The House version of the NDAA has been submitted to the Senate. It is also recommended that the NDIA express its support of Sec. 1745 to the Senate.
Letter to HASC Chairman Adam Smith, 10/6/20
Letter to Vice President-elect Harris, Subj: Acquisition Reform in the New Administration, 11/7/20
(Includes, as attachment, letter to OMB Director Vought, above)
Letter to HASC Chairman Adam Smith, 11/21/20
Subject: NDAA Conference Committee Support for H. R. Sec. 1745, Requirements Relating to Program and Project Management
Excerpts from letter:
Please defend H. R. 6395, Section 1745. As I stated in an email to Sen. Harris, I fear that NDIA/CODSIA may have lobbied to reject it to preserve the status quo.
If passed, implementation of your markup will result in lower costs of major capital acquisitions and earlier detection and reporting of cost overruns and schedule delays. Opponents of Sec. 1745 may allege that its passage will increase acquisition costs. They are wrong. My recommendations will result in a reduced regulatory burden, lower contractor costs, and lower compliance-review costs.
Letter to HASC Chairman Adam Smith, 12/12/20
Subj: NDAA Excludes Your Most Game-Changing Provision for Program and Project Management
Letter to Asst. Sec. of Defense for Acquisition, Kevin Fahey, 12/2/20
Subj: Enhance AAF by Publishing a “Government-unique standard” for Earned Value Management Systems
Excerpts:
This letter augments my previous letter, Subj: New PMI Standard for Earned Value Management: Comparison with EIA-748 and Recommendations to Reduce Costs of DCMA EVMS Compliance Reviews, dated Dec. 9, 2019. It includes a recommendation that you can initiate now to:
1. Reduce the costs of Major Capability Acquisitions
2. Provide a practical and contractual vehicle to meet the objectives of the Adaptive Acquisition Framework (AAF).
3. Implement a “Government-unique standard” for Program/Project Management (P/PM) that is “in accordance with standards accredited by ANSI,” as specified in the pending NDAA for FY 2021.
Letter to HASC Chairman Adam Smith, Subj: NDAA for FY 2021 and Unfinished Oversight and Legislation12/20/20 smith letter dec 20 rev1.pdf
Last week, I sent you a letter recommending legislation on Program/Project Management (P/PM) that would restore a provision in your NDAA markup that was receded, per the Conference Report. That was the last of five letters to you, since December 2019, regarding oversight and legislation that will result in lower defense acquisition costs and reduced fraud, waste, and abuse.
This letter includes a recap of the previous correspondence and actions that I hope you will consider for oversight and acquisition reform.
The status quo is a toxic triad of EIA-748, Agile Methods, and subjective award fees. In concert, they enable, not deter, fraud, waste, and abuse. My recommendations are intended to improve transparency and accountability of Major Capability Acquisitions and to reduce the cost of EMD programs that use EVM.
Letter to HASC Chairman Adam Smith, 1/9/21
Subj: Additional, Unfinished NDAA Legislation
Revise DFARS EVMS clause regarding Integrated Baseline Reviews (IBR) to add TPMs.
During (IBRs), the Government and the Contractor will jointly assess the Contractor’s baseline to be used for performance measurement to ensure complete coverage of technical performance measures in addition to the statement of work, logical scheduling of the work activities, adequate resourcing, and identification of inherent risks.
Letter to Sen. Kamala Harris, 1/17/21
Subj: DoD Acquisition Reform, Final Request to You as my Senator
Letter to Sen. Sanders, 2/8/21
Subj: Reduce the Military Budget and Related Fraud, Waste, and Abuse
Recommendations for legislation and oversight...address your goals to reduce the size of the military budget and the amount of waste in that budget.
Please lead DoD and coordinate with OMB to fix the policies, process, and FAR/DFARS. Chairman Smith’s markup to the NDAA for FY 2021 required OMB to adopt government-wide standards, policies, and guidelines for P/PM for executive agencies that are “in accordance with standards accredited by ANSI” (not EIA-748). Unfortunately, that provision was “receded” by the Senate.
If the right fixes are implemented by DoD and OMB, there will be no need for Congressional prodding. EVMS-lite contains a detailed plan for DoD, including interim and long-range actions.
Letter to Deputy Secretary of Defense Hicks, 2/9/21
Subj: DoD Acquisition Reform Needs and Recommendations
Current DoD Instructions and DFARS are not sufficient for effective management of
- Major Capability Acquisitions or
- Software Acquisitions when Agile methods are used.
- Please consider my assessments and recommendations that dovetail with your goals.
Please lead DoD and coordinate with OMB to fix the policies, process, and FAR/DFARS. Chairman Smith’s markup to the NDAA for FY 2021 required OMB to adopt government-wide standards, policies, and guidelines for Program/Project Management for executive agencies that are “in accordance with standards accredited by ANSI” (not EIA-748). Unfortunately, that provision was “receded” by the Senate. If the right fixes are implemented by DoD and OMB, there will be no need for Congressional prodding. EVMS-lite contains a detailed plan for DoD, including interim and long-range actions.
Letter to HASC Chairman Adam Smith, 2/15/21
Subject: Additional Rationale for Resubmittal of H. R. Sec. 1745 to Reduce Defense Costs
The status quo enables today’s contractors to understate true cost overruns and EACs. Passage of your bill would be a first step towards changing the status quo, the Earned Value Management Standard EIA-748, with a standard for Program/Project Management that is in accordance with ANSI-accepted standards.
Reforms are needed more than ever. We must eliminate enablers of contractor deception and ensure early detection of programs that are in trouble. That will prevent waste and reduce defense costs and budgets.
Letter to Sen. Padilla, 2/20/21
Subject: Reduce Capital Asset Costs and Budgets
I hope that you will initiate oversight actions and legislation that can lead to improvement of Program/Project Management (P/PM) policies and reduce the costs of acquiring capital assets for all agencies. We need to revise OMB policy and the Program Management Improvement and Accountability Act of 2015 (PMIAA).
My recommendations are intended to improve transparency and accountability of Major Capability Acquisitions, including embedded software, and to reduce the cost of Engineering and Management Development programs that use EVM based on EIA-748.”
Letter to Sen. Tester, 2/27/21
Subject: Lack of “Good Information” About Defense Programs
I listened to you on “Real Time with Bill Maher” last night. In response to Maher’s comments about the F-35 being a turkey, you responded:
“It’s about accountability” and
“You make good decisions based on good information.“
Unfortunately, the defense acquisition statutes and regulations are insufficient to hold defense contractors accountable for cost, schedule, and technical performance. In fact, the regulations enable defense contractors to submit monthly status reports that contain invalid information and are misleading. There is a systemic problem that enables contractors to understate the true cost overruns. Consequently, DoD program managers do not have good information to make good decisions and are unaware that a Nunn-McCurdy breach has occurred.
Letter to HASC Chairman Adam Smith, 3/8/21
Subject: Comments on Your Brooking Institution Online Conversation
Your online conversation at the Brooking Institution on March 5 included issues that are consonant with mine. Some of the acquisition reforms that I recommended to you in letters since May 2020 directly address your issues. Please consider them for inclusion in the NDAA for FY 2022 or for initiating oversight actions.
Letter to HASC Chairman Adam Smith, 3/13/21
Subject: ”We reward...process, not results” and Earned Value Management
You observed that “We reward people for process, not for results."
The larger problem is that we also reward contractors for process, not for results, on cost plus award fee (CPAF) contracts that require the use of EVM....However, DoD uses subjective award fee criteria that have nothing to do with excellent cost, schedule, or technical performance or with making real progress towards completing “a freaking product at the end of the day.”
A contractor can profit from being “Excellent” in its use of the flawed, discredited EVM process (as prescribed by the EIA-748 standard) regardless of actual cost, schedule, or technical performance. Back to your point, the above EVM process award fee criteria have nothing to do with results as measured by technical performance. The Section 809 Panel reported that “another substantial shortcoming of EVM is that it does not measure product quality. A program could perform ahead of schedule and under cost according to EVM metrics but deliver a capability that is unusable by the customer.”
Had your NDAA bill (Sec. 1745, Requirements Relating to P/PM) not been receded in Conference, then OMB would have to “Adopt governmentwide standards, policies, and guidelines for P/PM for executive agencies that are ‘‘in accordance with standards accredited by the American National Standards Institute (ANSI).’’ The Project Management Institute (PMI) standards meet that accreditation criterion. EIA-748, accredited by the SAE, does not. The PMI standards focus on the product scope. EIA-748 focuses on the statement of work and is silent on the product scope or technical baseline.
Please revive your bill and consider the other legislative and oversight recommendations included in my previous letters.
“Buy Products that Work, not Statements of Work.”
Letter to HASC Chairman Adam Smith, 3/19/21
Subject: GAO Report: Continuing Failures of F-35 Block 4 Software Development and Agile Methods
GAO issued report on the failures of F-35 Block 4 software development. The report, GAO-21-226, F-35 JSF, DOD Needs to Update, Schedule, and Improve Data on Software Development, provides additional evidence that neither DOD nor Lockheed Martin has fixed problems that have been reported since 2010.
- After 3 years of effort, the F-35 program continues to have issues with effectively implementing the C2D2 (Agile) approach to develop and deliver Block 4 capabilities.
Failure of F-35 Program Officials to Establish Targets for Software Quality Metrics
GAO reported that program officials did not establish targets for critical software quality and possibly linking those targets to future contractor monetary incentives, I have recommended specific quality metrics to DOD and a process for linking award/incentive fees to those metrics. However, DOD, has failed to implement the recommendations. Thus, DOD and contractors maintain the status quo. Contractors are not incentivized or held accountable to achieve technical performance objectives within cost and schedule objectives.
Failure of F-35 Program Officials to Account for Deferred Functionality
GAO also stated that program officials should measure progress against the target and forecast the amount of work needed to complete the drop until the increment is complete. By comparing the delivered functionality to the goal, program officials can see how much further the contractor needs to go, and then use that knowledge to help ensure the contractor delivers full functionality in the first increment of a software drop. But no targets have been established. So, there is no way to measure and report actual vs. planned technical and schedule performance or the derived measure of progress, earned value.
Letter to HASC Chairman Adam Smith, 3/22/21
Subject: Second Request for GAO Assessment of F-35 Cost and Schedule Performance Reporting
In the letter to you, dated May 12, 2020, I requested that you initiate oversight actions to address GAO findings in its report, “F-35 JOINT STRIKE FIGHTER Actions Needed to Address Manufacturing and Modernization Risk on the F-35 Program,” GAO-20-339. I had stated that those findings are symptoms of chronic Lockheed Martin F-35 failures to perform and to report the truth. Two problems cited by GAO in 2020, provide evidence that performance and reporting deficiencies that I have been disclosing for years have not been fixed, as follows:
- Unrealistic Cost Estimates at Completion (EAC)
- Deferred Functionality
In my letter to you, dated March 19, 2021, I cited the 2021 GAO report, to repeat the unresolved deferred functionality issue. Today, I also address the unresolved EAC issue.
Unrealistic, Optimistic EAC
GAO 2020:
F-35’s Block 4 development cost estimate did not meet the GAO Leading Practice that “The cost estimate results are unbiased, not overly conservative or optimistic and based on an assessment of most likely costs.” Per GAO, “Congress is left without a complete picture of what DOD intends to spend on the total Block 4 effort.”
GAO 2021, GAO-21-226:
The airframe contractor’s planned work is too optimistic and not reflective of historical performance. For example, for a recent software drop, Lockheed Martin completed only 64 percent of planned work and delivered only 69 percent of planned functionality on time…the contractor is consistently delivering less than the planned amount of work that was expected to be completed. As a result, we found that the remaining schedule is not achievable as it is based on optimistic assumptions about the amount of work that can be completed and is not rooted in reality.
Recommended HASC Oversight
It is now six years since I received Sen. McCain’s letter. My request to you from last year is still pertinent: Examine whether Lockheed Martin’s reporting of Block 4 cost and schedule performance to DOD and to Congress is credible and accurate.
Letter to Acting Deputy Under Secretary of Defense Cummings, 3/30/21
Subj: Enhance AAF by Publishing a “Government-unique standard” for Earned Value Management Systems
I have read the former President’s Management Agenda (PMA) and previously sent recommendations to former Dir. Vought that will meet PMA objectives to:
- Reduce the accumulated regulatory burden.
- Improve management of major acquisitions.
Most importantly, my recommendations will reduce military spending and Buy Products that Work, not Statements of Work. Please read the attached letters to former Dir. Vought, Sen. Sanders, and former Dep. Director Zients regarding acquisition reforms that should be implemented through OMB Circular A-11.
Letter to HASC Chair Adam Smith, 4/14/21
Subject: Your National Security Issues; Acquisition Policies and Requirements
Letter to HASC Chair Adam Smith, 4/15/21
Subject: Your Rejected NDAA Provision for Program and Project Management
with attached email to former SASC Gen. Counsel Peter Levine and former HASC staffer, Andrew Hunter, Subj: Acquisition Reform Provision in NDAA, 11/5/2010
Letter to HASC Subcommittee Chairman Donald Norcross,
Subj: Questions for Subcommittee Hearing on F-35 Issues on April 22, 4/18/21
Excerpts:
Includes questions that should be asked at the upcoming hearing, “Update on F-35 Program Accomplishments, Issues, and Risks,” to get to the truth regarding the status of the F-35 Block 4 Modernization program.
Excerpts:
GAO Assessment of Ineffective F-35 Metrics
- GAO has repeatedly reported on the failures of F-35 Block 4 software development. GAO concluded that after 3 years of effort, the F-35 program continues to have issues with effectively implementing the C2D2 (Agile) approach to develop and deliver Block 4 capabilities.
- Contractor continues to deliver capabilities late
- Remaining schedule contains significant risk and is not achievable based on the pace of past performance
- Block 4 metrics for software quality, performance, cost, schedule provide limited insight into aspects of software development quality under the Agile software development approach.
- Current metrics do not include the number of planned features and the number of completed features for each software increment, which would provide insight into progress against the planned schedule and help ensure that all capabilities are delivered as planned in the first increment of each software drop.
Recommended Hearing Questions
Have you established targets for critical software quality and forecasted the amount of work needed to complete a software drop until the increment is complete?
Do targets include the number of planned features for each software increment and do you report the number of completed features for each software increment?
Do you report progress towards meeting those targets in monthly Earned Value Management System Integrated Program Management reports or other, contractually required reports?
Do you account for deferred functionality in your forecasts (of final cost and schedule) for completion when you fail to complete the planned features or other quality targets on schedule?
Are your monetary incentives based on meeting established targets for critical software quality?
If monetary incentives are not based on meeting critical software quality targets, what are the incentive criteria?
Considering deferred functionality and other quality/technical performance issues, when will the F-35 Block 4 Modernization Program be complete and what is the final cost estimate?
Letter to HASC Subcommittee Chairman Donald Norcross, Subj: Today’s F-35 Hearing, Deceptive (or No) Performance Metrics, and Ponzi Schemes, 4/22/21
Excerpts:
In watching today’s hearing, I realized that there are great similarities between the F-35 program and:
1. The Casablanca scene in which Captain Renault said: “I'm shocked, shocked to find that gambling is going on in here!”
2. A Ponzi Scheme
So, I was not shocked, shocked to hear from Bg. Gen. Abba and GAO witness Diana Maurer that performance measures for the development of ALIS still do not exist. As you know, the ALIS failures are a major cause of high sustainment costs.
Systemic Lack of Technical Performance Metrics (TPM)
The lack of performance metrics and the focus on how much work was performed (instead of progress towards meeting the product requirements) are chronic and systemic in DOD. I have reported this to HASC Chairman Ike Skelton and to SASC Chairman McCain. I proposed specific recommendations to fix the problem to them and to DOD and the NDIA for over 20 years. However, all stakeholders preferred to maintain the status quo.
Letter to HASC Subcommittee Chairman Donald Norcross, Subj: More on Deceptive Performance Metrics and Ponzi Schemes, 4/25/21
Excerpts:
In practice, Management Reserve budget is often transferred to cover additional rework, drawings, code, or tests and hide the true cost overrun. It is used as a slush fund.
Deceptive MR Practices are Systemic and Pervasive in DOD Contracts
The whistleblower lawsuit was terminated in 2017 after five years of litigation. So, you may ask, why are the allegations relevant today? Because the contracting environment, including DFARS and EIA-748, has not been fixed.
If you authorize GAO to assess the F-35 program and similar development programs, I am confident that it will find current examples of MR being used for rework, additional drawings etc. and verify that the true cost overruns are hidden.
Letter to DCMA Director, Lt. Gen. David G. Bassett, Subj: Recommendation to Revise Mission, Methods, and Name of DCMA EVMS Center, 4/29/21
Excerpts:
The DCMA website states “DCMA is, first and foremost, a product delivery organization. Our nation’s warfighters expect our defense industry to produce and deliver the equipment they need to fight, survive and win.”
However, the mission and methods of the DCMA Earned Value Management Systems (EVMS) Center are insufficient to assess a contractor’s effectiveness in reporting progress towards developing the product scope or technical baseline.
It is recommended that you initiate reforms to meet the following objectives:
1. Rename the EVMS Center
2. Redefine its mission and methods to focus on
a. Integrated Program/Project Management (P/PM) and
b. Integration of EVM with Systems Engineering (SE).
c. Eliminating reviews of contractor compliance with EIA-748, EVMS Standard Guidelines.
d. Reviewing contractor’s compliance with the EVM elements in policies and guidelines for P/PM that are in accordance with standards accredited by the American National Standards Institute (ANSI), namely Project Management Institute (PMI) standards.
e. Reviewing contractor’s use of the Systems Engineering Management Plan (SEMP) and policies and guidelines for P/PM that are in accordance PMI standards.
Excerpts:
This is a follow-up to my letter to you dated February 6, subj: “Reduce the Military Budget and Related Fraud, Waste, and Abuse.” I have additional information that addresses issues that you expressed in your hearing on May 12, subj: “Waste, Fraud, Cost Overruns, and Auditing at the Pentagon” and in today’s article in the WP.
I agree with you that “large defense conglomerates, as taxpayer-funded entities, should face the same standard of accountability and transparency as their government counterparts.” However, as discussed in my first letter, current OMB policy and federal regulations enable contractors to deceive DoD program managers regarding true cost and schedule performance.
Letter to DCMA Director Lt. Gen. David G. Bassett, Subj: Amended Recommendation to Address HASC Chairman Smith’s Issues, 5/15/21
Cited white paper, “DOD Acquisition Reform: EVMS-lite to Program/Project Management, Rev. 25“ dated 7/19/21 (current version).
Letter to DCMA Director Lt. Gen. David G. Bassett, Subj: DCMA EVMS Compliance Procedures and Metrics Ignore Technical Performance Measurement, 5/16/21
Excerpts:
Both the DCMA EVMS compliance procedures and the DCMA EVMS Compliance Metrics (DECM) are silent on technical performance. Consequently, there is no assurance that the DCMA EVMS Center can accomplish its mission of “assessing contractor effectiveness which provides stakeholders with expectations of future performance and potential impacts on individual contractors and/or programs.”
Please expand the scope of DECMs and DCMA EVMS compliance reviews to include the effective use of TPMs. Support OMB and DOD needs for integrated TPM and EVM.
Letter to Sen. Warren, Subj: Lowering Defense Costs and Initiating Acquisition Reforms, 5/18/21
Letter to HASC Chairman Adam Smith, Subj: Failure to Meet PARCA Goal, “More Accurate Situational Awareness of Program Execution, 5/24/21, with attached white paper, “DOD Acquisition Reform: EVMS-lite to Program/Project Management, Rev. 25,“ dated 7/19/21 (current version).
Letter to Counselor to the President Jeff Zients, Subj: Acquisition Reforms to Reduce Defense Costs, dated June 1, 2021
When you were in OMB, I sent the attached letter, dated 12/13/09, regarding acquisition reforms for procuring capital assets, especially new weapon systems. These reforms are still needed to reach the following objectives:
- Acquire needed weapons within Pres. Biden’s budgetary goals.
- Increase transparency and accountability regarding acquisition of capital assets.
- Require contractors to submit valid reports of cost, schedule, and technical performance.
- Identify unfavorable variances requiring corrective actions sooner.
- Identify programs that should be cancelled, including Nunn-McCurdy breaches, years earlier. Achieve HASC Chairman Smith’s objective:
i. “We have wasted a spectacular amount of money on weapons systems that either haven’t worked at all or who have not lived up to their promise. The failure we wind up tolerating is failure on a massive freaking scale. Think F-35.”
ii. “I want to stop throwing money down that particular rathole.”
- Remove a barrier to entry to potential commercial bidders on defense contracts.
- Improve Program/Project Management training of civilian and military personnel.
Letter to HASC Chairman Adam Smith, Subj: DOD’s Failure to Perform, 6/4/21
Attached letter of appreciation from DCMA Dir. Scott, June 2004
Attached emails, HASC staffer Andrew Hunter, March-April 2010
Attached email, Asst. Sec. of Defense Katrina McFarland, 8//7/13
Attached email, Defense Legislative Fellow to Sen. McCain, 5/14/14
Attached letter of appreciation from Sen. McCain, 3/5/15
Excerpt: previous letters..include legislative and oversight recommendations that are needed to fix a corrupt weapons acquisition system and lower defense costs. Sen. McCain’s goals for WSARA have not been met.
Letter to HASC Rep. Jackie Speier, Subj: Request to Resubmit HR 6395, Sec. 1745, in NDAA for FY 2022, dated 7/2/21
Letter to HASC Rep. Jackie Speier, Subj: Addendum to Request to Resubmit HR 6395, Sec. 1745, dated 7/2/21
Letter to HASC Rep. Julia Brownley, Subj: Letters to Rep. Speier, dated 7/3/21
Letter to Sen. Joni Ernst, Subj: Proposed NDAA Markups to Fix the PMIAA; Delete “shall not apply to DoD,” dated 7/7/21
Letter to Sen. Bernie Sanders, Subj: Legislation to Reduce Acquisition Costs and Related Fraud, Waste, and Abuse,” dated 7/11/21
Letter to OMB Deputy Director S. Young
Subj: Recommendations to Improve Program/Project Management and Reduce Military Spending, dated 7/12/21
Letter to HASC Chairman Adam Smith, Subj: NDAA for FY 2022 Subcommittee Markups Omit Requested Provision, dated 7/29/21
Letter to USD(A&S) Gregory Kausner, Subj: Recommendations to Amend NDAA and DoD Policy: Agile Methods, Technical Debt, and Award/Incentive Fees, dated 8/6/21
Letter to USD(A&S) Gregory Kausner and HASC Chairman Smith, Subj: Refined Amendment to NDAA to Assess Agile Practices on F-35 Block 4 Software Development, dated 8/10/21
Excerpts:
augments the letter sent to Undersecretary Kausner on August 6, subj: Recommendations to Amend NDAA and DoD Policy; Agile Methods, Technical Debt, and Award/Incentive Fees. It provides additional, refined scope to the proposed amendment..includes Software Engineering Institute’s assessment of Agile practices on the F-35 Block 4 software development. The refinements address issues raised by Dr. Raymond O’Toole, Jr., Acting Director, OT&E in his testimony to the HASC Tactical Air and Land Forces Subcommittee (TAL) on July 13, 2021.
Dr. O’Toole addressed Lockheed Martin’s failures to meet the objectives of the Agile framework, known as Continuous Capability Development and Delivery (C2D2). The C2D2 process has failed to field a new software increment, known as a “minimum viable product” (MVP), every six months. He concluded that the six-month C2D2 cycle is not sound.
Letter to HASC Chairman Adam Smith, Subj: F-35 Block 4 Technical Debt Enhancements; Word Ploy? Word Play? --- Oy Vey!, dated 8/13/21
Excerpts:
GAO-21-222 Weapon Systems Annual Assessment: “The program office also noted that, subsequent to our review, it had recategorized 178 of the 861 category 2 deficiencies as “enhancements”.”
No information was provided to justify or clarify the recategorization. So, I requested that you
“add a review of the JPO’s failure to perform”.,,,If category 2 deficiencies could formerly impede mission success, how can they now be considered enhancements? What is an enhancement? Is this another scheme to conceal deferred or reduced functionality?”
new issues.
These deficiencies are the “technical debt” that was discussed in my letter to Undersecretary Kausner, Subj: Recommendations to Amend NDAA and DoD Policy; Agile Methods, Technical Debt, and Award/Incentive FeesI, dated August 6.
The JPO’s use of the term “enhancements” is ambiguous. Dr. O’Toole’s silence on “enhancements” is telling. Was the JPO’s choice of words accurate and informative or was it a play on words, or worse, a word ploy?
Letter to HASC Rep. Jackie Speier, Subj: Request to Amend NDAA for FY 2022 or Billion Dollar Boondoggle Act, dated 8/19/21
Email Letter to HASC Chairman Adam Smith, Subj: NDAA Markups Recapped, dated 8/26/21
Recommended NDAA markups were included in letters sent to you or upon which you were copied. The table below is a recap of the markups. Please consider them for next week’s markup.
Date To Topic
7/2 Speier Program and Project Management (P/PM)
Your Unfinished Business
I really appreciate your attempt, last year, to force DoD to adhere to a
higher standard. It reminded me of Hebrew National salami.
Unfortunately, your markup to the NDAA for FY 2011, HR 6395, Sec.
1745, Requirements Relating to Program and Project Management, was
receded, per the Conference Report. Please read my letter to Chairman
Smith, Subj: NDAA Excludes Your Most Game-Changing Provision for
Program and Project Management, dated Dec. 12, 2020. I had requested
that he again include the provision in the NDAA for FY 2022. I did not
realize, then, that the provision was yours. I hope that the provision will
not be rejected this year as the Senate no longer has a Republican
majority.
Date To Topic
8/19 Speier Award
Fees
Please consider amending Boondoggle Act to restore the omission, as
follows.
TITLE V—PREVENTING UNJUSTIFIED AWARD FEES
SEC. 501. ENCOURAGING EXCELLENT CONTRACT PERFORMANCE.
(a) LIMITATION.—For any cost-based contract entered into by
the Federal Government that includes an award or incentive
fee—
(2) at a minimum, the following factors shall be considered in
making a determination regarding whether, and in what
amount, the fee shall be paid to the contractor:
(A) Whether the contractor met cost goals.
(B) Whether the contractor met schedule goals.
(C) Whether the contractor met performance goals and
delivered the goods or services required to be
provided under the contract.
Alternately, please coordinate with Chairman Smith to mark up the
NDAA to restore the omission.
Date To Topic
7/7 Ernst (P/PM)
Request GAO to review DoD policy, guidance, and instructions to
determine if PMIAA is applicable to DoD. (I expect that GAO will verify
and corroborate my allegations).
3. Determine if a markup to NDAA is needed to transform PMIAA into a
law that requires all federal agencies to “adopt widely accepted (or ANSI-
approved) management standards that are often used in the private
sector, …ensure that taxpayer dollars are safeguarded by increasing
accountability throughout the federal government.”
Date To Topic
8/13 Smith
Augment my previous recommendations to amend the NDAA mark up.
Request Software Engineering Institute to:
1. Obtain a definition of “enhancements” as used by the F-35 JPO.
2. Break down the technical debt into three or four components, as
applicable: SDD residual defects, TR-3 defects, new capability
defects, enhancements.
3. Provide estimated, time-phased burndown plans for the above
components.
Request GAO to assess F-35 Block 4 final costs and schedule.
Date To Topic
7/14 Smith Agile Methods, Technical Debt, F-35 Award Fees
Also, please add a review of the F-35 program office’s failure to perform.
It recategorized 178 of the 861 category 2 deficiencies as
“enhancements.” If category 2 deficiencies could formerly impede
mission success, how can they now be considered enhancements? What
is an enhancement? Is this another scheme to conceal deferred or
reduced functionality?
It is recommended that you request GAO to expand the scope of
requested Report on Agile P/PM, to include “other information” that
includes:
1. An assessment of the effectiveness and use of incentive and award
fees when Agile P/PM is used to acquire software that is embedded
in weapon systems and other military-unique hardware systems.
2. Per my letter to you dated Oct. 6, 2020:
a. I am concerned that the current policies, guidance, and
regulations do not provide sufficient incentives to deliver the
Minimum Viable Product (MVP) within cost and schedule
objectives and to reduce the technical debt. More
importantly, the lack of disincentives or financial penalties
provide no deterrence to contractors from continuing to
receive undeserved award fees. Without more contractual
definition and oversight, contractors are enabled to earn the
Maximum Viable Profit despite delivering a subjectively
defined and downwardly revised Minimum Viable Product.
b. Please include appropriate provisions in the NDAA markup.
Conduct a study of the incentive or award fees most recently paid on
the F-35 Lightning II JSF…contract.
(2) ELEMENTS.—The study (1) shall include an assessment of the
following
(A) the extent to which the Department has established award or
incentive fee criteria that are based on the verification, on a
timely basis, of the quality and technical maturity of the
features and functions of the product being developed and
tested and that the Department has defined and
documented technical performance objectives such as
technical performance measures, progress against
requirements, and exit criteria of life-cycle phases.
(B) the extent to which the prime contractors have reported
schedule performance in their contract performance
reports for the most current incentive or award fee
period that is based on verified, objective measures of
technical performance.
(C) the extent to which the prime contractors have been paid
award or incentive fees, in the most recent period, that
is based on verified technical and schedule performance
Letter to Sen. Warren, Subj: Lowering Defense Costs and Initiating Acquisition Reforms; Part 2, 9/8/21
Letter to Defense Acquisition University President James Woolsey, Subj: NDAA for FY22 Acquisition Workforce Educational Partnerships revised, 9/14/21
attached white paper, "THE DoD ADAPTIVE ACQUISITION FRAMEWORK AND PROJECT MANAGEMENT STANDARDS," Sep. 13, 2021
Letter to USD(A&S) Gregory Kausner and HASC Chairman Smith, Subj: Revision to Letter dated August 10, Further Refined Amendment to NDAA to Assess Agile Practices on F-35 Block 4 Software Development, 9/15/21
Letter to HASC Chairman Adam Smith, Subj: Request to Revise H.R. 4350 Report 117–118 regarding Agile Contracting Mechanism, dated 9/19/21
Letter to USD(A&S) Gregory Kausner (copy to HASC Chairman Smith), Subj: Sec. 854 of NDAA for FY 2022-Recommendations for Cadre of Software Development and Acquisition Experts, 9/22/21
Letter to HASC Chairman Adam Smith, Subj: Proposed NDAA Conference Committee Action on Sec. 852 Technical Debt and Agile Methods, dated 9/24/21
Letter to USD(A&S) Gregory Kausner (copy to HASC Chairman Smith), Subj: Support for Two USD(A&S) Goals, 9/27/21
attached white paper, "THE DoD ADAPTIVE ACQUISITION FRAMEWORK AND PROJECT MANAGEMENT STANDARDS," Sep. 13, 2021
(this letter is augmented by letter dated 10/4/21)
Excerpts from letter:
EVMSIG is obsolete. It was published in 2018, before the new AAF policies. Also, it lacks guidance to address the development of embedded software and the use of Agile methods. EVMSIG should be revised to incorporate best practices from GAO guides and to be consistent with AAF policy and DoD instructions.
Attachment B is a table of requested EVMSIG revisions that are consistent with GAO best practices and AAF policies. A revised EVMSIG will enable improved situational awareness of the F-35 program and all similar programs.
The white paper provides guidance to employ the AAF policy, “Employ Performance Based Acquisition Strategies” and to select standards of the Project Management Institute for program and project management (P/PM) training. The overhaul will also support DoD objectives to attract new companies with emerging technologies and to ensure supply chain resiliency.
Letter to USD(A&S) Gregory Kausner, Subj: Misleading Guidance in NDIA Guides for Program Management and EVMS, dated 9/29/21
Letter to USD(A&S) Gregory Kausner, Subj: Additional Revisions to DoD EVMSIG, dated 10/4/21
(augments letter dated 9/27/21)
Letter to USD(A&S) Gregory Kausner, Subj: Modernizing the DoD EVMIG, dated 10/5/21
Letter to USD(A&S) Gregory Kausner, Subj: EVMIG Award Fee Criterion to Integrate Systems Engineering with EVMS, dated 10/7/21 and
attached email to Ms. Lord, dated 9/27/2020
Letter to USD(A&S) Gregory Kausner, Subj: Digital Engineering, GBSD Program, and Award Fees, dated 10/11/21
Letter to USD(R&E) Heidi Shyu, Subj: Modernize SE Processes to Leverage Digital Engineering and MBSE; Reduce Risks to GBSD Program, dated 12/16/21
DoDD 5000.01, The Defense Acquisition System (DAS), includes policies to speed up delivery of products that work as planned, e.g., products that meet the documented capability needs. However, several DoD instructions and guides should be revised to better enable achievement of DAS objectives. Revisions are needed to benefit programs with the following characteristics:
- Use the embedded software path to develop software embedded in weapon systems
- Employ Digital Engineering metrics
- Employ Model-Based Systems Engineering
Letter to Deputy Secretary of Defense Hicks, subj: Third Request to Revise DCMA EVMS Compliance Procedures and Metrics regarding Technical Performance; Implications for NDAA Sec. 16, Ground-Based Strategic Deterrent Program, dated 12/20/21
Letter to USD(R&E) Heidi Shyu, Subj: Continuing Failures in Using Agile Methods on F-35 Block 4 Modernization, dated 2/2/22
Letter to OMB Acting Director Shalanda Young, Subj: Improve Effectiveness of OMB Circular A-11, dated 2/21/22
Letter to OMB Director Shalanda Young, Subj: Improve Effectiveness of OMB Circular A-11 with Questions for NDIA, dated 3/30/22
Attached white paper, "DOD Acquisition Reform: EVMS-lite to Program/Project Management, Rev. 35" dated 3/30/22
Attached white paper, "Integrating the Embedded Software Path, Model-Based Systems Engineering, MOSA, and Digital Engineering with Program Management, " revised 8/21/23.
Letter to USD(A&S) William LaPlante, Recommendations for Pending Program Management Guides and DCMA EVMS Compliance Metrics, dated 6/5/22
Excerpts:
Recommendations for Pending Guides
Please consider the recommendations herein for inclusion in the pending program management guides and to improve the effectiveness of DCMA EVMS compliance. The pending program management guides are:
A Guide to Program Management (PM) Knowledge, Skills, and Practices
A Guide to DoD PM Business Processes
They should cite and be consistent with the Engineering of Defense Systems Guidebook, Feb. 2022 (Eng Guidebook), Ch 2.5, Systems Engineering Role in Contracting.
Letter to USD(R&E) Heidi Shyu, Subj: Error and Omission in DoD Risk, Issue, and Opportunity Management Guide, dated 7/7/22
Letter to USD(R&E) Heidi Shyu, Subj: Improve DoDI 5000.89, T&E, regarding Technical Debt and Deferred Functionality, dated 8/26/22
Letter to USD(A&S) William La Plante, dated 12/13/22, Subject: DoD Implementation of NDAA for FY 2023 Section 903
Excerpts:
I am pleased that the NDAA for FY 2023 contains Sec. 903, “Limitation on use of funds pending demonstration of product to identify, task, and manage congressional reporting requirements.” Sec. 903 limits the funds authorized to be appropriated or expended by this Act until the Secretary of Defense demonstrates a minimum viable product (MVP).
I had previously submitted recommendations to you and to those copied below to improve pertinent policies, guidance, and regulations. Please consider implementing those recommendations to enable success during development and demonstration of MVPs.
Letter to SASC Chair Tammy Duckworth dated 1/13/23, Subject: Acquisition Reforms on DoD Major Acquisitions and Software Development (redacted)
duckworth letter jan 13 redacted.pdf
Letter to Senator Susan Collins dated 4/29/23, Subject: Unfinished Work Regarding Your WSARA Amendment on Earned Value Management
Letter to DoD IG Assistant Inspector General for Audit Carol N. Gorman, Subj: Additional Objectives for Audit of DoD Oversight of Cost-Plus-Award-Fee Contracts (Project No. D2022-D000AT-0175.000), May 25, 2023
Letter to DoD IG Assistant Inspector General for Audit Carol N. Gorman, Subj: Supplement to Letter Dated May 25 Covers Earned Value Management System Interpretation Guide (Project No. D2022-D000AT-0175.000), May 30, 2023
Letter to Sen. Tammy Duckworth, dated 7/17/23, Subj: Proposed Amendment to NDAA for FY2024, Sec. 815, Earned Value Management System Requirements
Letter to Sen. Tammy Duckworth, dated 7/21/23, Subj: GAO Report and Need to Revise NDAA Sec. 815, Earned Value Management
Letter to USD(A&S) William La Plante, dated 8/1/23, Subj: Software Acquisition Policy and Congressional Oversight Issues
Letter to HASC Rep. Donald Norcross dated 8/8/23, Subj: F-35 C2D2 Oversight Issues Omitted from House Version of NDAA Sec. 219
Letter to HASC Rep. Donald Norcross dated 8/11/23, Subj: GAO Assessment of Reported F-35 Block 4 Modernization Expected Costs
Letter to Lt. Gen. Michael Schmidt dated 8/14/23, Subj: Block 4 Issues, Outcome-Based Metrics, and Systems Engineering Transformation
Letter to HASC Rep. Donald Norcross dated 9/12/23, Subj: Updated Request for Oversight of F-35 C2-D2 Oversight Issues including F-35 Block 4 Modernization Expected Costs
GAO and DOT&E reported the F-35 program’s lack of:
- Performance metrics and outcome-based metrics.
- Clear articulation of the capabilities required in the Minimum Viable Product (MVP), focused testing, comprehensive characterization of the product, and full delivery of the specified operational capabilities.
- Continual assessment of the value of capability delivered to support iterative software development.
Letter to HASC Vice Chairman Robert Wittman dated 11/24/23, Subj: F-35 TR-3 Upgrade Delays, NDAA Revisions, and F-35 Hearing Agenda
Letter to HASC TAL Subcommittee Chairman Robert Wittman dated 12/5/23, Subj: Additional Questions for Pending F-35 Hearing including TR-3
Letter to HASC TAL Subcommittee Vice Chairman Robert Wittman dated 12/7/23, Subj: Your Comments on “software at the forefront”
Letter to HASC TAL Subcommittee Vice Chairman Robert Wittman dated 12/7/23, Subj: Questions for F-35 Acquisition Program Update Hearing on Dec. 12
Letter to HASC TAL Subcommittee Vice Chairman Robert Wittman dated 12/12/23, Subj: Today’s hearing: Your Comments on F-35 Digital Engineering and Lessons Learned
Letter to HASC Vice Chairman Robert Wittman dated 12/13/23, Subj:
More Lessons Learned: "Earned Value? We don't need no stinking Earned Value."
Letter to HASC Vice Chairman Robert Wittman dated 12/16/23, Subj:
“Something of Value” not "Earned Value"
Letter to USD(A&S) William LaPlante, Subj: NDAA for FY 2024, Earned Value Devalued, dated 12/17/23
Letter to USD(R&E) Heidi Shyu, Subj: Include Technical Debt in Policy and Guidance (NDAA for FY 2022, Sec. 835 report) dated 12/26/23
Letter to USD(R&E) Heidi Shyu, Subj: Award/incentive fee criteria should include reduction of Technical Debt, dated 12/27/23
Letter to USD(R&E) Heidi Shyu, Subj: Improve Software Acquisition by Revising IMP and IMS Preparation and Use Guide, dated 1/3/24
Letter to Deputy Secretary of Defense Kathleen Hicks, subj: NDIS Implementation Plan and NDAA Sec. 827, Modification to EVMS Requirements, dated 1/22/24.
with attachment, "Deja vu all over again"
Letter to USD(A&S) William LaPlante, Subj: Outcome-based Metrics that Work to Build a Product that Works; F-35 Block 4, dated 1/29/24
Letter to Dep. Sec. of Defense Kathleen Hicks, subj: NDIS Implementation Plan and Final Report of the Commission on PPBE Reform, dated 3/7/24.
Letter to David L. Norquist, President & CEO, NDIA, subj: DoD’s Need for Output-based Performance Metrics, dated 3/21/24.
Email to Sen. Warren, subj: Your Questions about Program Management Information at SASC Hearing plus "Sentinel Missile; F-35 Deja Vu?," dated 3/30/24
Letter to HASC Vice Chairman Robert Wittman dated 4/8/24, Subj: Systemic Shipbuilding Delays, Congressional Defense Modernization Caucus, and Earned Value
Letter to Secretary of the Navy Carlos Del Toro, Subj: Holding Companies and Individuals Accountable for Reporting Poor Performance, dated 4/11/24
Letter to HASC Vice Chair Robert Wittman, Subj: F-35 TR-3 Delays Surprise; More Evidence of Pervasive Lack of Outcome-based Metrics, dated 4/18/24
Letter to USD(A&S) William LaPlante, Subj: DFARS Revision EVMS Case Fails to Meet Congressional Needs and GAO Recommendations, dated 5/5/24
Letter to USD(A&S) William LaPlante, Subj: DFARS Revision EVMS Case Fails to Meet Your Commitments to SASC, dated 5/7/24
Letter to USD(A&S) William LaPlante, Subj: More on DFARS Revision EVMS Case Fails to Meet Your Commitments to SASC, dated 5/10/24
Letter to HASC Vice Chair Robert Wittman, Subj: Second Request for NDAA Markup to Obtain Outcome-based Metrics, dated 5/16/24
Letter to USD(A&S) William LaPlante, Subj: F-35 Block 4 Subprogram Should Put Its Metrics Where Its Mouth Is, dated 5/16/24
Letter to USD(A&S) William LaPlante, Subj: Navy Frigate’s Failure Obscured by Metrics Based on Quantity, not Quality. RIP, EVM., dated 5/29/24
Letter to USD(A&S) William LaPlante, Subj: Place DFARS EVMS Case on Hold and Go the Whole Hog, dated 6/2/24
Letter to Lt. Gen. Gregory L. Masiello, Director, DCMA, Subj: DCMA EVMS Compliance Procedures and Metrics Ignore Technical Performance Measurement, Third Request, dated 6/9/24
Letter to USD(A&S) William LaPlante, Subj: NDAA for FY 2025, Digital Engineering and Digital Twins, dated 6/13/24
Letter to USD(A&S) William LaPlante, Subj: Digital Engineering Acquisition Policy and Congressional Oversight Issues, dated 6/25/24
Letter to USD(A&S) William LaPlante, Subj: LGM-35A Sentinel Program Fails to Provide Early Warning, dated 6/25/24
Letter to Sen. Elizabeth Warren: Failures of Sentinel, F-35, and Digital Engineering, dated 6/28/24
Letter to USD(A&S) William LaPlante, Subj: Realpolitik on Digital Engineering, Outcome-Based Metrics, and Project 2025, dated 7/17/24
Letter to HASC Vice Chair Robert Wittman, Subj: Lockheed Martin VP Bridgette Lauderdale, Put Your Metrics Where Your Mouth Is, dated 7/21/24
Letter to Subcommittee Chair Glenn Grothman. Subj: Unfinished Business to Reduce DOD Wasteful Spending and Inefficiencies, dated 7/28/24
Letter to Dep. Sec. of Defense Kathleen Hicks, subj: Commission on the National Defense Strategy (NDS) Final Report, dated 7/31/24.
Letter to USD(A&S) William LaPlante, Subj: Remedy for Toxic Mix of Earned Value Management, Agile Methods, and Award Fees, dated 8/10/24
Letter to USD (Comptroller)/CFO Michael McCord, Subj: PBBE Recommendations Regarding Output-based Performance Metrics, dated 8/29/24
Email, Subj: PBBE Reform: Performance Metrics Towards Completing the Freaking Product, August 31, 2024
Letter to Defense Subcommittee Chairman Ken Calvert, Subj: Pull Down the Curtain on Program Management Failures and Flawed Metrics,dated Sept. 19, 2024
Letter to Defense Subcommittee Chairman Ken Calvert, Subj: Another Navy Program Management Failure and Flawed Earned Value Metrics, dated Oct. 1, 2024
Email to NDIA, Subj: Comments Needed to add Technical Performance Measures to proposed NDIA IPMD Surveillance Guide, dated October 3, 2024
Excerpts:
The NDIA IPMD…updated the NDIA Surveillance Guide, revision 5, dated September 30, 2024…seek updates from NDIA IPMD members within 30 days.
Current version…lacks guidance, project artifacts, and examples to integrate technical performance measures with EVM. There are also gaps with TPM guidance in the DOD EVMSIG.
…the examples of project artifacts for review are silent on technical performance measures.
Letter to HASC Ranking Member Adam Smith, Subj: Outcome-based Metrics, Barrier to Entry, and NDIS Implementation Plan, dated Oct. 29, 2024
Letter to Trump Transition Committee, Subj: DoD De-Regulation - Under Budget and Ahead of Schedule, 8 years Later, dated November 6, 2024
Letter to Trump Transition Committee, Subj: 2025 Presidential Transition Project: PMI Project Management, not NDIA Earned Value, dated November 7, 2024
Letter to Trump Transition Committee, Subj: 2025 Presidential Transition Project: PMI Project Management, not NDIA Earned Value (Part 3), dated November 10, 2024
Letter to Mr. Musk and Mr. Ramaswamy, Department of Government Efficiency (DOGE), Subj: Recommendations to Drive Out Waste and Fraud at DOD, dated November 12, 2024
Letter to Lt. Gen. Gregory L. Masiello, Director, DCMA, Subj: DCMA Earned Value Management Award Fee Guidance, dated 11/16/24
Letter to HASC Ranking Member Adam Smith, Subj: Subject: Outcome-based Metrics, Barrier to Entry, and NDIS Implementation Plan, Part 2, dated Nov. 17, 2024
Letter to Hon. Russell Vought, Director-nominee Office of Management and Budget, Subj: “When you come to a fork in the road…”, dated November 23, 2024.
Letter to OMB Director Shalanda Young, Subj: Transition Actions for You and Chief of Staff Zients, dated November 24, 2024.
Joni Ernst letter, 11/26/24, Subject: Happy Thanksgiving to You, Mr. Musk, and Mr. Ramaswamy
Excerpts from Eng Guidebook: Another area to which incentives are tied is the EVMS. The PM should ensure that the EVMS, tied to any incentive, measures the quality and technical maturity of technical work products instead
of just the quantity of work. If contracts include EV incentives, the criteria should be stated clearly and should be based on technical performance. EV incentives should be linked quantitatively
with:
• Technical Performance Measures (TPM)
• Progress against requirements
• Development maturity
• Exit criteria of life-cycle phases
• Significant work packages and work products
They should also include a criterion, if applicable, that “EV incentives should be linked quantitatively with acceptance criteria of Minimum Viable Products (MVP).”
Note: For, additional guidance to implement Eng Guidebook, see Defense AT&L article, "Systems Engineering (SE) and EVM Support for Performance-Based Awards," Jan. 2007 Link to PDF
NDIA
Excerpts:
I am pleased that the NDAA for FY 2023 contains Sec. 903, “Limitation on use of funds pending demonstration of product to identify, task, and manage congressional reporting requirements.” Sec. 903 limits the funds authorized to be appropriated or expended by this Act until the Secretary of Defense demonstrates a minimum viable product (MVP).
CODSIA
Council of Defense and Space Industry Associations letter to DoD,
Ref: DoD Report to Congress on Implementation of Earned Value Management: Request for Industry Input, July 2, 2009 PDF Letter
The position paper regarding earned value management system (EVMS) includes CODSIA’s position that the use of EVMS performance metrics as award fee criteria or performance incentive should be prohibited. CODSIA warns that incentivizing contractors based on performance data could promote “poor behavior.”
Excerpt:
“In addition, inappropriate contractual incentives, such as focus on incentivizing or penalizing contractors based on performance data, promote poor behavior in the establishment of program baselines and EVMS implementations. An example would be the continuing use of incentives based on reported performance metrics, such as the cost performance index (CPI) and/or schedule performance index (SPI). Industry recommends a revision to the Defense FAR Supplement (DFARS) that prohibits the use of EVMS performance metrics as award fee criteria or performance incentives.”
Link to article, "Path": AT&L May 2011 path.pdf
Performance-Based Earned Value