|
|
| DoD provides guidance to integrate Earned Value with Technical Performance that is applicable to DoD organizations. The guidance overcomes the EVMS Quality Gap in the EVMS Standard. EVMS requires measurement of only the quantity of work performed, not the Quality. EVM Acquisition Reform addresses the need to revise federal acquisition policy and regulations to require that contractors link EV to technical performance. Technical Performance Measures (TPM) are described in the Defense Acquisition Program Support Methodology (DAPS), V2.0 and in the Defense Acquisition Guide (DAG), as follows: DAPS TPMs report the degree to which system requirements are met in terms of performance, cost, and schedule (3.3.4.3.c1) TPMs track the key indicators of system performance versus planned progress of Key Performance Parameters and other key effectiveness measures (5.1.1.c3)
In DAPS, TPMs flow down from the Capability Development Document (CDD) to the EVMS work packages, as follows: | Capabilities Development Document (CDD): Identifies Key Performance Parameters (KPPs) (1.3) System Specifications: A linkage exists between measures of effectiveness (MOE), system requirements (CDD and specifications), test and evalution (T&E) and critical technical parameters (CTP) (5.1.1.C2). | | Program Health Metrics: Identify and use TPMs that address Key Performance Parameters (KPP) and Critical Technical Parameters (CTP) (3.3.4.3.C2) | | IMP: IMP characteristic: linked to TPMs (3.31.C1b) | | IMS: • Maintains consistency with the IMP (3.31.C2b) • Maintains consistency with the work package definitions and the EVMS | | EVMS: • EVMS has no provision to measure quality (3.4.3.Q3) • Use TPMs to determine whether % completion metrics accurately reflect quantitative technical quality toward meeting KPPs and CTPs |
DAG 4.5.6.1 At the start of a program, TPMs define the planned progress of selected technical parameters. The plan is defined in terms of expected performance at specific points in the program as defined in the WBS and Integrated Master Schedule, the methods of measurement at those points, and the variation limits for corrective action. During the program, TPMs record the actual performance observed of the selected parameters and assist the manager in decision-making (performance or resource tradeoffs) through comparison of actual vs. projected performance. TPM parameters that especially need to be tracked are the cost drivers on the program, those that lie on the critical path, and those that represent high technical risk items. A well thought out program of TPMs provides an early warning of technical problems and supports assessments, and provides an assessment of the impacts of proposed changes in system performance. To accomplish this, the government and contractor both need to set parameters to be tracked. The government program office will need a set of TPMs that provide visibility into the technical performance of key elements of the work breakdown structure and the highest areas of cost, schedule, or technical risk. The TPMs selected for delivery to the government are expected to be traceable to the needs of the operational user and to the set of key performance parameters, key system attributes, and CTPs. The contractor will generally track more items than are reported to the government, as the contractor needs information at a more detailed level than does the government program office. TPM reporting to the government is a contractual issue, and those TPMs on which the government receives reports are defined as contract deliverables in the contract data requirements list.  Relationship of Major Technical Reviews to Technical Baselines to TPMs: Practical guidance and examples for linking EV to TPMs and to the technical baseline are provided in my book, Performance-Based Earned Value. Definitions of Measures of Effectiveness (MOE), Measures of Performance (MOP), and TPM per ANSI/EIA 632, Processes for Engineering a System Table C.5—Requirement 5 (Planning Process - Technical Effort Definition)
| Representative tasks
| Expected outcomes
| f) Identify technical performance measures | The following are determined (1) technical objectives related to success of the system, or portion thereof, [e.g., measures of effectiveness (MOEs) by which the user, customer, or acquirer will measure satisfaction or acceptance]; and (2) critical performance parameters that will receive management focus and are to be tracked using Technical performance Measurement (TPM) procedures. |
Table C.16—Requirement 16 (Requirements Definition Process - System Technical Requirements)
| Representative tasks
| Expected outcomes
| c) Define Performance Requirements
| The following are defined: (1) the performance expectations for each functional requirement (how well the function must be accomplished), (2) the set of measure of performance (MOPs), made up of the functional and performance requirement combinations, associated with each MOE, (3) the critical performance parameters (TPMs) selected from the MOPs that will be key indicators of end product or system performance, and if not met, that will cause the associated MOE to not be satisfied and will put the project in cost, schedule, or performance risk, and (4) functional and performance testability approach for each requirement statement. |
Definitions of Measures of Effectiveness (MOE), Measures of Performance (MOP), and TPM per IEEE 1220, Standard for Application and Management of the SE Process (ISO/IEC 26702) MOE: The metrics by which an acquirer will measure satisfaction with products produced by the technical effort. MOP: An engineering performance measure that provides design requirements that are necessary to satisfy an MOE. TPM: TPMs, when appropriately selected, are key to progressively assessing technical progress. Each critical technical parameter should be tracked relative to time, with dates established as to when progress will be checked and when full conformance will be met. Key technical parameters are measured relative to lower-level elements of the System Breakdown Structure by estimate, analysis, or test, and values are rolled up to the system level. TPM is also used to assess conformance to requirements.
It is recommended that the Annual Report of the Director, Operational Test and Evaluation (DOT&E) be reviewed to determine if contractors link earned value to TPMs: Suggested participants: GAO, DCMA, National Defense Industrial Association (NDIA) Sample programs to be reviewed to determine if earned value is based on technical performance. Technical performance was assessed and reported in the 2010 Annual Report of the Director, Operational Test & Evaluation (DOT&E), DoD | Service | Program | Milestone | OT&E Technical Assessment | TPM/EV linkage questions | Joint | F-35 | Block 0.5 software (SW) flight test (FT) completion | - 70% of planned FT points attempted
- SW problems occurring before and during flight test were not resolved
- Selected Block 0.5 FT points are reflying in the Block 1 configuration
- Mission systems SW development and test is tending towards familiar historical patterns of extended development, discovery in flight test, and deferrals to later increments.
| 1. Was Block 0.5 EV based on % of FT points completed by meeting requirements or on other TPM values achieved? 2. Was deferred functionality accounted for when points were deferred to Block 1? 3. Or was Block 0.5 EV subjectively reported as 95% or 100% complete after deferral of some of its functionality to Block 1? 4. One or 2 months prior to planned“ completion” of Block 0.5, did EV show a significant schedule variance and provide an early warning of the schedule slip? 5. Was budget transferred from Block 0.5 to Block 1 work packages to reflect the deferred FT points? If yes, was it transferred to the first month to preserve the schedule variance? | Joint | JTRS GMR | Joint Tactical Radio System (JTRS) Ground Mobile Radio (GMR) Systems Integration Test (SIT) completed 9/14/10. | - Low data throughput
- Low message completion rates
- Due to deficiencies in performance revealed during the SIT, the Army delayed the planned December 2010 LUT until June 2011 to allow time for reliability and performance improvements
| 1. Were milestones and TPM planned values established for data throughput and message completion rates? 2. Was EV in Sept. 2010 based on achieved vs. planned TPM values? 3. Or was SIT EV subjectively reported as 95% or 100% complete due to “completion” of SIT? 4. One or 2 months prior to planned“completion” of SIT, did EV show a significant and increasing schedule variance and provide an early warning of a schedule slip? 5. Did EV indicate a significant behind schedule condition that was consistent with a 6 month slip to LUT? 6. Was MR used to budget added tests and rework (described as added “scope”)? | Air Force | Global Hawk RQ-4 | 1. March 2010 Global Hawk Block 20/30 INTEGRATED Systems Evaluation (ISE) end-to-end system flight tests. 2. Block 30 ASIP developmental testing in April 2010. | 1. ISE identified significant Enhanced Integrated Sensor Suite (EISS) technical performance deficiencies (image quality, radar moving target detection, and system interoperability deficiencies). Aircraft all-weather capabilities, and system interoperability with supporting intelligence exploitation systems also experienced problems. 2.The ASIP sensor did not meet all system specification requirements. 3. The initial phase of Block 40 testing was delayed for 3 months due to competing Block 20 and Block 30 test program priorities. | 1. Were milestones and TPM planned values established for image quality, radar moving target detection, and system interoperability? 2. Was EV in March and April 2010 based on achieved vs. planned TPM values? 3. Or was EV subjectively reported as 95% or 100% complete due to “completion” of ISE and Block 30 ASIP testing? 4. Did EV indicate a significant behind schedule condition that was consistent with a delay of 3 months to the start of Block 40 testing? 5. One or 2 months prior to planned“completion” of ISE and SIP testing, did EV show a significant and increasing schedule variance and provide an early warning of a schedule slip? 6. Was MR used to budget added tests and rework (described as added “scope”)? | Navy | CVN 78 Nuclear Aircraft Carrier | F-35 integration milestones. | The thermal footprint from the main engine exhaust, shipboard noise levels, and information technology (IT) requirements need work. | 1. Were milestones and TPM planned values established for thermal footprint from the main engine exhaust, shipboard noise levels, and information technology requirements? 2. Is EV based on achieved vs. planned TPM values or percent of IT requirements met? | Links to Annual Report: Joint Programs: http://pogoarchives.org/m/ns/dote-fy2010-ar-dod-pp-13-44-1122011.pdf Air Force: http://pogoarchives.org/m/ns/dote-fy2010-ar-airforce-pp-179-224-1122011.pdf Navy: http://pogoarchives.org/m/ns/dote-fy2010-ar-navy-pp-97-178-1122011.pdf |
TPMs in DoD F-35 JSF Concurrency Quick Look Review (QLR): Key TPMs DoD F-35 JSF Concurrency Quick Look Review, Nov. 29, 2011: Technical Performance Measures (TPM) | Product | TPM Description | TPM Objective | Recommended evidence of linkage of Technical Performance to IMS and EVM (this issue is not part of the QLR) | Helmet Mounted Display EO-DAS | 1. Video image latency lag behind sightline movement 2. Basic symbology latency | 40 msec 30 msec | · IMS milestone with TPM completion criteria · TPM planned values at interim milestones · EV based on achieved performance | Integrated Power Package | Mean flight hours between maintenance events | Not in report | Same | Airframe: - F-35A Wing Forward Root Rib
- F-35B FS496 bulkhead
| Service or fatigue life | 8,000 hrs. | Same | CTOL | Weight | 29,731 lb. | Same | Panoramic Cockpit Display Electronics Unit | - PTMS Cooling Airflow/Temperature at PCD/EU
- Ambient temperature around PCD/EU
| Not in report | Same | Software (SW) Block 1B | SW functionality (requirements, shalls) | Not in report | 1. IMS milestone and work packages with baselined completion criteria traceable to the systems engineering requirements traceability matrix 2. Does earned value account for deferred functionality from Block 1B to subsequent blocks? | Link to DoD Report: DoD F-35 Concurrency Quick Look Review |
The PMI College of Performance Management is emphasizing Technical Performance Measures (TPM) during its EVM World 2012 Conference in Naples, FL starting May 30. PMI is looking for speakers to share lessons learned and best practices in its Practice Symposia. This year, preference will be given to papers that address the tools and techniques used to identify and implement TPMs that are integrated with EVM or other program performance measures. Examples of integrating TPMs with EVM are provided in a 2006 article, “Practical, Performance-Based Earned Value.” DoD CrossTalk article . Please share your recent best practices at EVM World and submit an abstract by Jan. 25 via http://evmworld.org/conference-overview/ . | |
|