1
|
- Chris Miller (Software Productivity Consortium)
- Roger Shepherd (Raytheon)
- John Evers (Raytheon)
|
2
|
- “The team discussed the approach to developing the SE metrics, and
discussed various factors associated with developing metrics. John reviewed the identified
requirements on metrics (from the SSS), and it was agreed that a
disciplined approach needs to be used.
Chris and John to get together and define a common approach to
use.
- It was noted that addressing metrics takes time to do it well. This raised a concern for the Process
Owners, in meeting their various commitments. Given expected transitions of
personnel, it was decided that the team needs to revisit the SE PI
Project Organization Chart, the Roles and Responsibilities, and the SEC
membership next week.”
- - excerpt from Minutes, 1/31/2001 SEC Stand-up Meeting
|
3
|
- Process Improvement goals drove measurement initiative
- Improve how Systems Engineering is done (better products with best
practices)
- Increase win rates (better proposals, better predictions)
- Increase productivity (better execution, less rework)
- Institutionalize EIA/IS-731 “Level 3”
- EIA/IS-731 “Level 3” (follow SW CMM Level 3 effort)
- Integrating toward CMMI Level 3 (and beyond)
- Leverage existing measurement reports and methods
- Earned value procedures and concepts
- Functional managers as advocates & change agents
- Institutional goals vs. project-specific measurement
- SE and SW measurement approaches coordinated, but not unified
|
4
|
- Measurement History
- Previous measurement efforts never employed
- Existing measurement procedures
- Primarily CPI, SPI for Senior Engineering Management Reviews
- Site management wanted
- Productivity measure
- Better bids
- Multiple measurement users (matrix organization)
- Varying information needs
- Process and Tools Manager, Program Director, Program/Project Manager,
Program Engineer, Cost Account Manager, Engineering Manager, Discipline
Lead
|
5
|
- Adoption of PSM and ISO 15939 Information Model
- PSM training provided a common vocabulary
- Consortium lead PSM training
- Overview training followed by Just-In-Time (JIT)
- Small group targeted to be measurement experts
- Facilitate and mentor projects over time
- Share measurement technology between Systems Engineering and Software
process initiatives
|
6
|
- 3/27/2001 – PSM tutorial
- 4/02/2001 – First Measurement Offsite
- Used PSM’s Issue-Category-Question Table to identify specific issues
for each of the information users
- Identified 66 candidate measures
- 4/9/2001 – Prioritized Measures
- Selected 16 candidate measures to define
- Selected criteria:
- Number of Information users
- Implementation Scope (new, rework, reuse)
- Effort to Implement
- Added Award Fee & Win Rate
- Developed measurement definition template based on ISO 15939’s
Information Model
|
7
|
- Conducted measurement writing sessions between 4/9/2001 and 7/2/2001
- EIA/IS 731 Evaluation during 3rd week of June
- 7/2/2001 – Measurement Offsite
- Unified measurement definitions
- Linked measures to affected engineering processes
- By 9/5/2001 MS Access database was created and contained measurement
definitions, consisting of:
- 33 Base Measures
- 20 Derived Measures
- 15 Indicators
- 18 Data Sources (Entities)
|
8
|
- No formal measurement training
- SE measurement activity was
communicated to project engineers
- Rollout
- Began data collection with piloting on projects, then broader roll-out.
- Used Incremental rollout for organizational reports
- Updated charts periodically by phasing in measures over time
- Implementation
- Emphasis on detail and collection of baseline data on work product
performance, characterization, and estimation
- Created duplicate collection paths to provide ease collection
- De-emphasized changes to project reports as rolled up to senior
management levels
- Validation
- Defined scoring (compliance) mechanisms to promote the adoption of the
practices that were to be measured
|
9
|
- What worked well
- Involvement of SE Discipline Leads (practitioner leads), addressing
various users of information;
- Involving functional leads and engineering managers in defining &
prioritizing the measures
- Pressed measurement issues gently, but consistently
- Leveraging existing data (SEMR charts, existing cost collection
systems, etc.)
- What didn’t work well
- Inconsistent WBS / cost collection structure across programs; driven by
types of business
- Made the measures TOO transparent to users
- Many aren’t aware of what’s collected
- Collection of HR data not progressing
|
10
|
- Currently the MS Access database consists of:
- 40 Base Measures, 27 Derived Measures, 14 Indicators
- Approximately 17 Full Time Equivalents (FTEs) over 15 months
- Measurement team consisted of
- 1 Measurement consult 20% for 6 months
- 3 Measurement experts 30% for 15 months
- 2 Measurement authors 10% for 2 months
- 1 SE PI Lead 10% for 15 months
- 5 Management 5% for 10 months
|
11
|
- Closing the Bid Loop
- Strong focus on collecting & analyzing data which feeds our bid
model
- Completing R6s project, and
pushing a broader activity and awareness of this issue (e.g., standard
bid code project )
- Emphasizing Simplicity
- Up-Tempo Use for Projects
- Some projects are adopting compatible measures to serve finer-grained
project needs.
- Supporting TPM Emphasis
- Providing measurement guidance to programs
- Measurement development method to develop better measures
- Captured approach details for development of standard workshop / guide
for use by other programs
- Tools
- MS Access & Excel - looking at more sophisticated tools for future
|
12
|
- Project level versus organization
- Focused initially on organizational issues, then teamed with projects
during pilot and rollout
- PSM was a stable technology that met many needs
- We checked other measurement structures to verify we covered all the
bases (the cube model; predictive / reactive; cost, quality, cost,
performance; process, product, program)
|
13
|
- Takes time to do it right
- Takes concentrated effort
- Use an agreed-on measurement process and measurement definition
structure
- PSM helped to validate our efforts
- Keep end users in mind
- Satisfy information needs
- KISS
|
14
|
- Learning the Lessons
- Move towards CMMI - integrated set of organizational measures based on
PSM structure
- Emphasize USE of the information (vs. data collection details) - keep it
beneficial to end users
- Seeking to refine / simplify collection
|