ISBN: 0-671-21426-8

My rating: 65/100

See Book Notes for other books I have read. If you like my notes, go buy it!

By Jay Alan Moody, William L. Chapman, F. David Van Voorhees, and A. Terry Bahill

Tagline: The title is it’s own tagline.

My first comment on this book is this: it completely gave up trying to have any structure. There are case studies, there are metrics, there is a new system for scoring engineering projects, the scoring system is used to evaluate the projects using the metrics, there is a definition of “systems engineering”, but the whole thing is very disconnected and there is literally no conclusion – it just ends. That being said, if you go in expecting no structure you should be fine. When I bought this book was expecting to get a list of useful metrics for evaluating the engineering projects that I’m working on – things like cost analyses, schedule evaluation metrics, performance metrics like speed, user satisfaction, and power consumption. Instead, the metrics they presented were used to classify engineering projects rather than the results of those projects: the product itself.

The key, novel takeaway from this book is the author’s system for classifying engineering projects by difficulty and resources and their relationship with formal systems engineering controls. The authors state that, in the past, large projects have typically been the only ones that end up having fancy systems engineering controls like highly regulated test procedures, systems integration protocols, etc. but they argue for a new method of deciding if your project requires higher levels of system engineering controls that is based on risk. If you plot these two  metrics, resources and difficulty, you end up with this graph where the areas of high risk are in the lower right and the upper left.

A few things to note about this graph.

• The High Tech area is where risks arise due to lack of resources compared with the difficult of the design. In this area you can find things like the invention of the light bulb – Edison and his little team tried almost a hundred ideas with very little investment. Projects in this area need to have technical performance measures in place to reduce the risks.
• The High Political Risk area is where a lot of resources are needed, but funding and motivation behind the project needs to be secure otherwise money might get cut when things aren’t going well. These projects need to have a strong funding base because they have little to no ROI and must have sufficient public support.
• The vertical line in the middle is traditionally where formal systems engineering tools have been used in order to manage the complexity of the project. The authors believe this is a mistake. They think formal systems engineering tools should be used in the areas highlighted as high risk. This is perhaps the most important concept in the whole book. 

Other key takeaways:

• Have a planned system for recording and incorporating future design improvements. Trying to include every little design change during every stage of development causes unnecessary delays.
• Successful projects always have clearly defined criteria for success. The engineers knew where they were going and were able to judge their success themselves. It is the systems engineer’s job to set up the criteria for success on all subsystems of a large project. 
• Full system tests should be a requirement for any technology project – this is the reason the Hubble Space Telescope error was missed. As a piggyback on this, undesirable test results should not be dismissed arbitrarily in favor of expected test results.
• Poor requirements development, inadequate configuration management, and weak program and technical  management appear to be the primary determinants of lower systems engineering fundamentals scores.

Chapter 1: Design Difficulty and Resource Metrics

The authors provide the metrics they used for their system, and how to score each.

Design Difficult Metrics:

• Design Type
• Knowledge Complexity
• Steps
• Quality
• Process Design
• Aggressive Selling Price

Resource Metrics:

• Cost
• Time
• Infrastructure

Chapter 2: Design of Resistor Networks

As the complexity of a system increases, the chances of obtaining a feasible solution increases. This is why initial designs of most products are so complex.

As the complexity of the system increases so does the feasible solution space. In this project, using more resistors increases the likelihood you can find a solution, but the complexity increases.

Chapter 4: Bat Chooser

Market pull usually produces success. Bat chooser was a technical success, but a market failure.

Before designing a new product, you should make sure that someone wants to buy it and has the money to do so.

Chapter 5: Scheduling a Pinewood Derby

Brainstorming to achieve a breakthrough which produces a perfect schedule will require lots of time with possibly no deliverable product. The best approach is to obtain a deliverable first, then iterate the design to get a better solution. If something is infeasible by the statement of the problem, then a solution can only be found by changing the problem statement, not by investigating many possible solutions.

The major Systems Engineering mistake that we made on this project was a failure to validate the requirements. That is, we did not prove that it was possible to satisfy the system requirements before we started designing. We were trying to design an impossible system.

Chapter 6: Second Opinion

Design is iterative. Validating the system to the customer’s requirements is essential to ensure product satisfaction.

However, most companies can never afford to develop five versions of the product as was done here. The average is four. The first is often called the alpha version (or the model), the second is the beta version (or the prototype), the third is the preproduction unit, and the last is the production version. Some software companies skip the preproduction model.

Chapter 8: Superconductors

Breakthrough designs are extremely rare because few people have the freedom to investigate whatever they want.

Chapter 9: The Incandescent Light Bulb

Breakthroughs appear, not when they are well funded and reasoned, but when a group of talented and dedicated people are determined to solve a problem. The implication for the system design process is that the only way to achieve a massive improvement in technology is to motivate talented people, give them some resources (although not everything they want), and then leave them alone. There were no managers, no finance people, an no marketing people: only engineers and scientists with clearly defined goals for success and a try-anything atmosphere.

Chapter 11: The Apollo Moon Landing

When feasibility cannot be achieved, an approach often taken is to increase complexity until a solution is found.

Chapter 12: Building a House

The most common mistake made by amateurs designing homes is failing to freeze the system requirements before manufacturing is begun. Changing the requirements during manufacture is costly.

Chapter 13: Central Arizona Project

The CAP planners overestimated the demand for the water and underestimated the cost of the project. In Systems Engineering terminology, they did not mitigate the cost risks.

Chapter 14: An Automobile Factory

By performing their work in parallel, with hundreds of little suppliers, the actual Japanese design time is reduced. This technique requires very strict interface specifications so that integration will go smoothly.

The change rate is high early in the design process for a Japanese design team. Keeping more options open longer also permits a better chance of finding superior solutions (or alternative minima) rather than locking in the design early in the system design process.

#keepoptionsopen

Chapter 20: The Manhattan Project

Because of the vast uncertainties and short schedule, the typical strategy was to explore all possibilities simultaneously. This exploration necessarily took the form of a combinatorial search. In one case, all elements of the periodic table were tested for properties as a tamper (reflector) for the bomb. In the end, non of the alternatives were adequate.

Only the CEO and hand-picked lieutenants were allowed to know anything about the project, but a surprising number of companies involved themselves anyway, including Du Pont (who would only accept a cost plus $1 contract).

Chapter 21: The Polaris Program

The program came within 2% of original cost estimates. (!)

A policy of decentralization and competition within a strict framework.

Research, development, test, and evaluation consumed only 24% of the funding.

A complete set of subsystem interfaces was defined very early in the program.

Judicious use of independent design agents.

The focus was on systems development, not technology.

There was a planned series of improvements to the original design. Any further progress would be incorporated in future versions. 

Chapter 22: The Hubble Space Telescope

The primary mirror was ground with a precision better than 150 angstroms. But it was ground in the wrong shape!

During the manufacturing process, there were many clues that something was wrong. A photograph taken in 1981 shows the flaw, but the test procedures directed the tester’s attention to a different part of the photograph.

This failure could have been detected if a full system test had been performed, but such a test was not required. NASA specified test to detect small errors, but not blunders of this magnitude. They thought a complete system test was unnecessary and would cost too much. However, the cost of a complete system test would have been much less than the $850 million it cost to fix the telescope in orbit.

The HST is the most complex scientific instrument ever constructed. With 10 times finer resolution and 50 times greater sensitivity than any other optical mechanism every constructed. If the mirror were scaled to the size of North America, the largest hill or valley would be like a small ant hill, while a typical eyeglass lens would have skyscraper-sized imperfections.

The HST procurement cost was $1.5B, which was three times the amount originally budgeted.

The HST resolution is like being able to read a car license plate at 300 miles away.

My note: plan maintenance by hardware requirements first, keeping resources in mind but second.

The undesirable test results should not be dismissed arbitrarily in favor of expected test results.

The primary mirror was ground in the wrong shape because this reflective null corrector was incorrectly assembled.

A key consideration in the decision not to conduct a total system test, also known as an end-to-end test, was the cost of the test. In the early days after the discovery of the spherical aberration, NASA estimated this testing cost as several hundred million dollars. When this figure was questioned by the technical community, NASA significantly reduced its estimate by an order of magnitude. In fact, the Air Force had the test equipment on hand. Worse, a simple “knife-edge” test commonly used by amateur astronomers could have detected the error for less than $100.

One of the greatest tragedies is that NASA had already recognized that the manufacture of the primary mirror was a critical and inherently risky task and, following good risk management practices, had Corning manufacture a backup mirror. This backup mirror was built and delivered, but was never tested. The namufacturing method of this backup primary mirror was completely different from that of the Perkin-Elmer mirror, and there is every reason to believe that the backup was within specification. Also notable is the fact that the backup mirror came in on schedule and within budget. However, the whole truth may never be known, as the backup mirror was apparently requisitioned for use as a U.S. national security asset.

Chapter 23: Lessons Learned from the Simple Case Studies

If your customer asks you to build a system that fits into the Star Wars quadrant (low resources, high design difficulty), you should tell them at the beginning that it cannot be done with the allocated resources.

So it seems that in the past, the decision of whether or not to do formal systems engineering has been based on resources. However, we think the decision should be based on risk.

If your project is in the area of high political risk, you must identify its source of support, and work very hard so that you do not lose it.

First, successful projects always have clearly defined criteria for success. The engineers knew where they were going and were able to judge their success themselves. It is the systems engineer’s job to set up the criteria for success on all subsystems of a large project. 

Each law passed by any political body should define its criteria for success. For example, a crime bill might state that it will reduce crime by xx% as measured with yy statistic within zz years. If it fails, the law would automatically be rescinded.

Critical path. The systems engineer should make sure that the invention (or basic research) is not being done on the critical path.

Attributes of a good metric. A good metric should measure aspects of the process than can be controlled; provide information that can be used to initiate change; provide insight into how well goals and objectives are being met; be simple, understandable, repeatable, and measurable; and have inexpensive methods to collect and analyze data.

The system design process is NP-complete which is just a fancy way of saying that it is mathematically hard problem with no known solution beyond brute force calculation.

Solving system design problems:

1. Additive approaches – point insertion, incremental improvement, etc.
2. Tree search – branch and bound.
3. Relaxation of Requirements
4. Local Optimization Techniques – 3-Opt and subproblems
5. Probabilistic Jumps – simulated annealing

The solution technique for the Pyramids and the Central Arizona Project relied on a tree search approach to explore the options, combined with an additive approach to handle additional requirements. The initial plans were laid out, the options considered and a path chosen. When problems arose, they were added to the original plans, and the construction continued.

In tree searches many options are kept open.

Solution techniques for NP-complete problems:

1. Use an additive approach for small design problems.
2. If many options can be examined, use a tree search.
3. All large problems must be broken into subproblems.
4. When no feasible solution seems possible, relax the requirements to achieve feasibility, then iterate towards the original requirements.
5. If a breakthrough is needed, then wide searches using many unlikely options must be tried. Expect many failures.

Chapter 24: What is Systems Engineering? A Consensus of Senior Systems Engineers

My note: I know the authors state this, but compare this with the product development process and they are basically identical. Why does this chapter exist? Its too high level to be useful, yet too detailed to be considered an overview.

Systems Engineering is an interdisciplinary process that ensures that the customer’s needs are satisfied throughout a system’s entire lifecycle. This process includes:

• understanding customer needs
• stating the problem
• discovering system requirements
• defining quantitative measure
• validating requirements
• prescribing tests
• exploring alternative concepts
• conducting sensitivity analyses
• performing functional decomposition
• system modeling
• creating a system design
• designing and managing interfaces
• conducting design reviews
• integrating the system
•  performing a total system test
• maintaining configuration management
• implementing risk management
• conducting reliability analyses
• performing total quality management
• providing project management
• documenting all activity

The Process

The system life cycle has seven general phases:

1. discovering system requirements
2. exploration of concepts
3. full-scale engineering design
4. manufacturing
5. system integration and testing
6. operation and maintenance
7. retirement, disposal, and replacement

The words optimize, maximize, and minimize should not be used in stating requirements.

Chapter 25: Metrics for Systems Engineering and the Developmental Environment

My note: In this chapter the authors explain the metrics that they used to rate each of the following case-studies.

An evaluation methodology is developed through the creation of figures of merit and a rating scale for each of the five characteristics of system development effort.

1. Performance. Involving technical, cost, and schedule performance.
2. Systems Engineering Fundamentals. Describing 11 crucial principles and practices that should be considered for application in any development.
3. Development Environment. 11 Aspects are suggest to be conducive to supporting successful system development.
4. Design Difficulty. Consisting of six figures of merit addressing various aspects of technical complexity.
5. Resources. In terms of cost, time, and infrastructure to complete development.

Chapter 26: The Boeing 777 Commercial Airplane

Airplane cost $125M

$4-5B startup costs.

[Traditionally,] design engineers designed the air craft separately and then gave the drawings and specifications to manufacturing for fabrication and to maintenance personnel for establishing maintenance procedures. This approach normally led to significant changes as the manufacturing engineers discovered design inconsistencies and unproducible component configurations. It also resulted in higher life-cycle costs, since the designs did not always emphasize ease of repair and a reduced need for maintenance. With the 777, Boeing decided on a radically new development approach that focused on the customer and placed great emphasis on ensuring that the development activities were done right the first time.

The first 777 produced and flown was a production configuration aircraft instead of an engineering prototype.

My question: When to build multiple prototypes vs. when to spend more time in design and build at or near production level?

Boeing has implemented a paperless manufacturing floor.

Boeing took the aircraft through a break-in period to identify and correct annoying problems, instead of having the airlines experience them. This “service-ready” approach was the guiding philosophy of the development program.

Chapter 27: Lockheed F-117 Stealth Fighter

The total cost was about $6.27B, $2B for development and $4.27B for producing 59 aircraft.

Lockheed’s approach to moving stealth from concept to reality was to use off-the-shelf hardware when possible, modify existing equipment where feasible, and invent new systems only when required.

According to Lockheed officials, one of the most difficult issues to solve involved the four small pitot tubes that extended from the aircraft nose to gather air data. the presence of these four small tubes had a significant impact on the size of the radar signature. It took engineers three years to come up with a design solution to preserve the F-117’s low observability with the pitot tubes.

The first production unit crashed because the airplane’s roll-rate and pitch rate gyroscopes had been crossed when they were installed.

My note: This exemplifies the importance of having test procedures at various levels in the assembly to make sure things are installed and operating correctly.

The golden rule: “He who has the gold, rules.”

The Skunk Works approach is encapsulated in 14 points or “rules” developed by its founder, Clarence “Kelly” Johnson.

1. The program manager must have complete control of the program in all aspects. It is essential that the program manager have authority to make decisions quickly regarding technical, financial, schedule, or operational matters.
2. Strong but small project offices must be provided both by the customer and the contractor. The customer program manager must have similar authority to that of the contractor.
3. The number of people having any connection with the project must be severely restricted, because more people add bureaucracy, and bureaucracy makes unnecessary work.
4. A simple drawing and drawing release system with great flexibility for making changes must be provided. This permits early work by manufacturing organizations, and schedule recovery if technical risks involve failures.
5. The number of require reports should be minimized, but important work must be recorded thoroughly. Responsible management does not require massive technical and information systems.
6. There must be a monthly cost review covering not only what has been spent and committed, but also projected costs to the conclusion of the program. Responsible management operates within the resources available.
7. The contractor must be delegated and must assume more than normal responsibility for obtaining good vendor bids for the subcontracts on the project. Commercial bid procedures are often better than military ones. The contractor must have the essential freedom to use the best talent available, but also operate within resources provided.
8. Responsibility for basic inspection should be given to subcontractors and vendors. The contractor should not be duplicating inspection.
9. The contractor must be assigned the authority to test the final product in flight, especially in the initial stages of development. This is critical if new technology and the attendant risks are to be rationally managed.
10. The specification applying to the hardware must be defined and finalized in advance of contracting . Furthermore, the Skunk Works’ practice of having a specification section clearly stating which military specification items will not be followed and the reasons for not doing so is highly recommended. Standard specifications inhibit new technology and innovation and are frequently obsolete.
11. Funding must be timely.
12. Mutual trust must exist between the customer project organization and the contractor, and there should be close cooperation and day-to-day communication. This brings misunderstanding and correspondence to a minimum. The goals of the customer and producer should be the same – get the job done well.
13. Access by outsiders to the project and its personnel must be strictly controlled by appropriate security measures.
14. Due to the small size of Skunk Works development teams, management must provide ways to financially reward people based on good performance and not on the number of people supervised. Responsible management must be rewarded, and responsible management does not promote growth of bureaucracies.

Chapter 28 Northrop B-2 Stealth Bomber

The unit cost was expected to be about $330M in base year 1981 dollars.

This system specification focused on performance and stayed away from functional requirements, giving the contractors greater flexibility in defining a solution. A consequence was a smaller number of engineering change proposals during development than normal.

Using computer models, Northrop assessed a variety of design approaches by running a series of iterations and by evaluating the performance and cost trade-offs.

The B-2’s extensive four year development flight test program involved six aircraft, five of which were planned for eventual delivery as operational units.

low observables – stealth technologies

cost-plus

The contractors looked to the Air Force primarily for performance requirements that would not change. Early in the B-2 program, however, there was a major change with significant consequences. As previously mentioned, soon after source selection, the Air Force added the requirement that the B-2 must have all-altitude capability as Northrup proposed. Then in 1983 during pre-FSED, extensive structural analysis by contractor and Air Force engineers indicated loads were significantly higher than originally believed. This initiated a redesign to the trailing edge. … This resulted in moving the cockpit 30 inches, which had significant cost and schedule impacts.

Air Force program directors were looking to change to an integrated product development (IPD) approach; this would formally establish interdisciplinary integrated product teams and force business and technical issues to be worked together. IPD was implemented in 1991, when the majority of development was complete.

Chapter 29: McDonnell Douglas C-17 Military Transport

The fixed-price development effort has an overrun of $571M on $2.5B stated in base year 1981 dollars. The unit cost of the C-17 was estimated to be $219M.

The components and assemblies of the C-17 were not designed with ease of manufacturability as a primary driver. Forty percent of the labor hours on the first two C-17s went to repair, rework, and nonstandard work. 

Douglas had problems keeping a core C-17 development team together even before political turmoil. The four year delay of FSED in the early 1980s prompted some of the key C-17 technical and management personnel involved with the earlier work to find jobs in other McDonnell Douglas programs that needed expertise.

Chapter 30: Learjet Model 60 Business Jet

Development cost about $90M. The unit production sales price was approximately $9.5M.

In the development of the Model 60, the Learjet team followed many of the fundamental systems engineering principles and techniques. However, much of the process was conducted with minimal formality. Given the relatively low complexity of this redesign effort, such an informal approach was adequate and appropriate. Some formality in the development process was evident in certain controlled documents. Specifically, the design requirements were documented in a system specification by the engineering department after receiving the top-level customer desires from the marketing department. The other formal documents consisted primarily of a test master plan, test plans and procedures, a configuration management plan, and a quality plan. Interface control documents were also maintained and controlled. However, no systems engineering management plan or equivalent was developed by Learjet, and neither were subsystem specifications. Subsystem specifications probably existed with the subcontractors, who had developed their items independent of Learjet, but they were not generated as part of the Model 60 development.

My note: When do you know when to use formal processes vs informal ones?

The team never performed a formal trade-off analysis of options. Furthermore, a risk analysis was never performed.

The program never performed a validation audit of its requirements.

Chapter 31: McDonnell Douglas MD-11 Commercial Airplane

$722M development effort. 1994 selling price is about $116M

The MD-11 has a total of 434,970 parts, not including nuts and bolts.

Concerning detail design issues, the development team made assumptions as to what the airlines would want. [this was a mistake]

My note: Add a document or database or form to capture design changes between product development phases.

Most of the MD-11’s design details were left to the Douglas engineers to determine without customer review or comment.

Douglas’s configuration changes control activities were not structured for quick and efficient changes.

At the beginning of the MD-11 effort, it had not developed a new aircraft in years. Therefore, it did not possess a large pool of design engineering and development expertise. As a consequence, a large number of engineers were hired to conduct the program. Although the engineering design team was technically capable, they of course lacked experience of working together.

32 Comparison of the Aircraft Case Studies

The picture that emerges most prominently from the aircraft case studies is that poor requirements development, inadequate configuration management, and weak program and technical  management appear to be the primary determinants of lower systems engineering fundamentals scores.

Since all three military efforts had marginal cost and schedule performance, this suggests that the development process for government aircraft is inherently more prone to cost and schedule difficulties.

 

MD-11: $722M development / $116M sale price = 16% of dev costs

Learjet: $90M / $9.5M = 10.5%

C-17: $4071M / $219M = 5.4%

F-117: $2000M / $72M = 3.6%

777: $4-5000M / $125M = 3.1-2.5%