Exhibit A-2. A retyped paper presented at the "Advanced Planning Briefing for Industry", Columbus, Ohio: 25 September 1969.

NAVAL AIRCRAFT SOURCE SELECTION

G. A. SPANGENBERG

Director, Evaluation Division, NAVAIR

Introduction

My task today is to tell you something of the system and procedures we will use in determining which of our major contractors will be selected to develop the new weapon systems described by earlier speakers. As some of you know, and as the rest of you will soon discover, I'm a very biased observer on the subject of source selection. I firmly believe that our Navy aircraft selection system has been, and still is, better than that being employed by others. If I didn't think so, I wouldn't be here today, nor would I have remained in the evaluation and selection business for the Navy.

I normally start off these discussions by apologizing to those whose feelings are going to be hurt. These are usually my friends in OSD, the Army, and Air Force, who keep developing new ways to make even the simple tasks more difficult. Our own Navy management friends often collaborate too willingly with such efforts.

This then leads to a brief discussion of the fact the world is made up of the "WEs" and the "THEYs". The "WEs" are always the good guys while the "THEYs" are always the bad ones. "WE" develop the good, the simple, and the inexpensive, while "THEY" force us into unsatisfactory solutions, usually with a great proliferation of paper. In my business, the "WEs" start with the engineers in the Naval Air Systems Command who are always sound, reasonable beings (who agree with me), while the "THEYs" include all those who don't -- particularly the system analysts across the river.

To further set the stage, I'd like to quote three paragraphs from a speech made by Mr. Lee Atwood of North American about 15 years ago, after North American had finished second in a competition in which there was but one winner. The principles enunciated were, in fact, those under which we had been operating.

"Although the Government owes the nation an adequate air defense, and such an air defense will assure a measure of continuity in our development and production efforts, I want to emphasize that the Government does not owe the aircraft companies a living. Quite to the contrary it is essential to the success of our development program that the Government award contracts as objectively as possible on the basis of genuine merit. The personnel responsible for procurement must put aside any ideas they may have been encouraged to develop in the past 20 years that they are social planners, economic stabilizers, or anything else but hard-boiled customers with a job to do and a determination to get the best equipment to do the job.

I am specifically asking for maximum realism in procurement competitions. Our efforts in industry will be most effective if the competition is tough, fair, and clearly oriented toward product quality. There is a disproportionate premium attached to winning a design competition. It is the ticket of admission to the production show, but after all a design is just a list of promises based on calculations, which in turn are predicated on assumptions that can vary with the optimism of the designer. Rarely if ever have there been any real penalties when the glowing forecasts of the design proposal were adjusted downward to the physical facts of the actual airplane. And it is then too late to change. I believe that this is a serious problem that deserves increased study by all responsible officers of both the Air Force and the Navy. If the imposition of financial penalties for non-attainment of performance guarantees is the only workable answer, then I believe such penalties should be invoked.

I believe that most of the responsible airframe, engine, and other contractors would agree wholeheartedly with the principle of awarding contracts objectively on the basis of design realism, economy, efficiency, and whatever other fair and impartial measures of comparison the procurement people can devise. There is really no other effective way to maintain the integrity and effectiveness of American aeronautical development."

As you all know, the stakes are even larger today, making it more necessary that we be not only a tough customer, but a sophisticated one, capable of clearly defining our needs and our mutual obligations.

Objectives

Before getting started on a description of our methodology, let's review our overall objectives. We seek the best product at a fair price. We must accomplish the job in a responsible manner within our own resources, while developing adequate data to justify the course of action we finally adopt. The task must be accomplished with a maximum of competition but with a minimum of expense to both government and industry. In particular there is certainly no single best solution to the overall source selection problem. Flexibility to tailor the methods to suit the individual procurement is essential.

Basic Needs

The basic steps in the procurement process are:

1. Establish the Requirement
2. Define the Program
3. Obtain Program Approval
4. Select a Contractor
5. Contract

While the selection process itself is the subject for this talk, it should be obvious that the success of any project depends upon how well each step is accomplished. A sound requirement is undoubtedly the most important single factor involved. The best of procurement plans cannot compensate for a poorly conceived project with impossible technical goals. The majority of failures in our business can be traced to the overambitious requirement generator, whose motto often has been, "Let's see what industry can do."

Procurement Methods

Now let's define in very broad terms what we are talking about. First, all of our aircraft procurements are made under the "Negotiated" procurement rules of the ASPR, and not under the rules of formal advertising. Unfortunately, this fact is occasionally used as an argument that there is no competition in the aerospace business. In actuality, of course, virtually every new procurement is competitive, with the degree varying from mild to cut-throat.

Within the "Negotiated" arena, we can split the contractor selection methods into a number of categories:

1. Sole source procurements are the easiest, cheapest, and fastest, but the most difficult to justify. They are used today primarily when we buy a modification of an existing design, e.g., an EA-6B, derived from the basic A-6 series.

2. "Source Selection" (in quotations) was initiated by the Air Force in the late 50s. In the beginning, it emphasized management proposals, with an aim of selecting the best source, rather than design. The method was highly formalized and was the basis of DOD Instruction 4105.52 covering Source Selection.

3. Design competitions had been employed by the Navy as the prime selection method for aircraft for over 30 years. The method, modified only as necessary to comply with the DOD directive, allows qualified bidders to demonstrate their overall competence by submitting design proposals. Selection is based primarily on the merit of the design.

Design Proposal

A modern design proposal is a comprehensive set of documentation describing not only the design itself, but including enough cost, management, and support information that we can draw up a definitive contract. The A-7 proposal in 1963 consisted of a 16 inch stack of technical reports, 5 inches of management reports, and 9 inches of drawings. These data were adequate for selecting the contractor and negotiating a fixed-price contract for initial development with fixed-price ceiling options for about 200 production airplanes.

Proposal Shelf Length

The amount of data being submitted has expanded significantly since the A-7. Our last two competitions resulted in average "shelf lengths" of about 7 and 12 feet for a single copy of each proposal. With multiple copies required, the data handling task is obviously formidable for both the contractors and the government. Pressure from higher authority to define all aspects of a procurement for years in advance has caused much of the increase. It is anticipated that some of the paper proliferation studies now underway will provide some relief in this area. [Later comment -- They didn't.]

As a matter of interest, there is little correlation between shelf length and success in our competitions. As most of you know, it is much harder to write succinctly than to ramble on at length. The more condensed the proposal the more favorably it is received. We get really annoyed when individual proposal reports are too large to fit neatly into standard file drawers.

Evaluation - General

With our proposals in hand, let's now look at what we do with them.

The evaluation job is done primarily by the NAVAIR functional organization with some assistance from our field stations when necessary. We do not use a large ad hoc type of organization as do some of the other services. Our evaluators are the same engineers who helped prepare the RFP, and who will later monitor the development of the aircraft. Since they wrote the detail requirements, they need no separate "criteria" against which to evaluate.

Simple comparison and elimination techniques are used as a basic methodology without resort to elaborate point rating systems. We do quantitative analyses of those characteristics such as weight and performance where numerical results have meaning. Qualitative ratings of design features, installations, and components are provided by the evaluators and left in that form. No attempt is made to transform essentially qualitative judgements into a quantitative score. Experience, and common sense, prove that the step is unnecessary. The reasons for relative ratings are necessary to convince those who must review decisions, while numbers in themselves are meaningless. A summation of all results allows elimination of the least promising proposals.

Evaluation Detail

For a little more detail on our methods:

1. Independent weight estimates are made by our weight control engineers.
2. Simultaneously, estimates are made on major performance items by our performance specialists.
3. All divisions review the proposals, rate each one in categories of acceptability, and rank each one relative to the others.
4. Our cost analysts derive comparable prices from the contractors' quotes.
5. The least promising designs are eliminated from further consideration
6. On an individual basis, changes are made in each design, to obtain the configuration which we would put under contract if that contractor were to be selected. Weight, performance, and cost figures are recalculated.
7. Detail comparisons between the designs remaining under consideration are made and results summarized for a decision.
8. After the decision, each contractor is debriefed on an individual basis as to the results of the entire evaluation of his design. This enables the contractor to do a better job on his next proposal. It also provides an incentive for each evaluator to do a good job since he must, in effect, justify his estimates or comments on the particular design.

Presentation Scope

When an evaluation is completed, the results must be presented to a series of decision makers. This is normally done in an oral presentation, scheduled for perhaps two hours using 60-70 charts or Vu-Graphs, which includes data on the following:

1. History of project, the evaluation methodology, personnel involved, and any significant problems encountered.
2. A detail description of each design, its features and unique solutions to design problems.
3. NAVAIR weight, performance, and flying quality estimates.
4. Ratings and rankings by each of our design divisions, Propulsion, Armament, Structures, Maintenance, etc.
5. Cost summaries, contractor quotes and NAVAIR estimates.
6. Procurement plan, contract details, and delivery schedules.
7. Pertinent management issues, and CPE results.
8. Proposal summaries.
9. Conclusions.
10. Recommendations

As the presentation moves up the decision chain it is condensed and tailored to suit the audience. Detail is eliminated and greater emphasis placed on the program and fiscal actions required.

Organizational Levels

My discussion of our procedures up to this point has not defined the formal decision chain. For most aircraft competitions the Commander, NAVAIR is designated as the Source Selection Authority. When this is the case, the Source Selection Advisory Council is chaired by our Vice Commander and made up of the Assistant Commanders. I normally chair the Source Selection Evaluation Board which is made up of the Directors of the various Divisions involved in the evaluation.

The evaluation, recommendation and decision levels are respectively, the SSEB, SSAC, and SSA. Prior to a decision announcement, or program go-ahead, concurrence is normally required through the Navy and OSD Command and Secretarial levels to SECDEF.

In a typical case, these levels are:

1. CNM
2. CNO
3. Navy Secretariat
4. OSD Secretariat
5. SecDef

Everyone should realize that intimate knowledge of the proposals exists only at the bottom of this chain. Actions at the top are based on briefings and analyses from the bottom. In the usual case, the review process is aimed at insuring adequacy of the justification, compliance with previously established program control documents and with the budget.

Although levels above SECDEF are not shown as involved in source selection, the higher executive and congressional levels have great interest in these decisions, and control their destiny by budget actions.

Decisive Factors

There is no magic formula for winning one of our design competitions. Among the factors which have been decisive in past competitions are:

1. Weight -- A good index of cost, logistics, and handling characteristics.

2. Performance -- Usually the most important factor for a combat airplane. For a fighter, we normally compute speed, climb, acceleration, thrust and lift limited maneuverability, range and radius, catapulting, and arresting characteristics. Any one of these items could be decisive. Maneuverability won for the F-8.

3. Flying Qualities -- One of the most critical factors, you can't win unless you rate well on this one.

4. Accessibility and Maintainability -- If its not maintainable, we won't buy it. One significant factor in the F-14 selection.

5. Design Excellence -- With equality in other areas, the detail ratings will win for you as it did for Grumman on the A-6.

6. Cost -- A significant factor in most firm-fixed-price competitions. It was determining in the A-7 competition between two designs of almost equal technical merit.
   
   

General Problems

Among the questions normally directed at us when we state that we base our decisions on NAVAIR estimates is the one which questions our ability to compete with industry due to the great difference in manpower which exists. That difference is more than offset by the three facts that our data base is greater, we have more experienced evaluators doing the work, and we are more objective.

As you already know, we have been going through a period when nearly all problems had to be offered for solution in a quantitative form, even though the subject matter was not particularly suited to that form. There is a continuing pressure to utilize numerical rating systems in competitions even within the Navy.

In general, the experts who are "helping" us with instructions on how we should do our business have no experience in the field. It seems that most of their ideas solve problems which don't exist, or have no bearing on the problems which do. There is altogether too much emphasis placed on standardizing methods and formalizing procedures which limit our ability to get the job done with a minimum of effort. Flexibility is needed.

Point Systems

A point system used by our sister service on a new fighter some time ago illustrates, in part, why I'm opposed to numerical rating systems. In this case, a total point score of 1000 was used, broken down into four "AREAS", "Operational", "Logistics", "Management" and "Technical." The latter was given 330 points out of the total. The 330 points were broken down into "ITEMS" with one of them, "Air Vehicle", assigned 100 points. In turn, the breakdown continued into four "FACTORS", one of which was "Aerodynamics", made up of 6 "SUB-FACTORS", the last of which was "Performance". It is apparent that out of the total of 1000 points, only about 5 were used for what we in the Navy consider the most important characteristic of a combat airplane.

Other disadvantages of point systems are well known to those experienced in the art. Almost as much effort is required to establish the point system as to complete an evaluation of the proposals. To complete the scoring system all designs must be given the same detailed treatment, requiring as much effort on the worst as on the best. This is an inefficient utilization of our limited manpower.

Summary

In summary, we believe that the design competition method of selecting a contractor, as we have practiced it in naval aircraft procurement, has demonstrated its worth. We take full advantage of our experienced personnel, we try to be fair with industry, and to date, have had no trouble in justifying our recommended selections. We know of no simpler system that is not objectionable on many other grounds.

We think our system works, and for those who disagree we have the following quotation:

"Your knowledge of the facts is so incomplete that you not only never did, but even more obviously do not now, know what you were or are talking about."