Exhibit A-3. Retype of a paper by G. A. Spangenberg presented at the 33^rd Annual Conference of the Society of Allied Weight Engineers, Inc.. 6-8 May 1974. SAWE Paper No. 1020.

AIRCRAFT PROPOSAL EVALUATION METHODS

This is a management rather than technical type of paper. It solves no problem for the weight engineer working on a new proposal but does give him some insight on the process of aircraft evaluation and contractor selection as it has been practiced in Navy design competitions, and as compared to Air Force Source Selections. The record of the Navy is defended and a case made that the design competition method is a fair one and should be continued. The opinions expressed are those of the author, who recognizes himself as a prejudiced observer, and are not necessarily those of the Navy.

In the normal course of events, military aircraft proposals are submitted to a government agency for evaluation and selection of the contractor who will receive a contract. It is essential to all participants that the evaluation and selection process be conducted with integrity and not allowed to disintegrate in a morass of engineering incompetence leading to a decision based on political favoritism. Although there appears to be wide-spread scepticism that any competitions are run fairly with awards based on merit ⁽¹⁾, the author knows from first hand experience that nearly all the winning contractors in Navy run, formal and informal, aircraft competitions since 1940, deserved to win. Below are listed the only cases which can be recalled in which the award was made to other than the best proposal as determined at the evaluation board level:

1. In 1962, Deputy Secretary of Defense Gilpatric overruled the Navy and directed award of the X22 to Bell on the stated grounds of "experience and past performance" (in the VTOL field). So called "industrial statesmanship" actually appeared dominant in the decision, with Douglas and Los Angeles needing work less than Bell and Buffalo. This was the first case in which a service decision was overturned by OSD. ⁽²⁾.



2. In 1958, the W2F-1 (now E-2) award was given to Grumman on industrial statesmanship grounds after an initial recommendation had been made for Vought. This decision was made within the Bureau of Aeronautics, and at a level which would now be the Source Selection Advisory Council.



3. In 1940, an award was made to Boeing for the XPBB-1, but only after Vought-Sikorsky was announced as the winner and paid for their design effort.

The record of the Naval Air Systems Command (NavAir) and its predecessor agencies, the Bureau of Naval Weapons and the Bureau of Aeronautics thus have a integrity average of about .980, with only the W2F-1 award really counting against them. The XPBB-1 award was honestly handled, and the X-22 was obviously beyond the control of the Navy. The procedures and methodology used in attaining this record have changed very little over the years, although the appearance has been tailored to conform with Department of Defense Directives and overall Navy Instructions. Every effort has been made to continue the tradition of giving all competitors a fair chance, ensuring that the rules of the game are understood, and avoiding as far as possible unduly restrictive practices in the name of security. If our acquisition system is to succeed, the best proposal must be allowed to win competitions, so that the rewards of production can be achieved by the contractor and the best possible weapon made available for use by the service.

Although occasionally an airplane, such as the A-4, is developed as a result of an unsolicited proposal followed by a sole source negotiation, the great majority of our weapon systems result from some form of competition. In the Navy's case, the process is still called a design competition, as it was in the 1920's, in which qualified bidders are allowed to demonstrate their technical competence by submitting design proposals. After the proposal reaches the government a two step process of evaluation and selection starts. These two steps, while distinct, are interdependent, and should not be separated. Probably the fundamental difference in philosophy between the procedures as practiced in navy aircraft competitions and those followed by the other services is that the navy has not separated them while the others tend to isolate the evaluation process from that of selection.

Although this paper concerns itself primarily with the evaluation and selection portion of a design competition, some comments are necessary on the whole of the acquisition process, which starts in the Navy case with a need developing in the fleet. The need is reduced to a "Requirement" in a closed loop operation between the engineers and operators involving conceptual studies, operational analyses, scheduling and budget studies. After completion of this phase, and with the project firmly in planned budgets, the competition process itself starts with preparation and issuance of a Request for Proposal (RFP) to a selected list of bidders for the development of an aircraft, and sometimes (and preferably) for a follow on production quantity. The process is completed with the signing of a contract. The procedures used are not necessarily applicable to study efforts, to small hardware developments, nor to exercises such as those conducted recently in the name of "prototyping." The rationale for that type of acquisition will be left to those speculative theorists who devised them.

In the Navy system, the RFP is prepared as a joint effort by individuals within the NavAir functional groups and divisions who will later evaluate the proposals, administer the contract, and monitor the development. The RFP includes a type specification for the aircraft which allows wide latitude for design freedom, but spells out the basic performance, strength, and mission requirements. Rules for government or contractor furnished equipment are given. Lists of acceptable power plants are noted. The intent, basically, is to define the product in enough detail to allow a wide open competition for the airframe portion of a well defined weapon system. For example, design competitions should not be used to select between different concepts such as a high endurance, subsonic, long range missile carrying aircraft versus a supersonic, short ranged, dog fighter. In addition to the aircraft design and contractual requirements, the RFP includes requirements for design data, tests, logistic programs, and management information for the development phase as well as the data requirements for the proposal itself.

The RFP is forwarded to a bidders list normally established in advance by a "letter of interest" to those manufacturers considered fully qualified to develop and produce the aircraft. Manufacturers who are not included on the bidders list are usually permitted to submit proposals if they request the opportunity. Their non-inclusion on the original lists however, serves as a warning that a large scale bid effort may be unwise on that particular development. In recent years, the amount of control exercised by OSD has increased to the point where approval of the RFP by elements of that organization has been required prior to release.

After a period of time, normally three to four months, the bidders deliver their completed proposals, and the evaluation phase of the competition starts. In NavAir, the effort is coordinated by a small division charged with that responsibility for many years. The objective of the evaluation effort, of course, is to develop as consistent a set of facts as possible on the competing proposals so that valid comparisons can be drawn and overall merit determined. The evaluation is conducted in place using the same people who collaborated in developing the RFP, and many of whom had also been involved in the conceptual work preceding the competition. Independent estimates of weight, performance, and cost are prepared, and all other parts of the proposals reviewed. After that review, each division prepares a memorandum summarizing the evaluation for the items under its jurisdiction and including ratings of acceptability for each design and relative rankings. The evaluation task is more difficult than might be surmised from the brief description above. The process must normally be conducted in too short a time with too few people. On an average, the government review period is about half that used by each contractor in his formal preparation of the proposal, while the number of evaluators available is but a fraction of those used by each bidder. This time/manpower constraint is particularly critical in the areas of weight, performance, and cost where the Navy, at least, produces its own estimates and uses them, almost exclusively for selection purposes. The individual evaluator must be experienced in his particular field, must be one whose estimates have proved sound, and whose judgment is respected by his industry counterparts. As an example, the engineer in charge of the weight estimates is expected to develop a full weight statement on each design as it would be delivered for service use, fully comparable with the other competing designs, and also with the operational inventory. The estimator obviously must have his methods well developed prior to the competition, and must have a close working relationship with the specialists in other design areas. Before the estimate is complete, it must reflect the changes and corrections being requested by other divisions, structures, materials, power plant, etc. The task is difficult; it is not as some imagine, a brief perusal of the submitted report followed by a scoring mark, nor the application of a percentage correction factor based on a bidder's past record. The job of the estimator is aided immeasurably by submittal of sound estimates by each bidder backed up by a clear, but concise, justification. When the government's estimate checks that of the contractor, the justification is obviously not as necessary as when differences do exist. When a bidder submits a low estimate, either in absolute terms or in comparison with the other proposals, his justification is the only hope he has for acceptance of the low figure.

Following completion of the initial evaluation with comments on hand from all divisions, the selection process starts. The methodology is simple, comparisons are made, and the least promising designs eliminated, including any design with a major deficiency. Although the precise definition of such a deficiency has never been made, it is really one which would require major redesign to correct. In general, the preliminary design groups in industry would agree on the type of change which would eliminate a competitor. Examples could be a weight empty estimate which is low by 15%, a wind over the deck required for catapulting of 20 knots when 0 knots was required, or a radius of 375 mile when 500 was required. It would not be the change of a flap angle to correct a single engine rate of climb or take-off distance deficiency, nor the selection of too low a maximum structural design take-off weight, nor the use of non-qualified ejection seat. All of the latter items while requiring correction, would not be regarded as the sole grounds for elimination. In the usual case, the final selection is made after narrowing the field, and re-estimating the remaining designs when each is corrected to the configuration which would be procured. In this step, structural design and equipment differences are resolved as necessary. The winning design is selected by straight comparisons, considering all factors, design, management, logistics, cost, and schedule. The initial selection is made by agreement between those who have been running the competition, the Project Officer, and the Program Manager. Current directives require a formal Source Selection Evaluation Board, Source Selection Advisory Council, and a Source Selection Authority. The latter is normally the Commander, Naval Air Systems Command,, while the Council is composed of his Assistant Commanders who head the functional groups. The Evaluation Board is composed of the involved Division Directors. The decision chain has really not changed from that always used in the Navy aircraft process. As would be expected, the individual evaluators are responsible to their Division Directors, who comprise the Evaluation Board, the Directors are responsible to their Assistant Commanders, who comprise the Councils and who in turn are responsible to the Commander.

During the competitive process, communications between the manufacturer and NavAir are minimized and controlled, but not eliminated. Questions from a contractor are answered directly as promptly as possible, and then the answers made available in writing to all bidders if the question and answer are of general interest. During the evaluation period, requests for clarifying data are again controlled, but transmitted promptly to the bidder. Additional information offered by a bidder after proposal submission is usually accepted, if the intent is to clarify or amplify the proposal rather than to change it.

Following announcement of the winner, bidders are individually debriefed if they so desire. Normally, the initial debriefing session is limited to an overview of the competition, perhaps two hours in length. The contractor is shown the weight and performance estimates used by the Navy, together with the evaluation comments on his design from all the divisions, and is provided with the rationale for the selection decision. Comparisons with other losing designs are not made. In subsequent sessions, the contractor and Navy engineers get together to review the evaluation in more depth in each area. Although the basic intent of debriefing is to enable the manufacturer to submit a better proposal in the future, by avoiding the repetition of mistakes, it also has a beneficial effect on the objectivity of the evaluation, since each evaluator knows he may have to discuss and justify his comments to his counterpart in industry.

In the mid-fifties, the Air Force changed their contractor selection method from that of a design competition to "Source Selection." Their version of a design competition had been similar to that of the Navy but conducted with greater formality, and security, and with the selection decision made at higher levels, normally by the Secretary of the Air Force. At the time of the change in procedures, the Air Force publicized the move as a means to reduce bidding expenses and to use technical manpower more efficiently. (Some cynics, however, attributed the change to the apparent number of decisions made in the Pentagon which differed from the recommendations emanating from the evaluators at Wright Field. If political decisions were to made, the procedure might just as well be adopted to it). In "Source Selection", as first conceived, only brief management proposals were required to be submitted from the industry. No actual design work was to be done until after selection of the winning contractor(s), when the design would be developed in close cooperation with the Air Force in a funded effort. Although the concept was widely supported and the theory later adopted by OSD, it actually is quite unworkable in the normal case. As might be expected, "Management" proposals from major producers are remarkably similar, and provide little or no data to justify the selection of one contractor over another. There are always several manufacturers available who are fully capable of producing the aircraft, and who are unwilling to concede that any one else is better qualified. Gradually, th Air Force increased its requirements for design data on a specific design until it equaled that formerly required in its design competitions. Despite the renewed emphasis on design, the Air Force continues with the premise that only a source is being selected. Obviously, this gives greater flexibility to the decision maker, but leads to greater justification problems, and tends to obscure the rules of the game for everyone involved.

By 1960, the Air Force procedure had settled down to one which was built around a System Source Selection Board (SSSB) which directed the activities of an Evaluation Group (EG), and which recommended a winner through a chain including their System Command, Logistics Command, "using" Command, the Air Council, Chief of Staff to the Secretary. A point scoring system was used with the Evaluation Group raw scores being weighted by the SSSB before their use for selection purposes. The point system was used as a guide at the SSSB level, and did not automatically determine the source, although this point was apparently not fully understood by higher authorities. The Army had a similar procedure. OSD started standardization efforts in the early sixties, which eventually culminated in the adoption of a system similar to that outlined in the Air Force regulations. A significant difference, however, was that the evaluation and selection processes should be divorced. The philosophy behind this change was expressed by a former OSD official during the TFX hearings ⁽³⁾ as:

".... it turns out in practice today to be difficult if not impossible to alter a source selection decision on a major procurement action unless you are willing to send the Source Selection Board back to work with new instructions. Otherwise you are placed in the position of making an arbitrary change in the face of a fully objective procedure and determination. That is, in general, infeasible. It is very time consuming, and it exposes the senior decision makers to potentially severe criticism and even outside pressures. It is practically never done.

The process I have described would put the senior, responsible people in the position of making rather than approving or merely presiding over decisions already made. Above all, it would mean that one could consider the relative merit of one selection in comparison with another by noting the effect of slight perturbations in the value assigned to the evaluation criteria. Sometimes a very small percentage change in these alters the outcome of the evaluation completely. These are the circumstances under which the overall judgment of the most senior and responsible officials is not only needed but is indispensable, for otherwise selection almost automatically results from evaluation."

It is clear that those espousing this philosophy had no confidence in the ability of those at lower levels in the Defense Department who had been evaluating and selecting aircraft for years. (However, it is freely admitted that the reverse was also true, viz., that those who had experience in aircraft evaluation and selection had no confidence in those who espoused the above philosophy).

Before commenting on the differences between the Navy and Air Force systems, a more detailed description of the latter as practiced in the TFX and C-5 cases. ^(4)(5)(6) As noted before, Army and FAA procedures have been quite similar. The SSSB is composed of one voting member from the "Using" Command, Systems Command and Logistics Command. Other representatives of each of these Commands are also members and attend the SSSB meetings. The SSSB utilizes a small ad hoc committee, normally including experienced personnel, as staff to prepare the SSSB action papers, including staffing of the Evaluation Group, establishment of the point system, and description of the criteria. The members of the Evaluation Group prepare "Standards" for the criteria against which the proposals are scored using a rigid scoring system (10 Excellent, 8 Very Good, .... 2 Poor, 0 Unacceptable). After the evaluations, the Group and team leaders present the evaluation findings and raw scores to the SSSB. After a full discussion of the evaluation with members of the Evaluation Group, the latter are dismissed. The board then converts the raw scores into the final weighted scores using weights previously agreed upon. The voting members, in executive session, formulate their recommendation for award with a very brief rationale for the decision. As noted previously, the recommendation is forwarded to the Secretary through the decision chain of the Commanders of the involved commands, the Air Council, and the Chief of Staff. Each member or group in the chain can either concur or recommend alternative courses of action. The entire recommendation process is even more closely held than is the evaluation itself. Briefings by the Evaluation Group members are usually given to those in the selection chain prior to the time they make their recommendations. When the Secretary's decision differs from that of the SSSB, a rationale is normally prepared by his staff. ⁽⁵⁾⁽⁶⁾

There are obvious similarities, particularly in external appearance, of the systems used in the past by the Naval Air Systems Command, and those used by the other services, but there are also significant differences, some of which are:

1. People -- The Air Force and Army use relatively large ad hoc evaluation groups (400 reported for the C-5A, about 100 for the Army's HLH), while the Navy uses its functional organization. The difference really is that the Navy calls on all of its staff on a part time basis, while the others use part of their staff on a full time basis. The Navy's job is also eased by the fact that it has had a small, relatively stable division charged with the responsibility of running aircraft competitions for many years (now on only its third Division Director since the 20's).



2. Location -- the Navy aircraft competitions have all been run in place, with each evaluator allowed to work at his own desk, with his own files and equipment. The Air Force normally groups their effort in a separate facility on the base at Dayton, while the Army seems to run to remote locations. In place evaluations are obviously much less expensive, and far easier on the evaluators. The arguments against the practice are interference by routine business, and lack of security. The lack of full time attention cannot be argued factually, but in no other practical way can all of the best people in the organization be brought to bear on the problem. The security argument is specious, as all parts of military procurement agencies routinely handle problems with an equally high degree of confidentiality.



3. Criteria and Instructions -- The Navy system permits the process to be carried out with a minimum of instructions to the evaluators (usually a single page memorandum), since all divisions have already been involved in the particular project, all have a normal responsibility for evaluation work, and have done them before. The other services have run from 15-20 pages of "Criteria" and "Instructions" to the evaluators to a 1 inch thick "Handbook" used by the FAA on the SST evaluation.



4. Estimating Quality -- The most effective deterrent to competitions becoming "Lying contests" is by the development and use of sound and comparable data on all the proposals during the evaluation. In the services other than the Navy, the ad hoc type of evaluation organization invariably suffers from a shortage of fully qualified personnel to do the most difficult tasks, the independent estimates of weight, performance, cost, and flying qualities. The shortage is related to the basic organizations of the other services which are of the project, rather than functional, type. No Program Manager will willingly release perhaps his only weight specialist to a new evaluation project for a significant period of time. The problem is also compounded by the requirement for estimators to prepare "Standards," fill out rating sheets, and write justifications for those ratings on factors and sub-factors included in the total point rating system. The Navy consistently rates each design against each other, while the Air Force has insisted on rating against a "Standard." Although the difference appears subtle, it assumes more importance when one is advised that even though "A" rates higher than "B" against a "Standard" it does mean necessarily that "A" is superior to "B." In general, more emphasis is placed in Air Force evaluation on th plan to achieve a design, than on the design actually presented.



5. Decision Process - The Navy selection decision starts at the evaluation level, is made by agreement between technical and operational types, who present their recommendations openly in all presentations. As the process reaches higher levels, the presentation is normally restructured to suit the audience. The Air Force gives the same evaluation briefing to all levels with no recommendations included since the presenters are not privy to that information. The recommendation channel is a one way channel without feedback. This can obviously lead to peculiar results when presenters from the Evaluation Board who believe "Contractor N" has been recommended answer questions from a Secretary who has been informed that "Contractor E" has been recommended by the Selection Board.



6. Decision Level -- In the Navy system, the Selection Authority has been held at the level where the technical language of the aeronautical profession is understood, and the implications of design deficiencies appreciated. In any system which requires the decision to be made by one who understands neither the language nor the implications, the chances of an unfair decision are greatly increased. The time constraints under which the Secretarial level is forced to operate also tends to preclude full disclosure of all relevant issues and factors.



7. Total Effort - Few statistics are available to compare directly the total effort required in the evaluation and selection process, and those that do appear suspect. The Air Force reported 275,000 man hours expended during the four rounds of the TFX selection, while the Navy reported 4000 man hours on the X-22. The Navy's part in the TFX effort was estimated at 10-12,000 man hours which appears fairly consistent. Although there is no doubt that the NavAir system requires less effort, it is not claimed to be in the ratio indicated above, but is certainly less than one half.



8. "Security" and "Control" - Evaluation efforts in the other services are hampered to a significant degree by restrictions imposed in the name of "security" and "control." There have been cases where evaluators were forced to complete one design before starting on the next, precluding comparative study. In other cases, "Technical" teams have been isolated from "Operational" teams. In many cases, cost proposals are withheld to an unrealistic degree. While restrictions against disclosure of evaluation results are understandable, restrictions against disclosure of evaluation methods are not. Nor can one understand the need for use only of specific notebooks locked up at all times except when in actual use. Prohibitions against contact with the manufacturer except by written communications through a contracting officer have prevented badly needed meeting of the minds, during both proposal preparation and evaluation. It is often quite unfair to not answer a question from a bidder, or not to obtain clarifying information.



9. Page Limited Proposals -- The requirement to limit the number of pages in a proposal has frequently been employed by the other services to the detriment of the evaluation process. All too frequently, the page limits are set without regard to the data requirements in the same RFP, which makes the proposer's job impossible. Not as apparent, however, is the fact that the evaluator's job is also made more difficult, both by lack of data and by unreasonably small printing. In many cases, missing data are supplied later in response to inquiries, but time has been lost.



10. Debriefing -- Debriefing in the Navy system has provided the government weight and performance estimates to the bidder, while the other services, although not consistent, have sometimes provided only generalized comment. Following the early rounds of the TFX source selection, for example, bidders were advised that deficiencies existed, but the magnitude (and sometimes even the direction) was not defined.



11. Before concluding this paper, some comment should be made on the role of cost quotations in the selection process. During the period when Navy initiated development with a cost reimbursement type contract for an experimental quantity of aircraft, cost quotes had virtually no influence in selecting a winner. This does not mean that price was not a factor in the weapon systems which were being developed but only that this type of cost quotation, in a competitive environment, had proved unreliable. Comparative cost could be judged more accurately by looking at other characteristics, weight, size and materials. As budget pressures increased, cost considerations become more important, leading the Navy to turn to fixed price type contracting in the early sixties. This eased some budget problems by eliminating the cost overruns experienced with cost reimbursement type developments, and also allowed price to become a significant factor in competitions. To reduce the possibility of "buy-ins" on the initial contract, ceiling price options for follow on quantities then became a standard part of the Navy acquisition process on all relatively conventional developments. This step made the evaluation and selection process more difficult by introducing the possibility of a Quoted Price/Design Merit tradeoff. Small differences in total cost would not change a merit decision, but large differences would require a corresponding margin in capability. No good solution is yet apparent to the problem of fixed price quotations from responsible producers which are well below the government estimates. The procuring agency is subject to criticism if the contractor is not held to his bid, but then is later criticized for accepting a "buy-in" when large losses occur. My personal conviction is that the government must base the award and negotiate the contract on the basis of the bid known to be low, but only when the total estimated loss is within the capability of the contractor to absorb. The Navy's record in holding a manufacturer to his commitments in aircraft competitions has been quite consistent, those who underbid on the fixed price deals lost money. The record of the other services has been less consistent. The Army held the line on the OH-6, one of the most underpriced awards in history on a percentage basis. The Secretary of the Air Force in the TFX decision gave credit to a slightly higher bid on the basis of cost realism, but used a lower price in the C-5 case to justify the award. ⁽⁶⁾ A repricing formula included in that source selection was apparently intended to limit the loss of any of the potential contractors (regardless of the actual price which might finally be reached), making the use of cost quotations as a primary selection factor of doubtful merit.



12. In my opinion, the record clearly justifies the continued use of design competitions as the preferred method for selecting contractors for new aircraft development. As practiced by the Navy, the rules of the game have been established by consistent usage to give the industry proposal teams a fair chance to win and earn their reward. Although no other system has been shown to offer such uniformly satisfactory results at a minimum level of effort, it is in danger of being destroyed by strict adherence to DOD policies and regulations adopted initially in the name of standardization. The system can also be destroyed by allowing the technical competence of the procuring activity to fall below the level required to produce accurate estimates of the weight, cost, and performance characteristics of the competing designs. In recent missile competitions, the Navy itself has followed some of the practices which have been criticized in this paper as non-productive.



13. It is also clear to me that the selection decision, and the control of the entire evaluation and selection process, should be made within the organization which is to bear the detail development responsibility. The selection decision itself should be made, in fact, by those who are intimately familiar with both the proposals themselves and the ground rules employed during the entire competition. Those higher in command within the service, or in OSD, are remote from the process and must use facts presented to them in a highly condensed format. When their decision differs from that of those more directly involved, the probable cause is a lack of communication rather than any fundamental difference in judgmental ability.

References

1. "The Law of the Proposal Jungle" by Frank R. Smith, "Astronautics and Aeronautics," October 1973.
2. Preparedness Investigating Subcommittee, Senate Armed Services Committee Report, "Award of the X-22 (VTOL) Research and Development Contract, " Release Date 7 February 1964.
3. Senate, Permanent Subcommittee on Investigations, Committee on Government Operations, "TFX Contract Investigation," Part 5, page 1295.
4. Ibid, Part 1.
5. Ibid, Part 2, Page 350.
6. Senate Armed Services Committee, FY 1970, Mil Proc. Auth. R&D, Part 2, Page 2009.