Furthermore, numerous studies have proven that the participation of users in the design of their work tools is also a factor contributing to project success. The advent of prototyping tools facilitates user involvement by acting as an intermediary, a mediator between the designer and the user.

This paper deals with the participation of users in the solution evaluation phase of a design project, that is to say, an exploratory phase for possible solutions and analysis of user requirements.

However, the problem of participation cannot be restricted to the evaluation strategy. The participation methods, the tools supporting participation, and the choice of users involved are intrinsically related.

Involvement of end users in design is not a recent issue, as evidenced by French ergonomic studies (Daniellou and Garrigou, 1993; De Montmollin, 1981; Garrigou, 1992; Wisner, 1989). Participation was also the spearhead of Scandinavian countries, which, in the 80’s, developed the Participatory Design approach to go with major technological changes in enterprises such as information technology and automation of production facilities (Bloomberg, 1993; Bodker and Gronbaek, 1996; Bodker, Grondaek, and Kyung, 1993). In the United States, user participation was focused on software design, in particular in HCI studies (Bannon, 1986; Dillon and Watson, 1996; Norman and Draper, 1986).

• Many studies have shown that user involvement improves the social acceptability (Damodoran, 1996; Noyes, Starr, and Frankish, 1996; Suchman and Luczack, 1996) of changes. For example, special attention paid to user-involvement in design resulted in much better positive attitudes of workers at the floorshop, and successful integration of computerisation, digitisation and automation in the industry. All industrial sectors have undergone these changes, which have affected all trades, from the least qualified to the most qualified. User participation was used as a way to understand how these radical changes in users’ work could be implemented and what had to accompany such changes, for example appropriate training, qualification, and so forth. Including the user in the design of his/her tool, meant that he/she played an effective role in work changes and in his/her own development, instead of simply having to put up with them. The Scandinavian Participatory Design concept left its mark on this epoch.

• User involvement also improves the usability of the work tools (Nielsen, 1993). The users’ needs are better known as well as the potential difficulties with the use of this new tool.

When looking back to this extensive literature, another fact is that most studies were focused on user intervention methods, but only a few of them shed lights on the very nature of data gathered from the participation, and the use of these data for the design process. Only a few studies considered in depth the dialectic between the choice of users and the type of data collected.

Also, studies were mostly dedicated to the office field and rarely to complex, high-risk fields such as the nuclear or aviation industries. Among the main differences existing between the design of office applications, and the design of aviation applications, are the users (education, motivation), the type of tasks, the type of product, and the environment.

• Concerning the users tasks, the complexity of high-risk applications is mainly due to the high level of constraints in terms of time pressure, the amount of data to process, the level of incertitude and the type of human-machine cooperation.

• As regards users, pilots represent for example a multicultural population, highly qualified and trained on the aircraft they fly, but they may also be skilled on other machines and in the use of general PC tools.

• The life of the designed products is very long compared to other domains: An aircraft is designed to last 30 to 40 years. This means that designers have to anticipate the evolutions to technology and users.

• Furthermore, the design of high-risk and complex systems is different from the design of office or general public systems or workstations. The aviation and nuclear domains have developed a risk culture, which is unmatched elsewhere, thanks to technological advances, automation, safety management, and the relevant precautions (system redundancy, failsafe and fail passive concept).

In the literature, the way to pose the problem of user involvement is quite simple. Most authors prefer the participation of end users but they deplore the fact that this is difficult or even impossible. Almost para- doxically, when balancing pros and cons, some authors suggest to avoid participation because the involvement of end users has more drawbacks than benefits, may hinder creativity, or slow down the design process.

To conclude this introduction, the user participation seems to be a profitable way to improve the efficiency and the acceptability of a new work tool, but the methods and practices developed for user participation should be reviewed to take into account the specificities of a high-risk system design.

This paper will try to define how users can contribute to design and what type of user is appropriate. The study takes part of a global effort to improve Human Factors in the aviation design (Sarter and Amalberti, 2000). The perspective is to provide aircraft manufacturer with means to improve the efficiency of user participation in the cockpit design. In the aviation domain, different categories of pilots can participate in design: test pilots and line pilots.

Until now, the line pilots are involved in the design mainly through meetings and task forces on specific aircraft systems. The test pilots are more integrated in the design studies. A global tendency in the human factors community, and in aviation regulations, asks for a greater participation of line pilots in the design process. The aviation industry is not against this extension of participation, and modifying accordingly the design process, but asks for a thorough validation of any changes. The proof that the modification is valid must be clear, justified by demonstrated improvements in terms of quality, safety, and financial costs.

If considering the present practices, the involvement of test pilots has been greatly criticised in the last few years. Several sources of bias have been identified, due to the test pilots’ specific experience and knowledge of aircraft. Test pilots do not perform the same routine operation, nor are they under the same constraints than line pilots. They use to be over-experienced in critical situations, very uncommon for line pilots.

These differences suggest that test pilots and line pilots have different mental models of the aircraft and of the environment. Nevertheless the systematic participation of line pilots is not the “cure all” and it can lead to poor design.

Defenders of the Human Factor approach claim that test pilots are not representative of the end user population and that can lead to a design centred on test pilots. For example, the Human Factors community says that the automation problem arises from the fact that there is no adequacy between the complexity and the pilot’s level (Tenney, Rogers, and Pew, 1998). The involvement of line pilots can be considered as a potential solution to better anticipate the difficulties of the population of end users, but this hypothesis has to be tested.

The study presented in this paper aims at understanding the precise contribution of the different categories of population of pilots. If the contributions differ, then the manufacturer has to decide which population of users is more relevant for a particular problem of design. We will work on the decision rationale in order to help the manufacturer manage the project.

The paramount questions are then: Who has to be involved, how, and for what?

For the aviation domain, the average pilot is generally considered as the end user. This is a perfect target for the design solution. But we can question whether a typical, “average” pilot really exists and whether we can characterise him/her. There are approximately 3 millions line pilots in the world!

In fact, several studies have dealt with the difficulties experienced by line pilots in understanding and mastering automation (Sarter, 1996; Woods and Sarter, 1998). A few studies (Amalberti and Deblon, 1992; Amalberti and Valot, 1990; Billings, 1997; Javaux and De Keyser, 1998) have investigated in more depth the mental models used by pilots to fly aircraft in the wide variety of contexts they confront. Moreover, some studies seem to be contradictory. Helmreich, Merritt, Sherman, Gregorich, and Wiiener (1993) indicate that there are differences in the attitudes of European, American and Asian pilots to automation. Recently, However, the recent JARTEL EC research project have shown much little cultural difference among pilots (see Hörman, 2001), hence the problem of cultural differences largely remains a land of dispute among aviation experts.

To come back to the question of user involvement, a very simple answer is that the end user is the most qualified to design his/her work tool because he/s is the one who knows the most about the work. If this can be relevant for the modification or the current evolution of a work tool, it is not a satisfactory answer for a new design, when the future end user has also to be created, when new user training is envisaged. In aviation, some pilots begin to fly with a glass cockpit and perhaps they will never fly previous aircraft generations.

Furthermore, it is known that there is a great diversity within the category of end users, and even inside categories with the pilots’ flight experience.

• The expert/novice paradigm: The authors who have studied this paradigm have found that experts and novices have different mental models of the expertise domain and different behaviour in their job achievement. For Visser and Falzon (1992), an expert can do the job more rapidly than a novice, because of s/he has a quicker access to the knowledge. He/s also has a better representation of the problem. He/s is able to develop inferences facilitating his task. An important difference between experts and novices lies in their level of reasoning. Experts reason at a higher level of abstraction (Bisseret, 1987; Guindon, 1990). This is explained by the fact that experts have a functional organisation capability whereas novices have a surface knowledge organisation. Amalberti and Valot (1990) performed an experiment where pilots have to describe a tactical mission analysis. They showed that pilots gave special emphasis in their discourse to the flight phases considered as difficult for them to carry out. Hence, novices over-described the low visibility-low altitude flight, whereas experts pilots insisted on mission preparation, target approach, and the return to base. Experts also exhibited better meta-cognitive capabilities than novices did. They were well aware of their strong and weak points and took steps to neutralise their weaknesses. The authors showed that experts used ranked knowledge and were able to change their cognitive level of control, going up and down the various levels of abstraction depending on the task demand.

This expert/novice dimension is relevant for the present study in a different way, since line or test pilots are experts in their respective specific field, but often beginners in the specific field of pilots belonging to another category. This makes us think that some pilots will apply different types of reasoning, which may have visible effects on their task completion strategies.

• Expertise in the sub-domain: Few studies have been dedicated to the reason of among-experts variation in task-completion. Visser and Falzon (1992) demonstrated that “different prescribed tasks entail different representations of a given object” and that “different repre- sentations of a given task may entail different effective tasks”. This was confirmed by Hansman, Pritchett and Midkiff (1995). During a study on future ground-onboard communications involving different operational categories of pilots (line, military and general aviation), they showed that the decisive factor in the pilots’ perception of communications was the instruction and scenario to play, and not the experience in the type of operation.

This leads to think that the participation of different categories of pilots is worthwhile if categorisation is based on different types of operations. This suggests that it is preferable to give priority to the number of categories rather than to the number of representatives per category.

In the case of the design of a commercial aircraft cockpit, test pilots and line pilots have the same basic training. Then they follow specific training courses and in practice carry out different, but not entirely dissimilar operations.

On one hand, the task of test pilots is to evaluate and develop aircraft under normal and high-risk situations. Pilots work in the entire high-risk domain (Figure 1) and their job is to use the aircraft systems in downgraded modes to detect the weaknesses and find any traps. On the other hand, the task of line pilots is to use aircraft on a daily basis and work in so-called normal situations, only very rarely encountering dangerous situations. Once again, the participation of the two categories of users seems relevant and complementary.

Some other reasons are advocated to work with representatives. They refer to a deliberate choice. The representatives may come from different origins and have different functions in the organisation.

The end user is represented by the client (Webb, 1996). The quoted advantage is that the client is aware of the functional needs of the end users but the drawback that we can emphasise is that he/s cannot be aware of the “real use made of the tool and the real use contexts”.

He/s can be also a professional tester as in the “general public” software design. The professional testers can be people whose main mission is to test a product as in some car manufacturer, or people who are specifically trained to test the product. In the aviation domain, the test pilots are trained professional testers. Their mission is to test the whole aircraft, for the flight quality, the performance and the human-machine interface. They know a lot about the aircraft and the commercial flight operations (many of the test pilots have a civil qualification) but they have not the same daily experiences than the line pilots.

Some human factors experts play the end users’ role. They are used to be in close contact with the end users or they can even use the products. This allow them to acquire a good knowledge of the users’ job and to combine it with a good knowledge of human factors design criteria (Poltrock, 1989) and methods.

They have the ability to propose solutions and the arguments to justify and to defend them.

We think that this last principle is highly subject to criticism and can be penalising for the design, as the human factors people are here both judge and defendant.

We prefer that the Human Factors experts keep an independent position and help the designers to criticise the solution.

Furthermore, we think that for complex systems, the Human Factors experts are not able to acquire sufficiently broad and sound knowledge to propose efficient and reliable solutions. It needs several years for a designer to be well trained in his/her speciality.

A major problem highlighted by the ethnographic movement working with user representatives is that there is a great difficulty to find the “well-informed informant” (Bloomberg, 1993), the one who can not only represent the users’ practices but also the beliefs of the whole community. Previously, ethnologists thought that a user representative would naturally incorporate the group information, but it has to be noticed that they have now changed their points of view.

To conclude, without real knowledge of the job and experience in the activity of the final users, the end user representatives may provide relevant assistance in the construction of the functional tasks of tools but, depending on their backgrounds, they cannot cover all users’ tasks. In particular, they cannot help find the “parasitic” tasks, that is to say, those related to the use of the tool in use contexts stressed by multiple recurrent minor problems.

We can conclude that designers need a user requirements analysis and a model of the end users. This is not simple because the model of the users is a dual one: It integrates a model of users in terms of age, sex, training, and a model of the users in terms of domain-dependent knowledge. If these models do not explicitly exist, the designers use their own scope of knowledge.

The need for the fields of information can be summarised by the Figure 1.

This line of thought leads one to consider the means to involve the users. The Human-Computer Interaction community had developed the user centred design approach (Norman and Draper, 1986), based on the leading role in design of the conceptual models of the system constraints, of the user’s model, and ultimately of the use of systems by users. To do that, The HCI representatives traditionally prefer to work with psychologists than to involve the users directly, but their approach is changing. Today they are closer to the French ergonomists or the participatory design movement. These trends have always recommended direct user involvement.

The recent works of Vicente (1999) on the ecological approach confirm this idea. He criticises the pure cognitive approach, which he considers as being too simplistic. In fact, he considers this approach as not taking sufficiently into account the real requirements of the operator’s work and consequently it hides part of the complexity of this work. He recommends that the reality of this complexity be integrated in the user goals.

For aviation, we emphasise the need to take into account various risk domains that may be described as follows. The human-machine system comprises several components, the crew, the technical equipment of the aircraft, the aircraft in its flight domain and the environment (traffic, weather, etc.). This can be described as three domains. The normal domain is that in which the pilot, the technical equipment, the aircraft and the environment are in failure-free behaviour or one where there is no major disturbance. All components of the aviation system function with operating margins. The second domain is downgraded: The components are in a downgraded operating mode and entail a decrease in safety and manæuvring margins. The third domain is the centre of more important malfunctioning and potentially dangerous consequences. Designers must take into account all these three domains during design to provide pilots with aids adapted to each domain. In particular, there is a need to help pilots to return to the normal domain when they move out of it.

We take the side to say that the direct users’ involvement is the suitable and necessary way to capture the users’ requirements and to test the design solutions along the design process. We think that coverage of all the information type and risk domains make it necessary to work with various users, novice as well as expert users adapted to this multiple range of requirements and work practices.

Nevertheless, the participation of users incorporates inconveniences. Critical analysis of these studies most often reveals methodological weaknesses with regard to the means implemented to cooperate with users and to the processing of the data collected.

Participatory design must be within the context of the current choice of a design rationalisation approach, with regard to both the selection of the participating user population as well as the participation methods.

But the aviation domain is not fully comparable to the other industries as a cockpit is designed for 30 years. How can somebody ensure that the human, in 30 years, will be satisfied?

With the same technical aircraft the difficulties change over time because of the evolution of the pilots, of the environment, of the regulation, etc.

The other constraint is that modifications in the aviation domain come rarely from the users, they are imposed by the aviation macro system and some can be difficult to be accepted by the users.

We also think that the need of aviation design as well as high risk system goes beyond this requirements and even that the design decision can be in contradiction with the apparent end users’ wish to fulfil the safety requirements. A very suited solution has to be found thanks to the designers’ creativity.

As we are in favour of the users involvement, we are doubtful about the end user involved alone in the design of complex systems as those designed in aviation, but not for the reasons evoked in the literature: The end users contradict themselves. We think that the test users also have an important role and that is what we are going to study through the experiment that is presented below. The problem after will be to determine a set of decision matrices to chose the best users to test the design solution.

We are going to study the role of test pilots and of a certain category of line pilots. We have seen that test pilots and line pilots have different knowledge and that they carry out different types of operations. Therefore, they have different mental models of aircraft and of their use. One might think that they would make different contributions in the design of onboard systems. Our approach will focus on the arguments they can put forward to criticise solutions rather than on the scenarios underlying their critical judgement.

This experiment relies on the evaluation of a Data Link interface. The Data Link system is considered as a future means of communication between the ATC controllers and the crew. It is based on digital technology compared to the present HF or VHF communication. A major change for the human operators is that they will have to read the ATC messages displayed in the cockpit whereas today they are used to receiving them by the audio channel. Special design studies accompany this work change.

Twelve pilots were engaged in the experiment, six test pilots and six instructor line pilots (gathered in three test crews and three line crews).

The experiment consisted for the pilots in flying a scenario (one hour and twenty minutes long) in a simplified aircraft simulator, coupled with a ground traffic environment simulator. The ATC management was realised by real controllers. The pilots were arranged in three test crews and three instructor line crews. Very few instructions were given to the pilots except that they had to fly using their usual practices. The pilots had been trained to understand and to handle the new Data Link system during half a day, the week before the simulation session.

The flight was audio and video recorded with five cameras, facing each pilot, the main instruments (including the Data Link Interface) and controls.

At the end of each simulation, the pilots were interviewed by an ergonomist in order to collect their critics and to understand their behaviour.

The instructor line pilots are line pilots who train other pilots for a variable period of time in their career. They have a very good experience as line pilots before becoming instructors and they continue to fly in the airlines everywhere in the world IOE. This creates a very interesting population among the line pilots as they can bring their own experiences and the experiences of the trainees they work with to the designers. This is a particular case for the line pilots, but numerous pilots are instructors in their companies or in training centres. Moreover, they are more accessible than the line pilots are. Those who participated in the experiment came from different airlines. They had never participated in the Data Link design studies, so that they had no knowledge about it. On the other hand, like all line pilots, some might have greater knowledge of a particular system, according to what they preferred.

Among the test pilots, three of them were involved in the Data Link design as co-designers.

The flight was a normal commercial flight, without intended failures. The traffic was simulated by a traffic generator and it comprised aircraft equipped by Data Link and aircraft with vocal communication. The ATC controller and the pilots were asked to fly as usually. The Data Link communication system proposed several applications allowing the pilots to reply to the ATC clearance (simple or multiple), to request some changes to its route (DIRECT TO), etc. To be sure to test the Data Link system entirely, the flight was broken down into different phases. The last one was managed by a controller, who played a special role for the experiment: He was in charge of provoking the conditions to activate different applications if it had not been done before, during the “free phases”.

• Task-related: “evaluative referents”: They correspond to the questions “Who performs the task?” (the human or the machine) and “What is the utility of the tools and the displayed information?”. The sub-evaluative referents are “tasks context, human-human cooperation and human-machine cooperation, task interference and sequencing, task complexity and dynamics, and adequacy to the reference situations” (Daniellou and Nael, 1995).

• Artefact related “evaluative referents”: they correspond to the questions “How is the information presented?” And “How can pilots use and handle the interface?”

• User model, organisation and design, and training processes “evaluative referents”: Their description is given in the Table 1.

TABLE 1 :
Definition of the evaluative referent
Définition du référent évaluatif

The test and line pilots brought redundant and complementary contributions to the design of the Data Link system. As a consequence, both categories of pilots serve the design in a meaningful and different manner and this can be explained by the effect of their respective job characteristics.

TABLE 2 :
Number of criticisms and scenarios put forward by each pilot (test and instructor line pilots) during the evaluation
Nombre de critiques et de scénarios évoqués par chaque pilote (pilotes d’essai et pilotes-instructeurs de ligne)

Task related “evaluative referents”:

• Task context (e.g., the need to separate the main flight tasks): The pilots gave very little data on this evaluative referent and the data were very consistent within the subjects.

• Human-machine cooperation: This evaluative referent is related to the pilots’attitude towards using or not the automatism that was proposed in the design solution. As the test pilots rejected the automatism as it had been proposed, the line pilots accepted it. The arguments for the test pilots were that the automatism was not safer and the pilots were no more in the loop when using it. They put forward “that this automatism can be difficult to understand and they are not so much confident in the automatism”. The line pilots’ arguments were coming from another point of view: They found that the manual use of the system, as it was proposed, was too workload demanding, and the automatism should alleviate their job. Moreover, these pilots gave another argument in favour of the device: It could monitor the pilots’ action and detect their errors. On the other hand, they thought that the tasks performed by the device were not “vital”.

• The test pilots put forward “safety” arguments and the line pilots “ease of use” arguments:

Artefact related “evaluative referents”:

Finally, all the pilots gave equivalent judgements and arguments on the evaluation of the surface level of the interface. They were based on the legibility, understanding of the information coding, logical organisation of the information, handling quality, etc.

TABLE III :
Number of scenarios given by each crew according to their categories (test pilots and instructor line pilots), categorised by risk domains
Nombre de scénarios évoqués par chacun des douze pilotes en fonction de leurs catégories
(pilotes d’essai et pilotes de ligne instructeurs), catégorisés par domaine de risque

There is no significant difference in the number of scenarios evoked by the various categories of pilots. All pilots put forward more low-risk scenarios than high-risk scenarios.

On the other hand, contrary to our hypothesis, test pilots did not put forward more critical scenarios than line pilots did.

The different categories of pilots have different mental models, which influence their judgements in front of the design solution.

We have seen that pilots may have different opinions on interfaces, especially automated systems. This may be explained by the fact that they carry out different types of operations due to their jobs.

The test pilots’ arguments are oriented by the safety point of view. The test pilots must remain the masters of the aircraft trajectory and of strategic decisions. They want to be “in the loop”. This standpoint is very easy to explain by the fact that test pilots are guarantors for safety studies. We have seen that test pilots are trained to test system performance and system effect on aviation transport safety.

Furthermore, the analysis highlight a lack of confidence in automatic systems, offset by a strong confidence in human capabilities:

This approach is directly expressed (e.g., “there are always traps”) or indirectly expressed (“you don’t know when I’m going to do it, just let me get on with it”).

Test pilots, given their job, have a more profound knowledge of automation systems; their job is to test automation systems and to discover the traps. They often work with technical systems in project status, whereas line pilots only see completed solutions.

They have developed a suspicious attitude with regard to technologies and automation systems. This can be an important factor in the design process as it is a defence against over-automated system design.

On the other hand, one must not conclude that test pilots, generally speaking, are against automated systems, but simply that the automated system proposed in this study does not seem to be acceptable and does not correspond to their human-machine cooperation model.

Test pilots put great faith in the human. Once again, test pilots, due to their training, acquire specific skill in piloting the machine, in doubled-checking themselves and thinking about their actions and the related consequences on the machine (high level of meta-knowledge: Valot, 1998).

For instructor line pilots, the essential arguments are more oriented towards “ease of use”. Instructor pilots look for a technical system that does not hinder them in the performance of their main task.

Unlike the test pilots, the instructor line pilots have a lack of confidence in humans, offset by a strong confidence in technical systems: The instructor pilots’ reasoning is based on the assumption that human is fallible.

Line pilots consider that one of the major qualities of the automatic systems proposed in the experiment is that these automated systems will operate like a top layer, a safety belt in relation to the action of the human. They are capable of detecting human errors (e.g., “the machine can monitor”). The system may be corrected and recovered. What is interesting in this type of result is that line instructor pilots seem, most of all, to need a system to monitor their actions and not necessarily an automatic system that replaces the human in performing the tasks.

To sum up, line instructors or test pilots adopted the same evaluative referent, with the same interest (number of statements), but from different and not opposite standpoints, formed by their experience. Garrigou (1992) discussed social-cognitive orientations to express the fact that designers and users explore a new work situation according to what they might have been through in the past.

For design, one might say that the two categories of pilots are highly complementary and enable designers to understand the advantages and drawbacks of a technical solution, according to different standpoints.

One can speak about the complementary contribution of pilots (test and line pilots) coming from complementary viewpoints. These viewpoints are explained by the specific nature of the two jobs.

This result is important its the theoretical and methodological repercussions. The different categories of pilots (test and line-instructor) have no specific domain of evaluation, but they have a different view for a given theme. Considering the results, it does not seem relevant to specialise the pilots in the evaluation of specific themes or items of the design. In fact, it is more important to make the different categories of pilots participate in the overall evaluation, knowing that these themes will be evaluated according to different viewpoints.

If the experiment had only included one category of pilots, the evaluation results concerning the design choices would have been different. For example, the design choices would have satisfied either safety or ease of use.

The explanation of this phenomenon is probably to be found in three facts:

Firstly, the initial scenario was intentionally elaborated by designers as a sequence of normal operational situations. If abnormal operational situations occurred, they were not scheduled, and their number was less than the number of normal operational situations. Furthermore, they have never considerably affected the flight. The literature on basic reasoning (Kolodner, 1993; Schank, 1986) behind these case reports that the cases remembered most easily are the closest to the initial cases. Therefore, it is normal that the normal operational cases, or those with very slight interference, were the most frequently evoked. It may have been different if the interviewer had evoked the most critical cases during the discussion.

It would be worthwhile testing, during an evaluation implementing high-risk scenarios, whether pilots produced more high-risk operational cases.

Secondly, the subject evaluated is a special case. Data Link was scheduled only to be used in oceanic zones, with flight control, which excludes high-traffic-density zones or low-altitude flight (potentially risky).

Also, voice communications remain redundant with communications via the digital medium. Pilots do not easily lose contact with the ground. Therefore, the operational field of the data link is not, for the moment, a field which is part of the classical safety context like manoeuvres close to the ground during the takeoff or during approach phase, loss of aircraft control, highly downgraded meteorological situations, total loss of power supplies, and so on.

The safety analysis of the FANS A system currently reports very few cases with potentially dangerous consequences.

Finally, this can be explained by the instructions given to crews during theoretical training. Pilots were informed of the conditions of use of Data Link (cited above) and of our desire to work during this experiment under normal conditions of use.

Concerning the downgraded domain, the contributions of crews differ.

Test pilots more frequently put forward operational cases than line instructor pilots did.

The explanation is undoubtedly due to the fact that they had previously thought about or been exposed to, under experimental conditions, the perturbing cases reported in the discussion. These cases are diversified and involve meteorological conditions, loss of data link connections, loss of reception from air traffic control, erroneous answers from pilots to ground, but there are not enough statements to make definitive conclusions.

Consequently, the great number of operational cases put forward by the pilots cover designer requirements. Thus, designers have acquired a certain amount of knowledge on the conditions of use of current systems and can make assumptions on the conditions of use of future systems, at least in the normal situation range. For the abnormal situation range, it will be necessary to develop some cases proposed by pilots and to carry out other evaluations.

Even if the study contributes to build a model of the different pilots, it will not be possible to create a definitive model of end users and to stop involving them directly in the design as for a new design, the human behaviour being never predefined. Moreover, if we evaluate from an old solution with current pilots, we can expect different results than a few years ago. The problem is not that automation is adequate to the human, but that automation is adequate to the human at one moment, in a specific training and organisation, in a specific environment, with special rules. Perhaps, we have also to question the way to consider the design. Currently, the problematics of design is often presented as a problem of transferability of users’ previous experiences and knowledge in a future system. We think that there is a new way to understand the design process, in order to discover the future hazards due to the innovation.

This is a major reason to involve test pilots and line pilots currently in the loop. Nevertheless we think that the manufacturer has to write the model of the end users when performing an experiment, especially because the design is good at one moment, depending on the model of the end users. It takes part of the hypothesis of design.

We had few pilots, but our goal was not to gather all the knowledge of all pilots. The purpose was to determine whether pilots, with different specificities and different jobs, had different contributions to make and whether these contributions could be useful for final design.

By showing that the line pilots ask for an automation device to assist and to monitor their action, we could think that we have brought a contribution to the pilots’ model.

We should not reinforce the old idea that the pilots have a lot of weaknesses, that they are “intelligent but fragile”, and that pilots, when the task requirements are too high or too complex, need to be replaced. These models are too simplistic, and Hoc and Amalberti (1995) have proposed a dynamic control model of cognition: Regardless of the scope of limitations, the human is self-protected against the risk of losing control by a series of extremely efficient mechanisms. Moreover, humans are using errors to optimise these mechanisms. As a consequence, the design solutions have to be intelligent and to propose some devices that keep the natural human way of defence to the crew.

What we have done is not complete. To continue on the dynamic control, Amalberti (1999) has shown that the pilot could encounter difficulties in the use of automated systems, as some create “negative interactions” when they differ from the crew’s spontaneous behaviour (acquired as an heritage from the old generation of aircraft manoeuvring for example). These different strategies can only come from the lines pilots through the observation of their behaviour when confronted with the machine and the environment.

Paper received: October 2000.

Accepted in modified form: April 2001.