“iAssyst” – The Flight Trainer in the Eyes

Where pilots look during a maneuver indicates to what extent they have internalized the necessary procedures in the cockpit. For a large-scale study initiated by Swiss International Air Lines, David Rudi of the Swiss Federal Institute of Technology (ETH) Zurich developed a system that precisely records and reproduces where student pilots are looking in the simulator. The airline was able to draw initial conclusions for the training by analyzing the data.
“We don’t actually know where either experienced pilots or beginners are looking in the cockpit.” Recognizing this, pilot Christoph Amman, then head of training at Swiss, laid the foundation for a large-scale study in 2017 to research the question: “Where do pilots look in the cockpit, and can we train this?” Scientists from the Faculty of Geoinformation Engineering at ETH Zurich, headed by Prof. Martin Raubal, as well as NASA and the University of Oregon, conducted research under the leadership of Swiss. The Swiss Federal Office for Civil Aviation, FOCA, covered 40 percent of the costs.

During flight, pilots collect, evaluate and process a flood of visual, acoustic and spatial information. They practice observing and operating the instruments in the correct sequence during maneuvers in the flight simulator. Even experienced flight instructors do not know whether the student pilot is looking in the right direction in the cockpit and has actually internalized the procedure. While they are training and observing students, instructors are also supervising the training program and operating the simulator in the background. They sit behind the trainee with no eye contact. “Trainers can detect the direction the student’s head moves, but not which of the control panels – which are only one or two centimeters in size – they are actually looking at,” says David Rudi from ETH Zurich, whose doctoral thesis was devoted to developing a software program that provides instructors with precise data on their trainees’ line of vision.

In close cooperation with the instructors from Swiss and Lufthansa Aviation Training GmbH, which supported the study with its A320 flight simulator in Zurich, he tried out different recording devices. Eye-tracking glasses were not suitable, as they annoyed the trainees. Instead, the team chose equipment consisting of four cameras and two infrared sensors, positioned in such a way that it did not disturb the pilots, but still provided high-quality recordings and could be removed quickly. To evaluate the data, Rudi developed software tailored to the needs of the trainers, while Swiss instructors worked with researchers from NASA and the University of Oregon to develop a training program for the maneuver.

In order to display the eye tracking data optimally, he programmed many new components for his “Instructor Assistant System” (iAssyst) software. This software links video, audio and simulator recordings and clearly shows the exact movements of the trainee pilots’ eyes via measurable data. It then displays individual flight parameters of the simulator as text, and others as film, for example flight data and navigation displays. A camera records the setting in the simulator. Seven instructors currently working for Swiss evaluated the new program. The feedback was consistently positive. “iAssyst increases instructors’ capacity and makes their observations more precise,” says Swiss pilot Christoph Amman.

“With iAssyst, flight instructors can verify their observations after the training session and fill any observation gaps that occur, for example while taking notes.”

Dr David Rudi, Institute of Cartography and Geoinformation at ETH Zurich, on its development

Tracking pilots’ eyes is not a new concept. The first studies on this were conducted in the 1950s. “What’s new is the context. Now, for the first time in commercial aviation, we have conducted research on a system for instructors together with pilots and trainers in a flight simulator,” Rudi emphasizes.

He believes that the eye-tracking system could be used beyond aviation: “iAssyst can be used wherever people learn how to operate certain systems: controlling power lines on several screens, for example, or in medical training when doctors use simulators to practice surgical procedures.

In order to answer the question “Can anticipatory perception be trained in the cockpit?”, 43 pilots each completed two maneuvers in the flight simulator. Aviation psychologists from NASA and the University of Oregon developed a corresponding training program and examined its effect on the participants’ flight behavior using the data collected. Although the second maneuver differed from the first, the researchers observed some improvement in flight performance among the students who had received the training.

“The study allowed us to prove our thesis that mental models can be trained.”

Christoph Ammann, Flight Captain Swiss International Air Lines Ltd, on the findings of the eye-tracking study

Swiss has already implemented the findings of the study in a new guideline for the visual monitoring of automatic flight control.
Text by Marion Frahm
Photos and Video: © David Rudi, ETH Zurich
Send this to a friend