⟵ back to the overview

Mixed Up Realities

the need for a taxonomy

by Philipp Maruhn & Lorenz Prasch

Mixed Reality, Extended Reality, Virtual Reality and Augmented Reality

what is it anyway and why does Human Factors Research care?

Already back in 1994, Milgram & Kishino Paul Milgram & Fumio Kishino (1994) A taxonomy of mixed reality visual displays Google Scholar realized that some clarification was in order. And order they provided (quite literally). Between the diametrically opposed extremes of completely virtual worlds on the one hand, and good old reality on the other, they describe a virtuality continuum (VC). Here they place the broad spectrum of combinations of virtual and real content.

They call this area mixed reality and further divide it into augmented reality and augmented virtuality. Augmented reality is close to reality, but individual virtual elements are displayed. In augmented virtuality, the user is situated in a virtual environment, which is enriched with individual real objects. These combinations of real and synthetic content offer a variety of opportunities for human factors research across many application domains. But let's take a look at the four distinct levels described:

A real-world test environment is the gold standard in many cases. The behavior of users can be studied in reality without having to consider any confounding factors such as the wearing of VR glasses, the display resolution, or the degree of realism. However, the implementation of a real test environment is often difficult or even impossible. For example, when safety is of particular concern (as it is for example in road or air traffic), an ethical experimentation with human subjects is often not feasible.

In contrast to completely real environments, AR offers the possibility to examine and design interactions with products even before a physical prototype exists. This is particularly interesting against the background of human centered design, where it is important to involve users as early as possible in the development process. Augmented reality can take on many more forms than Hollywood sci-fi productions would have you believe. In fact, we encounter AR almost every day, for example in the form of Snapchat video filters or during sports broadcasts.

Augmented virtuality already has a long tradition in the field of human factors engineering (you can find an overview here). We often know such setups as simulators. Starting with the first seat boxes and flight simulators in aviation, they are increasingly used in other areas, such as in traffic research as driving simulators. The user is usually in a real cockpit (or an approximation) around which the virtual world is displayed. In contrast to VR, the advantage is that the users can naturally perceive their own bodies.

Sometimes such setups are also called caves (Cave Automatic Virtual Environment). Depending on the size of the CAVE, it is possible, for example, for pedestrians to cross a virtual street. With increasing size, there is a broader range of applications, but the associated costs also rise. Large rooms, projection surfaces, projection technology and tracking systems are needed, with the associated acquisition and maintenance costs.

In VR, the display device is directly in front of the viewer's eyes. It is not called head-mounted display for nothing. However, this also means that the viewer is completely cut off visually from the outside world. Since the viewer's own body can no longer be seen, it is usually displayed in the form of a virtual representation, a so-called avatar. Compared to conventional simulators (AV), the technical simplicity of these setups is particularly interesting. In most cases, only a room, a PC and a VR system are required to investigate a variety of human factors issues. Even simpler systems can be realized with a smartphone, two plastic lenses and a bit of cardboard (for example Maruhn 2021 Philipp Maruhn (2021) VR Pedestrian Simulator Studies at Home: Comparing Google Cardboards to Simulators in the Lab and Reality Frontiers in Virtual Reality ).

Closing thoughts

now what?

Milgram & Kishino Paul Milgram & Fumio Kishino (1994) A taxonomy of mixed reality visual displays Google Scholar created a taxonomy as early as 1994 into which most of today's systems can be classified. At this point, it should be mentioned that the taxonomy has additional subscales for classification, but these are rarely used, if at all. Still, the widespread use (3864 citations and counting) of the classification system speaks for itself.

The entire framework has one major limitation though, which the authors themselves already mention: it is only a classification of optical systems. Considering developments in the past, it is reasonable to assume, however, that additional modalities will become increasingly important. This means that more and more senses will be fed virtual content. For example, haptic input and output devices are currently being developed intensively. And even for taste-simulation there are first prototypes. Skarbez et al. (2021) Richard Skarbez, Missie Smith & Mary Whitton (2021) Revisiting Milgram and Kishino's Reality-Virtuality Continuum Frontiers in Virtual Reality even argue that it is only true VR when all senses are immersed in a fully synthetic world, and add matrix-like VR to the continuum after VR.

So what is extended reality (XR)? Well, that's actually a good question. We would argue that everything that is to the right of reality on the continuum extends it in some form. Therefore, this is what should be considered extended reality.

⟵ back to the overview