Just like in Star Trek
Biologists are making it possible to carry out new experiments in virtual reality - A team of researchers led by Professor Iain Couzin at the University of Konstanz contributes to solving a fundamental problem in current research on collective behaviour
How do we orient ourselves when we experience new environments? How do we use street signs or landmarks, for example, in order to estimate how far we have travelled? How do individuals become leaders or followers within groups when moving together? To uncover the basic principles behind spatial cognition and collective behaviour, neuroscientists with the support of Professor Iain Couzin from the University of Konstanz and the Max Planck Institute for Ornithology in Radolfzell, are striving to answer such questions. An international team has recreated a kind of holodeck, a staging environment which can simulate any desired virtual environment or habitat, in order to create new opportunities for studying animal behaviour. Animals perceived three-dimensional simulated objects in this environment as real and changed their behaviour in different visual environments. Their research results have been published in the current edition of the scientific journal Nature Methods.
"Until now, we have envied an invention from the world of science fiction: a holodeck like they have in Star Trek. In the real world, we would not be able to carry out such experiments in which we can artificially decouple an animal's movement from its perception,” says the principal investigator from the University of Freiburg, Professor Andrew Straw. Animals and humans use every sensation they experience to update their mental map. However, every movement is inextricably linked to their sensations. Thus, the researchers had to decouple the movement process from the sensation process in order to understand how the brain processes different information.
The group built a flexible system for mice, flies and fish – three species commonly used in neurobiological and behavioural research. "We created an immersive 3-D virtual reality in which the animals could move freely," explains Professor Straw. “We wanted our visual scenery to tie in naturally with the animal's own action-perception cycle." The visual landscapes included, for example, vertical pillars, various plants and other organisms. Multiple high-speed cameras tracked and recorded the precise 3-D position of the animal. A computer program registered its every movement within milliseconds, allowing for an updated and reactive, virtual environment to be created.
The researchers created experiments using visual stimuli to change and control flies' direction of flight, to test if mice are afraid of virtual heights and to make fish move virtually between two different worlds, thereby changing their behaviour depending on the visual environment.
Furthermore, the team was able to solve a fundamental problem in collective behaviour research: How individuals become leaders or followers within groups. Up until now, it has been impossible to directly manipulate interaction between multiple individuals. Professor Iain Couzin and postdoctoral researcher Renaud Bastien from the University of Konstanz and the Max Planck Institute for Ornithology developed a photo-realistic virtual fish that could interact, dynamically, with real fish. “A key challenge in the study of such inter-individual interactions is that it is often difficult, or impossible, to establish how social feedback among organisms gives rise to group-level properties,” says Professor Couzin. “The structure, motion and body posture changes of our virtual fish are extremely realistic. This allowed us, for the first time, to have virtual fish swim freely in the tank with real individuals”.
The Konstanz team was thus able to determine how individuals establish leadership roles within groups, and how individuals can best influence others. “Immersive, reactive virtual reality has the potential to become an invaluable tool in the study of social interactions and collective behaviour,” explained Professor Couzin. “Next, we will connect five virtual reality systems together in what we call ‘The Matrix experiment,’ allowing multiple real fish to inhabit the same virtual world. This will give us unique insights into the social world of these animals and help us reveal further the principles that underlie collective behaviour”.
Original publication:
John R Stowers, Maximilian Hofbauer, Renaud Bastien, Johannes Griessner, Peter Higgins, Sarfarazhussain Farooqui, Ruth M Fischer, Karin Nowikovsky, Wulf Haubensak, Iain D Couzin, Kristin Tessmar-Raible, Andrew D Straw: Virtual reality for freely moving animals. In: Nature Methods. DOI: 10.1038/nmeth.4399
Facts:
- Participating research teams: Professor Iain Couzin, University of Konstanz and Max Planck Institute for Ornithology; Professor Andrew Straw, University of Freiburg; Professor Kristin Tessmar-Raible, Max F. Perutz Laboratories, Vienna (Austria); Dr Wulf Haubensak, Research Institute of Molecular Pathology, Vienna (Austria); Professor Karin Nowikovsk, Medical University of Vienna (Austria)
- Funding for Professor Couzin and his research team was provided by: National Science Foundation (NSF), Office of Naval Research (ONR), Army Research Office (ARO), Max Planck Institute (MPI) for Ornithology, University of Konstanz
- Published in Nature Methods, August 2017
Note to the editors:
A selection of images is available for download here: strawlab.org/freemovr-press