During my school years, I went on an excursion to London, and there I was first introduced to virtual reality (VR) in the game Zone Hunter. The technology immediately hooked me, and I realized that in the future I want to work in this field! Now, for more than 12 years, I have been working on virtual industrial simulators and writing software for VR systems.
I am the founder and president. We offer tools for creating VR applications, such as MiddleVR - middleware that allows 3D applications (for example, based on Unity) to run in any VR system (virtual reality rooms, headsets, etc.). I have a blog about virtual reality that I started long before it became popular, and you can find me on Twitter.
Today, you might think that creating VR experiences couldn't be easier—you just need to coordinate the camera's movement with the Oculus Rift tracker and you're done. Sometimes this is really enough, but for the vast majority of cases this approach will not work.
The main thing in virtual reality is the effect of presence. If a person can't get into the game, then you've done something wrong. You can trick the mind into perceiving what is happening as another world, but it is not as easy as it seems. The effect of presence is a very fragile feeling.
Texts on VR often go too deep into technical aspects. I think the main thing here is what happens to the user's mind. In this article, I want to cover some basic aspects of immersion in the virtual world and speak about the importance of developing applications with an eye to this technology.
Virtual reality in 2013
Virtual reality immerses a person in a three-dimensional environment using special helmets, glasses or other immersion systems. That's why we often use the term iVR (immersive VR) to differentiate ourselves from virtual worlds like Second Life or World of Warcraft. In the early 90s, these technologies attracted everyone's attention, but failed to provide the expected sensations.
However, they continued their development on the serious gaming front, and today have evolved into useful tools used in several areas:
- Training in virtual simulators is an order of magnitude more effective than real practice: you can control the simulated environment with high precision, view replays and fearlessly practice real manipulations in a variety of potentially dangerous situations. Such simulators are used to train surgeons, military personnel, police officers, firefighters, dentists, and even exterior building workers! This allows enterprises to save on expensive materials and avoid various risks, giving a more transparent picture of the trainees' abilities.
- All leading automakers have VR systems for testing the design and ergonomics of products that have not yet seen the light of day, allowing for faster iteration of different variations compared to real mock-ups. This is also used in the production of boats, airplanes, tractors, production lines, factories and even kitchens! Take a look at VR applications and systems from Peugeot or Ford!
- Digital models look very realistic: you can view your future home from all sides or evaluate the urban layout long before construction begins. For an example, watch the demo video from Enodo.
- VR is a useful tool for market research in the retail industry: you can get a live look at the appearance of your store before it was built or moved, follow the movement of visitors and the direction of their gaze. This is helpful when assessing your hardware placement and helps you make sure your design stands out from the rest.
- Virtual reality is a good way to treat phobias: if you have a fear of heights, you can be transported to a simulated cliff and feel your fear. In this case, the therapist’s assistance will be more effective than in real conditions on a real rock. The same applies to fears of flying, spiders, dogs, and public speaking. This is, for example, the work of Stéphane Bouchard at the Cyberpsychology Laboratory at the University of Quebec in Ottawa.
And of course, virtual reality can be used in games! But since the mid-90s there have been very few such games and they were created, as a rule, either in research laboratories or by enthusiasts. Assembling a VR system and programming the game itself requires appropriate skills and equipment. As far as I know, not a single commercial VR game has been released in the last 10 years.
Here is a modest growing list of famous VR games. But this was before the advent of Oculus Rift, but now there will be a holiday on our street every day! And we are already witnessing new virtual fun (guillotine, for example).
Software
3D modeling of photorealistic images
Software packages that allow you to create three-dimensional graphics, that is, simulate virtual reality objects and create images based on these models, are very diverse. In recent years, commercial products such as:
- Autodesk 3ds Max
- Autodesk Maya
- Autodesk Softimage
- Cinema 4D
- Houdini
- Modo
- LightWave 3D
- Caligari Truespace
- as well as the relatively new Rhinoceros 3D, Nevercenter Silo and ZBrush.
- In addition, there are open products that are freely distributed, for example, the Blender package (allows you to create 3D models with subsequent rendering), K-3D and Wings3D.
SketchUp
Visualization of 3D graphics in games and applications
Modeling of parts and mechanisms for production
, (3D displays, 3D screens) - displays that, through stereoscopic or some other effect, create the illusion of real volume in the displayed images.
Head-mounted displays, video glasses
When (not) to add VR to games
First of all, you need to answer the question of whether your game really needs virtual reality. It's like 3D. Not every activity automatically becomes more interesting in 3D, and something that isn't suitable will look even worse in VR.
In that case, where would VR be an appropriate idea?
The goal of virtual reality is to make you feel like you're in another world, whether it's realistic or not. In general, for me the effect of presence is the definition of VR. No sense of presence - no VR!
Obviously, among the genres that are great for VR will be games with a first-person view. Imagine Mirror's Edge or Call of Duty in VR! In some games (Assassin's Creed, Splinter Cell, or Gears of War), the behind-the-scenes view could potentially be converted to an in-eye view so we could feel like we're in the hero's body. I believe we will see a revival of quests and adventure games. Virtual reality will probably appear in completely different games. God simulators? Guitar Hero?
But I believe that the games that will benefit the most from VR are those that tap into emotions.
Horror can be very impressive. You can also remember Heavy Rain. Great game, I was really immersed in it and really worried about it. However, at times it was spoiled by stilted interactions and there is no element of physical presence. And this is where virtual reality can help!
1.1. Types of AR
There are usually four types of augmented reality considered:
- marker-based
- markerless,
- projection-based
- based on superposition (superimposition-based).
In marker-based AR
a specific template is required, a marker, for example, a QR code, on top of which a virtual object is superimposed.
Rice.
1. Marker AR As the name suggests, for markerless
application marker is not needed. A mesh is overlaid on the environment and key points are detected to which the virtual model is snapped.
Rice.
2. Markerless AR Projection-based AR
designed to make factories smarter, safer and more efficient. The projector eliminates the need for paper instructions by creating a digital layer on virtually any work surface.
Projection-based AR
Superimposition-based AR
partially or completely replaces the original representation of an object with an augmented representation of the same object.
Rice. 3. Superimposition-based AR
VR as a new media format
Here I must warn you right away: adding virtual reality to games can be difficult if its support was not originally intended. VR is like radio or TV at an early stage of development: at first only operas were broadcast on the radio, and only performances were shown on TV. Little by little, people began to create content specifically for these new formats. Thus, camera work and editing became the basic concepts for filming.
It will be exactly the same with virtual reality! First will be adaptations of existing games that do not use the full effect of presence. They will be of little use for a new area: even if the display allows for a new degree of immersion, awkward controls and inappropriate gameplay can lead to an adaptation that is inferior to the original.
I'm pleased to see that many indie developers are planning VR in their games from the beginning, that is, doing everything as expected. Why not? VR is an incredible experience! If any of the readers are working in this direction, do not hesitate to write to me, we will be happy to share our opinion on your game.
3D models
The first thing you will need to develop AR/VR is 3D models. There are two options: make them yourself or use ready-made templates. Creating models yourself is a difficult path, but in the long run it can be the best and most cost-effective. In this case, you will have to master special programs. For example, these:
- Blender;
- Autodesk Maya;
- Autodesk 3ds Max.
To create your own models, it is worth mastering 3D scanning. Objects captured by a special scanner in the real world become a virtual three-dimensional model. These devices aren't perfect yet, but they will help you get started:
- David SLS2;
- Da Vinci 1.0 AiO;
- Structure Sense.
You can always find template 3D models on the Internet:
- TurboSquid;
- Free3D;
- CGTrader;
- Sketchfab.
This option is suitable for beginner developers without skills.
To work with videos and photos in 360° format (you will definitely need them in virtual reality), you also need to purchase a suitable camera, like RICOH THETA or Samsung Gear 360. Ready-made versions of panoramic photos and videos can be easily found on Flickr or other stock file hosting services. There are no restrictions on sound effects.
Presence effect
As I said, for me the definition of VR is the effect of presence. Without the feeling that you are in some other place, the game will remain an ordinary interactive 3D environment, and not a true VR environment - even if millions of dollars have been invested into it. Believe me, I've tried a few of these and they're just a disaster.
If there is an effect of presence, the player will demonstrate natural reactions and emotions. On a high cliff you will experience fear of heights (guaranteed). If someone throws a virtual ball at you, you will try to catch it. If a drawn person saves you from certain death, you will smile at him. I'm serious!
Presence is a complex and sensitive topic. At the moment, the most interesting research is being carried out by Mel Slater. In a rather famous article, he divided the sense of presence into two types: cognitive (mind) and perceptual (feelings).
People often say that games, movies, books, and even just a story told by someone give them a sense of presence (how deep are the roots of VR!). This is a cognitive presence - your imagination takes you to other worlds.
Creation
To obtain a three-dimensional image on a plane, the following steps are required:
- Modeling
— creation of a three-dimensional mathematical model of the scene and objects in it;
- Texturing
— assigning raster or procedural textures to model surfaces (also implies setting material properties - transparency, reflections, roughness, etc.);
- Lighting
— installation and configuration of light sources;
- Animation
(in some cases) - giving movement to objects;
- Dynamic Simulation
(in some cases) - automatic calculation of the interaction of particles, hard/soft bodies, etc. with simulated forces of gravity, wind, buoyancy, etc., as well as with each other;
- Rendering
(visualization) - construction of a projection in accordance with the selected physical model;
- output the resulting image tooutput device
- display or printer.
Perceptual presence
The above methods of immersion do not involve perceptual presence, which actually deceives your senses realistically. Vision, hearing, touch, smell, proprioception (from Latin proprius - “own, special” and receptor - “receiving”; from Latin capio, cepi - “accept, perceive”), deep sensitivity - a sense of the position of parts of one’s own body relative to each other, then Google Wikipedia)… Don’t forget that human perception is not perfect: the human brain simplifies many things. Knowing these limitations—the core of VR theory—allows you to create perceptual illusions, like walking in the wrong direction or spaces with impossible geometry.
How can this be achieved?
I find the easiest way to achieve perceptual presence is to track your head movements. Turning the head and turning the camera in a three-dimensional world is the basis for the action-perception cycle.
So you need to be able to make movements, and those movements need to be reflected in the virtual world. Your body gets involved in the process. As Antonio Damasio said: “The mind is in the body, not in the brain alone.”
Features of simulation systems
There are many different types of virtual reality, but they all share the same characteristics, such as the ability to view 3D images. These images appear to be life-size.
They also change as we move through the environment, corresponding to changes in the field of view. The goal is to achieve a smooth connection between head and eye movements and a corresponding response, such as a change in perception. This ensures that the virtual environment is both realistic and enjoyable.
The virtual environment must provide appropriate feedback as the environment is explored. Problems arise when there is a delay between the system's actions and reactions. Or a delay that interferes with their perception. The person realizes that he is in an artificial environment and adjusts his behavior accordingly, which leads to an unnatural, mechanical form of interaction.
The goal is to create a natural, fluid form of communication that leads to a memorable experience.
Minimal VR system
I would recommend head tracking (rotation and translation), at least one hand (pivot and translation) and a joystick with a couple of buttons. From personal experience I can say that such a minimum allows you to cross a certain threshold and the brain accepts another reality much easier.
What this means to me is that OculusRift itself is not (yet) a minimal VR platform. It lacks full head tracking and has no hand tracking at all. I know that all of this can be fixed on my own, using devices like the Razer Hydra. But until we have a comprehensive VR platform, manufacturers will not be able to safely rely on a single hardware standard.
2.1. Types of VR
Standalone VR
works on its own, without the need for additional external equipment (computer, console), for example, Oculus Quest.
All you need is a headset and VR controllers. Oculus Quest
VR on PC
– these are any headsets that require a constant connection to a PC, such as: PC Oculus Rift S, Valve Index, HTC Vive, Pimax and Windows Mixed Reality.
HTC Vive
When they talk about VR for the console
, usually referring to PlayStation VR for PlayStation 4.
Rice. 4. PlayStation VR
Delays
For virtual reality, the number one enemy is delays and lags. If, after turning your head, the image changes after a full second, the brain will not perceive it as reality. Moreover, you may feel nauseous.
John Carmack says that “at latencies of less than 20 milliseconds, the real magic begins—the 3D world appears immutable!”
Some researchers even recommend achieving a delay of less than 4 ms from the moment the movement begins until the required image is displayed on the screen. For a visual representation, I’ll say that when playing at a frame rate of 60 fps, 16 ms pass between frames. Add to this input device latency, which can range from a few milliseconds to over 100 ms in the case of Kinect, and display latency, which can also be low to over 50 ms for consumer VR headsets.
In the case of stereo images, you need to take into account that the game will require processing two images simultaneously. As a developer, there is nothing you can do about input and display lag, but you must ensure the game performs well!
Summary
Virtual reality is the creation of a virtual environment presented to our senses in such a way that we experience it as if we were actually there. Achieving the result uses a variety of technologies and is a technically complex process that must take into account our perception and cognition.
VR technology is becoming cheaper and more widespread. We expect many more innovative applications in the future and perhaps a fundamental way of communicating and working thanks to virtual solutions.
A consistent world doesn't have to be realistic
We figured out that perceptual presence is a realistic deception of the senses. Cognitive - a deception of the mind, but not of the senses - stems from the feeling that you can influence the virtual world and that the events in it actually happen. This means that you have to believe in the “rules” of the simulation. To do this, you need to make sure that your world is not so much realistic as it is coherent and consistent. For example, inconsistency may occur in the fact that a player can take one glass from the table, but cannot take another. The interrupted effect of cognitive presence is very difficult to restore. The player constantly remembers that the world around is not the real one, and it will take time for it to seem real again.
If you set out to create a visually believable environment, the likelihood of presence interruption will be very high. This is due to the fact that the brain will demand from virtual reality what we cannot yet achieve technically: realistic physics, feedback so that the hand does not pass through objects, destructibility of objects, smells, etc. In a world that does not pretend to be realistic, expectations will be lowered initially, so that the effect of presence will be more lasting.
If you have achieved cognitive presence and the player's mind has already been deceived, the events of the simulation will begin to deceive his senses. If an attractive character looks into the eyes of a shy player, his heart rate will increase, he will blush, and so on. People with a fear of public speaking will speak in front of a virtual audience with anxiety in their voice.
That's why I think Verdun 1916-Time Machine is the most immersive app I've ever seen. It deceives many senses at once: sight, smell, touch... But what’s most interesting is that for the best experience, they specifically limited interaction with the world. You can only turn your head because you are a wounded soldier.
Given this strict restriction, it will be very easy to keep the player from interrupting his presence. You can't move your arms, so they won't fall through objects either; you aren't forced to move with unnatural button presses. It has been noticed more than once that people smiled when they saw a virtual friend running up to help!
Application
Car driving simulator
Air traffic controller training simulator
Computer games
Interactive computer games are based on the interaction of the player with the virtual world they create. Many of them are based on the player's identification with the game character, visible or implied.
There is an established opinion that high-quality three-dimensional graphics are required to bring the virtual world of a game closer to reality. If the virtual world of the game is not distinguished by graphic beauty, is schematic and even two-dimensional, the user’s immersion in this world can occur due to exciting gameplay, the characteristics of which are individual for each user.
There is a whole class of simulation games for any type of activity. Flight simulators, car simulators, various kinds of economic and sports simulators are common, the game world of which simulates important physical laws for this type, creating a model close to reality.
Specially equipped simulators and a certain type of slot machines add other sensations to the output of the image and sound of a computer game/simulator, such as tilting a motorcycle or shaking a car seat. Similar professional simulators with corresponding real controls are used for pilot training.
The inconsistency of user interface commands with the actions performed in the game, its complexity can interfere with immersion in the game world. In order to solve this problem, not only a computer keyboard and mouse are used, but also a computer steering wheel with pedals, a target designator in the form of a pistol, and other game controllers.
Theoretical material for students.
3D graphics
(3D (from the English 3 Dimensions - “3 dimensions”) Graphics, Three dimensions of the image) - a section of computer graphics, a set of techniques and tools (both software and hardware) designed for depicting three-dimensional objects.
A three-dimensional image on a plane differs from a two-dimensional one in that it includes the construction of a geometric projection of a three-dimensional model of the scene
onto a plane (for example, a computer screen) using specialized programs (however, with the creation and implementation of 3D displays and 3D printers, 3D graphics does not necessarily involve projection onto a plane). In this case, the model can either correspond to objects from the real world (cars, buildings, hurricane, asteroid) or be completely abstract (projection of a four-dimensional fractal).
3D modeling is the process of creating a three-dimensional model of an object. The task of 3D modeling is to develop a visual three-dimensional image of the desired object. With the help of three-dimensional graphics, you can create an exact copy of a specific object, and develop a new, even unrealistic representation of a previously non-existent object.
Presence measurement
The problem is that it is very difficult to calculate the degree of immersion of the player in the virtual world. There are currently no absolute indicators that reveal this. You can monitor your heart rate or skin conductance levels to monitor your anxiety. But this only works in stressful situations.
However, you can try to evaluate how natural the player's reactions are. We have already mentioned some of them - trying to catch the ball, fear of heights, fear for one’s health when under threat of attack, trying to avoid a collision...
Application
Three-dimensional graphics are actively used to create images on the plane of a screen or a sheet of printed material in science and industry, for example, in design automation systems (CAD; for creating solid elements: buildings, machine parts, mechanisms), architectural visualization (this includes the so-called “ virtual archeology"), in modern medical imaging systems.
The widest use is in many modern computer games, as well as as an element of cinema, television, and printed materials.
3D graphics typically deal with virtual, imaginary, three-dimensional space that is displayed on a flat, two-dimensional surface of a display or piece of paper. Currently, several methods are known for displaying three-dimensional information in volumetric form, although most of them represent volumetric characteristics very conditionally, since they work with a stereo image. From this area we can note stereo glasses, virtual helmets, 3D displays capable of demonstrating a three-dimensional image. Several manufacturers have demonstrated production-ready 3D displays. However, 3D displays still do not allow the creation of a full-fledged physical, tangible copy of the mathematical model created by three-dimensional graphics methods. Rapid prototyping technologies, which have been developing since the 1990s, are closing this gap. It should be noted that rapid prototyping technologies use a representation of the mathematical model of an object in the form of a solid body (voxel model).
VR Gaming Tips
Let's finish with philosophical thoughts and move on to practical advice:
Scale 1 to 1
The scale of the game world must be real. The camera should be positioned at a height corresponding to normal human height (unless, of course, you want to play as a child, as in Among the Sleep). Head movements should not increase (unless you use redirection techniques).
The easiest way to achieve real scale: the unit of length in the virtual world must correspond to the real one - 1 virtual meter is equal to 1 real meter. The field of view should perfectly match the viewing angles of your display. In an ideal virtual world (or a large industrial VR simulator), the distance between the eyes should be calculated with high accuracy. The brain will process all these signals; you may not achieve immersiveness or it will be inconsistent—and users may feel nauseous—if you don't strictly follow this rule.
Check out the hardware
Review the tracking capabilities: Does the device track displacements or only rotations? Is the sensor capable of reporting positioning data and to what extent? What is its accuracy? When is tracking data no longer useful? Familiarize yourself with the field of view: Following the advice about scale, you should not distort the virtual field of view. With a narrow field of view, the user will be forced to shake his head more often and risk missing important events in the periphery. Check the resolution: if the user needs to read the text, it will have to be placed closer to the eyes. Just like with Android development, your game will end up running on a lot of different devices. We can soon expect a war between multiple platforms with different characteristics. Tools like MiddleVR can help you work with different VR systems.
Don't change your point of view
If you're making a first-person game, avoid cutscenes and third-person vehicle controls. This breaks the immersion.
Fight bad habits
Many avid gamers have a bad habit: when they put on a helmet, they will sit upright, as if in front of the TV. Those who play rarely will immediately start looking around. Players need to be weaned off of today's gaming restrictions. In tutorial missions, you need to motivate the player to look around and move his hands. The game should benefit from this. For example, in one of my recent prototypes, enemies appeared on the right, left and above, and it was impossible to move/look around with buttons or the mouse. To win, the user is forced to turn their head and aim with their hand. In another recent prototype of mine, the only interactive object was a candle in a very dark environment. A great way to get the player to explore an area is to take a candle and walk into the darkness, moving and setting some objects on fire while solving puzzles.
Keep players active
In the same Heavy Rain you are almost never taken away from the gameplay. There are a lot of cutscenes that look like non-game ones, but then, suddenly, you are given control. If you don't have a controller in your hands at this time, you won't have time to complete the action. This forces you to always be on your guard.
Another interesting feature of Heavy Rain is that events happen in real time, which means you need to think and act quickly: shoot the guy before he kills my comrade? You are forced to make decisions quickly and, just like in real life, you will never know how correct they were.
Create realistic puzzles
Again an example from Heavy Rain: you need to quickly call one of the hotel rooms. Can you remember her number in 15 seconds? As in life, you have to strain your memory when experiencing severe stress.
Lastly, work as hard as possible on being present.
Creating the effect of presence is not easy. Start small, test often. Work on presence gradually, make small changes and test again. The player's experiences occur in his head! You do not create experiences, but provoke them. The presence effect should be natural. Study user reactions and make changes. Don't throw all your good ideas together just for the sake of a spectacular trailer. Many promising videos turned out to be terrible games.
3D game engines and programming languages
Experience from previous years shows that C# and C/C++ are most often used to develop AR/VR products. This is also true for the major game engines you need to learn:
- Unity – to work with it you will need the C# programming language.
- Unreal Engine - This requires C++ and a node language called Blueprints Visual Scripting.
All VR devices have SDKs for both engines, so you can choose either one to develop your apps. It's not easy to thoroughly learn these game engines: most experts recommend mastering Unity due to the greater number of learning resources. On the other hand, Unreal can offer you better graphics and more power.
Unity has many versions with different prices, but there is also a free one. Unreal is available for free, but requires recurring royalties equal to 5% of gross product revenue (after $1,000,000 per game per calendar quarter). It is recommended to try both options to see which one suits you best.
Google provides SDKs for Android (in Java) and iOS (in Objective-C) when developing for Daydream and Cardboard devices.
Benefits of virtual reality
Modern technologies are quite logically integrated into various areas of business, production and public life. According to statistical calculations and expert forecasts, from 2016 to 2022 the VR market, which includes equipment - glasses, helmets and software - applications and content, will grow 7-10 times. This growth is due to the fact that the creation of special effects and augmented reality worlds has a number of undeniable advantages:
- Interactive interaction with goods and services. With the development of technical capabilities, virtual devices and applications will become widespread like mobile gadgets. Firstly, they allow you to inspect the product and study the service as fully as possible before making a purchase. Secondly, you can immediately see how a product, for example furniture, will look in an apartment setting.
- Demonstration of projects at different stages of implementation. An interactive three-dimensional presentation with the effect of immersion in a simulated space helps present a future project to investors in a favorable light or work out various nuances in an existing one.
- A clear and visual demonstration of the advantages and consumer characteristics of products. It’s better to see it live once than to read or watch a video a hundred times. That is why the next step in business development is advertising in cyberspace.
- Playful and easy to obtain information. Involvement in the learning process. High-resolution screens provide highly realistic simulated world. Thus, the virtual person is immersed in the material or game, which contributes to good memorization and skill development.
- WOW factor. The generated reality, with its effectiveness, makes a vivid impression on the user, which facilitates the rapid assimilation of information or reduces the period of making a purchase decision.
