Guide to virtual worlds: AR and VR

On October 28, Facebook founder Mark Zuckerberg announced the renaming of the corporation to Meta. At the same time, the new company will focus on creating and maintaining a metaverse within which users will be able to fully live, work and have fun. Facebook is not an innovator in the development of virtual reality technologies, but its company pays close attention to this segment. Lenta.ru will tell a brief history of VR, the prospects and advantages of living in a parallel reality, and how new technologies can change our lives.

1.1. Types of AR

There are usually four types of augmented reality considered:

marker-based
markerless,
projection-based
based on superposition (superimposition-based).

In marker-based AR

a specific template is required, a marker, for example, a QR code, on top of which a virtual object is superimposed.

Rice.
1. Marker AR As the name suggests, for markerless

application marker is not needed. A mesh is overlaid on the environment and key points are detected to which the virtual model is snapped.

Rice.
2. Markerless AR Projection-based AR

designed to make factories smarter, safer and more efficient. The projector eliminates the need for paper instructions by creating a digital layer on virtually any work surface.

Projection-based AR

Superimposition-based AR

partially or completely replaces the original representation of an object with an augmented representation of the same object.

Rice. 3. Superimposition-based AR

Application of virtual reality

VR can be used in areas such as:

Education . Today, interactive reality makes it possible to simulate a training environment in those areas and for those activities for which preliminary preparation is necessary and important. As an example, this could be operations, equipment management and other areas.
The science . VR makes it possible to significantly accelerate research in both the atomic and molecular worlds. In the world of computer reality, a person is able to manipulate even atoms as if it were a construction set.
Medicine . As noted, with the help of VR, you can train and educate medical specialists: perform operations, study equipment, and improve professional skills.
Architecture and design . What could be better than showing the customer a model of a new house or any other construction project using such reality? It is this technology that allows you to create these objects in virtual space, in full size, for demonstration, whereas previously manual layouts and imagination were used. This applies not only to construction projects, but also to equipment.
Entertainment . VR is incredibly popular in the gaming environment. Moreover, both games and cultural events and tourism are in demand.

1.2. Software for AR

To create an augmented reality application, you will need a software development kit (SDK). Below we have reviewed the most popular SDKs.

ARKit

ARKit is an SDK for creating augmented reality apps and games for iPhone and iPad. It is characterized by the following features:

Robust face tracking - easily apply facial effects or create facial expressions.
Monitoring the level of illumination of the environment for the correct setting of the light environment of virtual objects.
Detection of 2D objects: horizontal, vertical and inclined planes. That is, for example, tables and walls.

Integration with Unity and Unreal Engine.

Reality Composer

– editor for creating 3D models.

RealityKit

– rendering and animation.

Supported platforms: iOS 11.0 and higher.

Price

: for free.

ARCore

Google responded to ARKit with its development – ARCore.

ARCore uses the phone's camera to identify function points and change their position over time. Visual information is combined with data received from sensors to calculate the position and orientation of the phone in space.
Detects flat and inclined surfaces.
Automatic light control. For example, if you are in a dimly lit room, ARCore will automatically adjust the image.
Linking virtual objects (kitten) to physical objects (table). If you leave the room and return, the kitten will remain in the same place.
Integration with Unity and Unreal Engine.

Supported platforms: Android 7.0 and above, iOS 11 and above.

Price

: for free.

Vuforia

Official site. Peculiarities:

Recognition of text and various types of visual objects (box, cylinder, plane).
Supports the creation of marker and markerless ARs.
3D scanning of the environment.
Detection of multiple objects.
Simulation Play mode, which allows you to walk through or around a 3D model and watch it on your computer.
API: C# for Unity, C++ for iOS, Java for Android.

Supported platforms: Android 4.4.4 and above, iOS 11 and above.

Price

: from 99$/month.

EasyAR

Official site. Characteristics:

Scan the environment and generate a 3D mesh in real time.
Compatible with Android smartphones that ARCore does not support.
Tracking 3D objects.
Real-time recognition and tracking of flat images.
Screen recording in H.264/AAC/MP4 formats.
Unity integration.

Supported platforms: Android, iOS.

Price

: for free.

Lens Studio

Official site. Possibilities:

Tracking face, neck, elbows, hands.

Changing hair color and applying masks to faces.

Delayed publication of “lenses”.

Create 3D models without writing code in the Material Editor.

Supported platforms: Snapchat.

Price

: for free

Spark AR

Official site. Functions:

Making masks for Instagram.

Color filters.

Tracking of face and body parts.

Supported platforms: Instagram.

Price

: for free

Virtual reality is a computer simulation of an alternative reality.

Mozaik3D

Price: free Platform: |

Mozaik3D is an interactive educational application that teaches you geography, history, mathematics, physics, biology, chemistry, technology, fine arts. It has over 1200 3D scenes to explore.

On the app's home page, you'll find the option to filter 3D images by topic or use the search bar to get a specific 3D scene. To activate VR mode, simply tap the VR headset icon in the bottom right corner.

2.1. Types of VR

Standalone VR

works on its own, without the need for additional external equipment (computer, console), for example, Oculus Quest.
All you need is a headset and VR controllers. Oculus Quest
VR on PC

– these are any headsets that require a constant connection to a PC, such as: PC Oculus Rift S, Valve Index, HTC Vive, Pimax and Windows Mixed Reality.
HTC Vive
When they talk about VR for the console

, usually referring to PlayStation VR for PlayStation 4.

Rice. 4. PlayStation VR

Life or cinema?

For the first time, the story of people living in a fictional reality was presented in a blockbuster of Hollywood origin. The story of people limited by the framework of an invented reality looks so convincing that the heroes, and the audience, perceive it as reality.

Afterwards, other films appeared, trying to answer the questions of what is truth and what is fiction. Which half of humanity is right: those who consider the Universe to be fiction, or those who are convinced that we are all part of a big game?

For example, the famous computer technology businessman Elon Musk believes that the ratio of the fictional world to reality is approximately 1,000,000:1.

And the no less famous Ray Kurzweil, a researcher of artificial intelligence, makes the assumption that the Universe is nothing more than a large scientific experiment conducted by one of the young scientists of another Universe.

An interesting fact is that some scientists agree with this possibility. This question even became the subject of discussion at one of the meetings at the New York Museum of Natural History.

2.2. Types of motion tracking

The most common way to track movements is the Inside-Out

with cameras built into the headset. This approach is implemented in the Oculus Rift S, HTC Vive Cosmos and all Windows Mixed Reality headsets.

There is also an Outside-In

with base stations placed in the corners of the room to help the headset and controllers track their exact location (HTC Vive Cosmos Elite, Valve Index).

Rice.
5. Outside-In and Inside-Out Tracking Advanced headsets use simultaneous localization and mapping (SLAM) to track features of the physical environment around a person.

Example of how SLAM works

How can you imitate life?

The basic trick remains the same: reduce detail until you find the best balance between quality and complexity so that our minds can't detect the difference

There are many tricks we can use to reduce the computational power required to simulate the universe to a level we can believe in it. The most obvious one is: don't render something that no one is looking at. You probably know about the Heisenberg uncertainty principle and the observer effect. Modern physics tells us that reality, or rather the smallest particles of which it is composed, depends on the observer. To put it bluntly, shapes don't exist until you look at them. And try to prove the opposite.

The next trick you can use is to create a universe that seems huge and limitless, even if it isn't. By reducing the detail of distant objects, you can save enormous amounts of computing power and generate objects only as they are detected. For example, there is a game called No Man's Sky - it uses procedural generation of worlds as they are discovered, and the number of them is truly endless even in this small computer game.

Finally, one can add basic physical principles that will make it much more difficult or impossible to reach any other planet. Creatures will be tied to their own world. (The speed of light or the exponentially expanding universe, cough cough).

If you combine these tricks with mathematical tricks like repeating patterns and the basics of fractal geometry, you can get a completely working heuristic model of the universe that will seem almost infinite and limitless. Yet this does not explain why the virtual universe theory has become so popular. Why are we, with a high degree of probability, in such a world?

If you are interested in science and technology news, subscribe to us on Google News and Yandex.Zen so as not to miss new materials!

2.3. Basic terms and concepts

Field of View (FOV)

. In the context of VR headsets, field of view refers to everything you can see in the virtual world while using the headset.

Rice.
6. Field of view (FOV) Degrees of freedom (DoF)

. The more degrees of freedom, the more physical movements the headset will track. 3DoF (3 degrees of freedom) tracks only head movement: roll, pitch, yaw, but not position in space along coordinates (x, y, z). Modern headsets use 6DoF, which tracks both head movements and head coordinates.

Rice.
7. Degrees of freedom 3DoF and 6DoF Frames Per Second (FPS)

– number of frames per second.

Screen Resolution

– the number of pixels displayed on the screen.

Spatial Audio (3D audio)

– spatial sound (used to create sound that comes from a specific point in the virtual world).

Rice. 8. Spatial Audio (3D audio)

It all starts with games

The reality existing in parallel with the usual way of life was described in numerous science fiction novels and films of the 20th century. However, attempts to look at familiar things from the other side have been made before. In 1838, physicist Charles Wheatstone described the principle of indicating the perception of depth and three-dimensional structure - stereopsis. Based on his discovery, the scientist created the first stereoscope - a device designed to perceive three-dimensional images. Wheatstone used a pair of mirrors positioned at 45 degrees relative to the eyes of the beholder. The physicist has proven that the brain combines images of objects obtained from different points, as a result of which the object appears three-dimensional.

Charles Wheatstone Stereoscope

Image: Getty Images

In 1935, science fiction writer Stanley Weinbaum published the novel “Pygmalion’s Glasses,” in which he talks about a certain device that made it possible to perceive images through eye contact, as well as touch fictional objects, feel their taste and smell. Essentially Weinbaum is describing something similar to modern virtual reality headsets. The idea that people can communicate with each other without being physically in the same location was introduced in 1975 by artist Myron Kruger. He founded the Videoplace laboratory, which placed the user in an interactive environment. The person’s movements were recorded by a camera located behind his back and immediately transferred to the projected silhouette. This gave the observer a sense of presence even in the absence of direct tactile feedback.

Back in 1957, inventor Morton Heilig, who is called the father of virtual reality, created the world's first VR simulator, which he called sensorama. The device was a movable chair with a screen, as well as speakers, fans, and odor emitters. Using the device, the user could immerse himself in a certain virtual environment: for example, find himself behind the wheel of a motorcycle speeding along the road. Heilig's ideas influenced the development of VR and the emergence of the virtual video game segment. In the future, there will be many simulators based on maximum immersion of the gamer into the gameplay. For example, in the 1970s, General Electric introduced a fighter jet flight simulator that was constructed of three screens surrounding a virtual pilot's cockpit. A little later, the VPL company created DataGlove virtual reality gloves, which read the movement of the fingers and transferred them to the game. In the 1990s, Nintendo released the Virtual Boy console, which is considered the first console capable of playing 3D graphics. The VR headset was based on stereoscopic LED eyepieces that used sensors to track the user's eyes.

In the 2000s, it seemed that the development of technology had stopped until relatively inexpensive and advanced virtual reality headsets began to appear on the market. The Oculus Rift VR glasses, which projected images using LCD screens, tracked head turns and had several degrees of freedom, were considered a breakthrough in the industry. Against the backdrop of interest in the industry, similar products are beginning to appear, such as PlayStation VR, HTC Vive, Samsung Gear VR and other devices that allow you to play games in fully immersive mode.

IT giants, who paid attention to the promising industry, began to offer their innovations in the field of VR. Facebook, which did not have any serious products as of 2014, acquired Oculus for two billion dollars. It is believed that around this time, Mark Zuckerberg became interested in virtual reality, which he would later propose to insert into the concept of the metaverse.

2.4. VR via web browser

WebGL

– cross-platform API for 3D graphics in the browser. It executes as an HTML5 element and is therefore a full part of the browser's Document Object Model (DOM API). Can be used with programming languages that support the DOM API: JavaScript, Rust, Java, Kotlin, etc.

There are various frameworks available to make working with WebGL easier. All of them are distributed under a free license (MIT, Apache 2 or BSD), written in JavaScript and have a similar set of functions: working with geometry, material, light, animation, cameras, shaders, text and 3D audio. Here are some of them:

three.js – website, GitHub, MIT.

Babylon.js – website, GitHub, Apache 2.

A-Frame – website, GitHub, MIT.

PlayCanvas – website, GitHub, Apache 2.

React 360 – website, GitHub, BSD.

AR.js - GitHub, MIT.

Interpretations of quantum mechanics

Instead, many other theories have been put forward. The most famous of them was invented in 1927 in the city of Copenhagen.

Copenhagen interpretation

Scientists Niels Bohr and Werner Heisenberg suggested that elementary particles are both waves and particles at the same time.

Niels Bohr and Werner Heisenberg

So, in order to measure an electron, that is, to carry out observations on it, it must be struck against the quanta of a measuring device. And it is precisely because of this impact that the wave functions of the electron “collapse”, and it becomes only a particle. Thus, the observer himself does not influence the particle with his observation - only the quanta of the measuring device influence.

Since this explanation of quantum mechanics was formulated in the city of Copenhagen, it was called the Copenhagen interpretation.

It's funny, but if this interpretation is correct, then it still does not disprove the matrix hypothesis, since it can be adjusted to this explanation.

For example, a photon program can propagate through a network as a wave, and then restart when a node is overloaded, becoming a particle. This explains both quantum waves and the collapse of the wave function.

Many-Worlds Interpretation

After the Copenhagen interpretation, the second most popular explanation for the reasons for the strange behavior of microparticles in the double-slit experiment was the Many-Worlds interpretation.

Its essence lies in the fact that perhaps there are, as it were, parallel universes, in each of which the same laws of nature apply.

And that with each act of measuring a quantum object, the observer seems to split into several versions. Each of these versions “sees” its measurement result and acts in accordance with it in its own universe.

What a strange explanation!

Which of these interpretations to believe in more is up to you to decide.

For example, a survey of scientists conducted in 1997 at a symposium under the auspices of UMBC (University of Maryland, Baltimore County - University of Maryland in Baltimore) showed that the majority of physicists do not believe either the Copenhagen or many-worlds interpretation. The votes were distributed as follows:

13 people voted for the Copenhagen interpretation;
8 – for Many Worlds;
several scientists - for other, less popular interpretations;
18 physicists opposed all proposed interpretations at that time.

Until now, the debate about the correct interpretation of quantum mechanics continues around the world. It is conducted between university scientists, at conferences and even in bars and cafes.

Meanwhile, in 2006, technological developments made it possible for the first time to conduct an even more sophisticated version of the double-slit experiment.

It's called a delayed choice experiment.

2.5. Software

Blender

Blender is a professional and free software designed for creating 3D computer graphics. It includes tools for modeling, animation and rendering.

Unity

Unity is the most common VR development tool. With its help, you can import models created in Blender, configure interaction with these models, as well as their physics. For scripting, C# and UnityScript, reminiscent of JavaScript, are used. This engine can be used to create applications for WebGL, Oculus Rift and HTC Vive. There is also integration with ARCore, ARKit.

Price

: free and does not require deductions from profits.

Limitation

: the volume of attracted investments is no more than 100 thousand dollars over the last 12 months.

Unreal Engine

Unity's main competitor is Unreal Engine. It allows you to write in C++ and Blueprints (visual programming). Supported platforms: AR (Android, iOS), VR (Google VR, Oculus, Gear VR, SteamVR, Windows MR)

Price

: for free.

Limitation

: 5% of sales revenue if quarterly revenue exceeds $3 thousand.