How you can become an AR/VR developer
The Augmented Reality (AR) and Virtual Reality (VR) industries are still in its infancy, however, this is why now is the perfect time to learn how to create content for these technologies. They need it to make the jump to the next level of adoption. Luckily, from a developer standpoint, the skills needed to get into AR and VR are very similar and the barriers to entry are still relatively low. This article will give you an overview of how to get started.
By Estaban Herrera at Pusher
No matter whether you’re a newbie who is starting with programming or a veteran with years of experience in the field, the main skills you’ll need to become an AR/VR developer are in the 3D area. In fact, it’s not very different from 3D games development.
Why? Because AR and VR are about creating immersive worlds or environments that can be interacted with in three dimensions, like in real life.
So depending on how deep you want to go, you may have to learn about 3D modeling and/or scanning, 3D games engines, 360° photos and videos, maybe a little bit of math and geometry, programming languages like C/C++/C# and software development kits (SDKs), and how to design experiences for users in 3D.
So let’s take a deeper look at each of the topics you need to address to become an AR/VR developer, focusing first on VR.
Augmented/Virtual reality (and 3D development in general) demands high-end hardware. However, as the price of hardware is constantly going down, this may not represent a big problem as it did a few years ago.
If we take a look at the requirements of the two most popular VR platforms (HTC Vive and Oculus Rift), we’ll see they are basically the same. At the time of this writing, these are the recommended hardware specifications:
Does it mean that if you have an i3 processor or a GTX 1050 card you won’t be able to do any development work?
It depends.
The above are the recommended specifications to have a smooth experience, so for some projects, maybe a little less would do the work, and for others, maybe you’ll have to compensate for the lack of power in one area with an increase of power in another one (for example, by pairing an i3 processor with a GTX 1070).
There are minimum requirements and tools to make sure your hardware is compatible (here and here), but the suggestion is to try to aim for the higher end and use desktop PCs to have an easy (and cheaper) upgrade path.
In this area, we have many options. Let’s categorize the most popular by their degree of freedom (DOF), which refers to the ways in which an object can move. We have two options: three and six DOF.
Three DOF means that you will be able to interact with the virtual world in three dimensions (in an X, Y, Z coordinate system) by moving your head using a head-mounted display (HMD), however, you wouldn’t be able to move forward or backward.
With six DOF, you can move forward/back, up/down, and left/right, so you now have three more types of movements, hence the name. You can learn more about degrees of freedom in this article.
The following devices support three DOF:
While the following can support six DOF:
Of course, each device uses different SDKs, programming languages, and has different constraints, but you’ll find that they have some things in common:
At this point, and depending on what you want to accomplish, you may want to do some research and choose one or more device(s) to develop on. Of course, you can do more things with desktop headsets, but you may want to start with something simple, and transfer your skills from one device to another later. It’s a matter of learning the right tools for the job.
Now let’s talk about the most commonly used software to develop VR content.
A survey published in The State of the Developer Nation Q1 2017 report puts C# and C/C++ as the most used programming languages for AR/VR development. This is not a coincidence. The most popular game engines you’ll need to learn use them:
The good news is that all VR devices have SDKs available for both engines so you can use only one of them to develop AR/VR applications and target more than one device.
The bad news is that the learning curve is relatively steep for both. When in doubt, most people recommend Unity because it’s easier to learn and more resources are available. However, Unreal can offer you better graphics and more power.
Another thing to take into account is that Unity has many versions with different costs, but there’s a personal version that is free to use. Unreal is free to use, with a 5% royalty on gross product revenue after the first $3,000 USD per game per calendar quarter from commercial products.
The recommendation is to try both of these engines to see which one suits you best.
In addition, it’s worth mentioning that Google provides SDKs for Android (in Java) and iOS (in Objective-C) to develop for Daydream and Cardboard devices.
The first things you’ll need for AR/VR development are art assets, especially 3D models. You have two options here: make them yourself or use models made by someone else.
Making 3D models by yourself is the most difficult option, but in the long run it may be the best (and most cost-effective). If you choose this path, you’ll have to learn to use programs like:
A technology that can help you create your own models is 3D scanning. Basically, things captured by a 3D scanner in the real world become a virtual 3D model. They may not be perfect yet, but they can help you get started, and there a lot of options with a wide range of price points. Some of these are:
Otherwise, you’ll want to get 3D models from places like:
This option is fine when you’re starting and you don’t have the skills, but sooner or later, you’ll need to make modifications to get exactly what you want, so the suggestion is to combine the two options while you learn to make your own models.
Something similar happens with 360° videos and photos, used in Virtual Reality to provide an immersive experience.
You can buy a 360 camera like the RICOH THETA or the Samsung Gear 360 (only compatible with Samsung devices) to create your own media or use sites like Flickr or other sites for royalty-free stock images and videos.
In the area of audio, we have it easier because there are no special requirements; we can use the same sound effects used in traditional games and videos from royalty-free audio sites like FreeSound, and audio recording and editing software like Audacity or Adobe Audition.
Maybe you have a web development background and you’re wondering if your knowledge of JavaScript can be of any help.
Luckily, WebVR is an open standard with a JavaScript API that makes it possible to experience VR in your browser.
It works with most VR devices, but you don’t need any of these or plugins as any compatible browser (even in your Android/iOS phone) will do the job. You can check for the latest browser support at WebVR Rocks and find more information on the Mozilla VR site.
Although knowledge of 3D terminology and modeling still applies, if you know JavaScript, it can be a good choice to start with WebVR instead of a VR SDK, Unity, or Unreal.
In this area, there are two frameworks worth mentioning:
As both are based on Three.js, the most popular Javascript library for using 3D graphics in a web browser, knowledge of this library will give you a strong foundation.
Unity has the option of writing scripts in a language similar to JavaScript called UnityScript, but it’s not the same as the JavaScript you’ll use to develop for the web, and C# is the recommended language for Unity.
We have been focusing on Virtual Reality, but Augmented Reality is a different concept.
Virtual Reality is about creating a virtual and immersive environment to replace the real world. Augmented Reality adds layers of data to the real world instead of replacing it. For example, it can translate the words you see in another language, or it can make you see monsters when you walk near some place. In contrast, Virtual Reality can take you to another country or a land full of monsters.
When you join both concepts, you get what is called mixed reality, where you interact with both the physical and virtual environment.
However, from a development standpoint, VR and AR are pretty similar. In fact, you can use Unity and Unreal (with the help of some plugins) to develop AR content. For example, a simple AR app will recognize an object and present a 3D model that you could manipulate as if it were real, so the skills needed for VR are applicable to AR also.
One of the most popular tools for developing AR is Vuforia, which is available for Unity, Android, and iOS, provides a lot of features, and support for many devices, including Tango phones. But Tango is gone now. Curious what happend? Take a look at this article by Daily Wireless.
Talking about devices, Augmented Reality can be used with head-mounted displays, but also with other devices like smartphones, glasses, and specially-made visors, perhaps being the most popular Microsoft HoloLens(although more inclined towards mixed reality).
The AR/VR industry is new and therefore there are not many best practices yet for developing this kind of experiences, but we should take for granted that they are different from traditional 2D apps. For example, how do you handle input? A keyboard in a virtual world may not be the best choice in some situations.
On the other hand, one of the biggest problems about VR is simulator/motion sickness. People can get sick either by lagging, unnatural movements, and mismatches between physical and visual motion cues, among others.
Here are some resources on the topic:
