Taxonomy of Assembly Simulation Systems in VR and AR

What are VR and AR assembly simulation systems?

In the context of this article VR (virtual reality) and AR (augmented reality) based assembly simulation systems are defined as:

A system with “the capability to assemble virtual representations of physical models through simulating realistic environment behaviour and part interaction to reduce the need for physical assembly prototyping” (Seth, Vance et al. 2011) in an immersive fully (in the case of VR) or semi (in the case of AR) computer generated environment.

Reasons for the advent of assembly simulation systems

A major challenge in industry is reducing the cost in time and money of product assembly. The assembly process has been known to constitute a large part of the cost of a product. Therefore good assembly planning is an essential part the product design lifecycle and should take into consideration time, cost, ergonomics and operator safety.

While CAAP (computer aided assembly planning) is itself an area that focuses on generating automated assembly sequences based on product geometry, in order for it to be most effective it requires human experience and intervention. Many factors to do with ergonomics and operator safety cannot be effectively incorporated into such a system. Also the number of possible sequences increases exponentially with the number of parts and it may be difficult to arrive and at the optimum solution without human intervention.

VR technology allows the user to interact directly with virtual design models and allow them to simulate the assembly process without the need for physical prototyping. Therefore designers are able to perform assembly/disassembly evaluations of the further upstream in the process allowing changes earlier on minimising their impact on the product design lifecycle. Furthermore these systems can be used for training assembly operators, guiding maintenance operations and in ergonomics assessments.

Summary of different types of assembly simulation systems

Assembly simulation systems described in research thus far can be categorised by many different characteristics; the assembly environment, display methods, human-computer interaction methods and most importantly part-part interaction methods. Many of the systems encountered in research exhibit some combination of these characteristics, and each methods have their advantages and limitations. The characteristics will be discussed in detail below.

Types of part-part interaction methods

Part-part interaction methods refer to the ways that the assembly process are modelled:

Constraint based systems

When two parts interact in a constraint based system, their final position in the assembly is determined by the constraints imposed on them. Within this category, constraint based systems may be further divided into positional constraint and geometric constraint systems.

In positional constraint systems, the assembly model is pre-assembled with a traditional CAD system prior to its input to the simulation application, or the final position and orientation of the components are pre-defined. The user is therefore not fully free to decide to assembly the components in any way they see fit as the parts will eventually only snap into place as they were initially defined.

Geometric constraint systems calculate and solve constraints between geometric features in the parts in real time and apply these constraints based on proximity. This allows the user to assemble the parts in most ways the user sees fit, as long as the constraints they are trying to impose on the parts are recognised by the system as valid.

It is also common in these systems for the human-part interface to also be done through gesture recognition rather than physical simulation of hand/tool force.

Physics based systems

Physics based systems attempt to simulate real world physics when there is interaction between parts or between part and human. Physics simulated include friction, collision forces and deformation. Many physics based systems also use haptic devices as the method of human-computer interface, this allows components of force and even tactile feedback to be given to the user.

Assembly of components is done here as it would in real life and in an ideal case, the constraints that are applied on parts hold because they are physical.

Hybrid systems

Hybrid systems are a combination of constraint and physics based systems. Currently physics based systems are too computationally intensive to be used at a precision for it to be realistic enough. Therefore some constraint based snapping is required to accurately assemble the part based on interacting features. Also physics simulation is required to make the assembly simulation realistic and intuitive for the user. Therefore hybrid systems are currently the most popular method used in research.

Types of human-computer interaction methods

The method of human-computer interface (HCI) used in assembly simulation is extremely important as the assembly interactions need to be as realistic as possible for an accurate simulation.

Mouse and keyboard

The standard HCI, although commonly used everywhere is not an intuitive and realistic interface for assembly simulation.

Wearable hand tracking

The most common examples of this are Data Gloves and CyberGrasp. Other lesser known systems are Sixth Sense or Rutger’s Master II, exoskeleton. These are simply wired glove devices with sensors that relay the hand pose information to the application which can then be used to recognise gestures or used for physics calculations of hand or finger force.

Haptic devices

Haptic devices in this case are characterised as devices that provide force feedback. They may be wearable glove devices with an external rig such as the CyberForce or Master II discussed above. They may also be devices such as the PHANToM.

Computer vision tracking

Computer vision based tracking methods use frame by frame analysis of a video stream from a single or multiple cameras to track an object in the scene. In the context of assembly simulation interface computer vision tracking methods can be divided into two main categories: marker based and marker less methods.

Marker based methods use markers to identify and track a tool or hand. Instances of this have implemented a marker on a side face of a tool, or on the tips of two fingers to track these points. From here the pose of the tool or hand may be estimated and gesture recognition or physics based interactions can be implemented.

Marker less method similarly uses CV to track bare hands or tools without the use of markers.

This can be implemented with any camera and platforms such as ARToolKit and Vuforia provide functions for marker based tracking. Special cameras rigs with stereo vision and infrared capabilities have been used to further optimise CV based tracking interfaces, such as in the Leap Motion controller.

Types of Assembly environments

The surrounding background behind the assembly model also has an impact on the simulation.

Virtual reality

An artificially constructed environment or background. Suitable simply for virtual prototyping and virtual part-part interactions. In this case a virtual version of the hand or tool must be made to represent the real tool, and any real objects required in the environment must be pre-constructed.

Augmented reality

In augmented reality assembly environments, the real scene is overlaid with virtual parts and interface. AR assembly environments are suitable for mixed prototyping applications where assembling virtual parts on top of real parts is required.

Types of Display methods

2D monitors

This is the standard type of display for most human-computer interactions. However due to the difficulty in judging depth in a 2D screen, it makes it a poor choice of interface for 3D assembly simulation. The 2D screen also breaks the users immersion in the virtual environment, in the case of a purely VR assembly environment.

Projected CAVE configuration

Cave automatic virtual environment (CAVE) is simply a small room with the virtual reality projected onto its walls, fully surrounding the user in a virtual environment. The user also wears stereoscopic viewing glasses in a typical CVAE configuration which gives the user the impression of a 3D image.

Head mounted display

A head mounted display (HMD) is a wearable device that typically has two lenses that show a stereoscopic image, giving the wearer the impression of a 3D environment. This commercial products for this type of device now exist (Oculus, HoloLens). Mobile phones are also capable of being used in head mounted configuration for stereo VR and AR applications (Google cardboard, Gear VR).

Areas of application of assembly simulation

The intended application of the assembly simulation system largely influences their interaction capabilities and features.


Assembly validation system’s main goal is to confirm the feasible assembly of a sequence. This type of system will mostly be used in the design or prototyping phase of the product lifecycle. Such a system will only require a VR environment and hybrid part to part interaction. It will also require easily useable HCI such as a HMD for quick tests.

Training and guidance

Training and guidance systems are more concerned with showing the user how to assemble a product by providing instructions and feedback to the user based on their assembly interactions. This type of system may be used in maintenance scenarios as well as in an assembly line. As a result an AR environment may be required for assembling virtual components on real components. It will also require an easily usable HCI method that is also possibly portable, such as a mobile device in HMD configuration.

Ergonomics assessment

These types of systems focus on the human aspect of product assembly. They will involve the capture of data such as human pose and force exerted. This type of system will require physics based part to part interaction with a haptic device as HCI for force feedback.

The future of assembly simulation systems

As technology continues to advance we can perhaps look at more powerful mobile computing with more sensors (such as infrared cameras, depth sensors) capable of delivering a fully physics based assembly simulation system in mobile AR, with the ability to perform virtual to real part assembly. Wearable haptics will also play an important part in providing force feedback for virtual interactions.


Nee, A. Y. C., et al. (2012). “Augmented reality applications in design and manufacturing.” CIRP Annals 61(2): 657-679.

Seth, A., et al. (2011). “Virtual reality for assembly methods prototyping: a review.” Virtual Reality 15(1): 5-20.


Leave a Reply