Scope & Architectures Project Roadmap Features GitHub Repos Releases Tutorials Requirements
Features
The MPEG-I Scene Description reference architecture and features are described in the Technical Resources. This page describes those implemented in the 5G-MAG Reference Tools.
Check below the:
Features implemented in the XR Unity Player
The XR Player takes 3D scenes in glTF format, supporting extensions that enable extended reality use cases. These extensions enable features such as XR anchoring, interactivity behaviors, and media pipelines.
Media pipelines
Support for media sources (eg. mp4, dash, rtp,…) exposing media buffers to the presentation engine through the MPEG_media, MPEG_accessor_timer, MPEG_buffer_circular glTF extensions.
The media pipelines APIs are designed to fetch and decode timed media such as video textures, audio sources, geometry streams,…
| Reference scene | XR Phone | XR HMD | Test content |
|---|---|---|---|
| Sample scene with media pipelines | ☑ | ☑ | studio_apartment/studio_apartment.gltf |
Video texture
Supports video textures buffers through the MPEG_texture_video glTF video extension. Video decoding is implemented by media pipelines.
| Reference scene | XR Phone | XR HMD | Test content |
|---|---|---|---|
| Sample scene with video texture | ☑ | ☑ | studio_apartment/studio_apartment.gltf |
Spatial audio
Supports audio sources positionned in 3D through the MPEG_audio_spatial.
For each audio source the extension specifies attenuation parameters controling the audio source loudness as a function of the viewer’s distance.
| Reference scene | XR Phone | XR HMD | Test content |
|---|---|---|---|
| Sample scene with spatial audio source | ☑ | ☑ | studio_apartment/studio_apartment.gltf |
Anchoring
The XR Player supports XR anchoring using the MPEG_anchor glTF extension which enables anchoring nodes and scenes to features (Trackable) tracked by the XR device. In augmented reality applications, anchored nodes are composited with the XR device’s environment.
The XR player leverages Unity’s ARFoundation to support both handled mobile devices such as smartphones and head mounted devices (HMDs).
| Trackable type | XR Phone | XR HMD | Test content |
|---|---|---|---|
| TRACKABLE_VIEWER | ☑ | ☑ | anchoring/anchorTest_viewer_n.gltf |
| TRACKABLE_FLOOR | ☑ | ☐ | awards/scene_floor_anchoring.gltf |
| TRACKABLE_PLANE | ☑ | ☑ | awards/scene_plane_anchoring.gltf |
| TRACKABLE_CONTROLLER | ☐ | ☑ | anchoring/anchorTest_ctrl_n.gltf |
| TRACKABLE_MARKER_2D | ☑ | ☐ | anchoring/anchorTest_m2D_n.gltf |
| TRACKABLE_MARKER_3D | ☐ | ☐ | anchoring/anchorTest_m3D_n.gltf |
| TRACKABLE_MARKER_GEO | ☑ | ☐ | anchoring/anchorTest_geoSpatial_n_cs.gltf |
| TRACKABLE_APPLICATION | ☑ | ☑ | anchoring/anchorTest_app_n.gltf |
Interactivity
The XR Player supports specifying interactive behaviors in a 3D scene through the MPEG_scene_interactivity and MPEG_node_interactivity glTF extensions.
An interactivity behavior combines one or more triggers that condition the execution of one or more actions.
The table below provide an overview of the supported triggers and actions:
| Trigger type | XR Phone | XR HMD | Test content |
|---|---|---|---|
| TRIGGER_COLLISION | ☑ | ☑ | gravity/gravity.gltf |
| TRIGGER_PROXIMITY | ☑ | ☑ | gravity/gravity.gltf, geometry/UseCase_03-variant1-geometry.gltf |
| TRIGGER_USER_INPUT | ☑ | ☑ | gravity/gravity.gltf, geometry/UseCase_03-variant3-geometry.gltf |
| TRIGGER_VISIBILITY | ☑ | ☑ | geometry/UseCase_03-variant3-geometry.gltf |
| Action type | XR Phone | XR HMD | Test content |
|---|---|---|---|
| ACTION_ACTIVATE | ☑ | ☑ | gravity/gravity.gltf |
| ACTION_TRANSFORM | ☑ | ☑ | gravity/gravity.gltf |
| ACTION_BLOCK | ☑ | ☑ | gravity/gravity.gltf |
| ACTION_ANIMATION | ☑ | ☑ | geometry/UseCase_03-variant1-geometry.gltf |
| ACTION_SET_MATERIAL | ☑ | ☑ | gravity/gravity.gltf |
| ACTION_MANIPULATE | ☐ | ☐ | |
| ACTION_MEDIA | ☐ | ☐ | 🚧 issues/19 |
| ACTION_HAPTIC | ☐ | ☐ | |
| ACTION_SET_AVATAR | ☐ | ☐ | issues/203 |
MPEG extensions to glTF implemented in Unity Player and Unity Editor
Note that “Unity player” refers to the compiled application, while “Unity editor” refers to the development environment which also allows running the app without actually compiling it for the target platform.
| glTF extension | Unity player | Unity editor v1.1.0 |
|---|---|---|
| MPEG_media | ☑ | ☐ |
| MPEG_buffer_circular | ☑ | ☐ |
| MPEG_accessor_timed | ☑ | ☐ |
| MPEG_audio_spatial | ☑ | ☐ |
| MPEG_texture_video | ☑ | ☐ |
| MPEG_scene_interactivity | ☑ | ☐ |
| MPEG_node_interactivity | ☑ | ☐ |
| MPEG_node_interactivity.type | ☑ | ☐ |
| MPEG_anchor | ☑ | ☐ |
| MPEG_sampler_YCbCr | ☐ | ☐ |
| MPEG_primitive_V3C | ☐ | ☐ |
| MPEG_avatar | ☐ | ☐ |
| MPEG_lights_texture_based | ☐ | ☐ |
| MPEG_light_punctual | ☐ | ☐ |
| MPEG_haptic | ☐ | ☐ |
| MPEG_mesh_linking | ☐ | ☐ |
| MPEG_scene_dynamic | ☐ | ☐ |
| MPEG_viewport_recommended | ☐ | ☐ |
| MPEG_animation_timing | ☐ | ☐ |