Apple's Breakthroughs in AR/VR Performance

Head-mounted displays for virtual and mixed reality applications that have facial interfaces designed to provide proper positioning and comfort for the displays while accommodating facial movements. The interfaces engage regions above and below the user's eyes. They have different vertical and fore-aft stiffnesses in those regions to distribute loading and prevent excessive restriction of facial movements. The upper facial support has greater vertical and fore-aft stiffnesses than the lower support. This allows the display to move more vertically and fore-aft when the user smiles or looks down without excessive movement restriction.

Source

Populating XR environments with variations of objects to make them look more realistic and reduce the need for manually customizing numerous copies. The method involves generating variations of an object in a XR environment by assigning values for visual properties based on distribution criteria rather than copying and manually editing multiple instances. This provides a more realistic appearance when populating XR environments with objects like books or cars, reduces memory usage compared to storing multiple variations, and avoids repetitive user customization.

Source

Guided stretching session system that uses 3D environments, user profiles, and head pose tracking to provide personalized and interactive stretching exercises. The system presents visual cues and a direction indicator in a 3D environment based on user profiles and head pose. As the user moves, the indicator updates and feedback is provided when cues are completed. This adaptive feedback helps users maintain proper form and progress through the session.

Source

Head-mounted display device with compact optical modules for VR/AR headsets. The modules have a lens, display, and camera all connected to a housing that can be adjusted to vary the interpupillary distance. This allows the modules to be moved closer together to reduce overall size. The cameras also move with the modules. This compact design enables more compact headsets by reducing the distance between the modules and cameras. The adjustable interpupillary distance also accommodates different user head sizes.

Source

Electronic devices like head-mounted displays that merge real-world and virtual images to create augmented reality experiences. The devices have displays to show virtual content and optical systems to combine real-world and virtual images. The optical systems include components that merge the two types of light to create a single view for the user. This allows virtual objects to appear overlaid on the real world. The devices can have transparent displays, projectors, or holographic systems to create the mixed reality experience.

Source

Head-mounted devices like VR/AR headsets that resist motion with adjustable tension to improve comfort and prevent blurring. The devices have sensors to detect user head movement, and a tension controller analyzes the data to predict future motion. It then sends tension commands to an adjuster that applies force to resist head movement. This prevents the device from moving relative to the user's head, improving visual clarity and comfort compared to passive headstraps. The tension can be actively adjusted based on user behavior profiles and motion predictions.

Source

Electronic devices with optical components like head-mounted displays that have improved lens support structures to prevent detachment during drops. The lens support structure has an inner frame around the lens that is attached to the device housing. The lens is then mounted on the inner frame using a flexible lens retention mechanism. This allows the lens to flex and deform slightly during impact to absorb shock and prevent detachment.

Source

Wearable electronic devices like head-mounted displays with adjustable optical components that can improve user experience by dynamically enhancing visibility of virtual content overlaid on real-world objects. The devices have adjustable optical components like spatially addressable light modulators or lenses that can selectively darken parts of the user's field of view. This allows bright real-world objects to be dimmed when overlapped by virtual content, making the virtual content more visible. The adjustable optical components can be integrated into the device to avoid bulkiness and weight issues.

Source

Modifying virtual content presented on opaque displays to simulate how it would look on see-through displays. The system takes virtual content and modifies it based on simulation characteristics of see-through displays to more accurately replicate how it would appear on an actual see-through display. It also evaluates placement of virtual objects to avoid user perception issues when presented on a simulated see-through display.

Source

Enhancing user interactions with electronic devices by generating tactile outputs in response to device orientation and proximity to objects. The device generates tactile feedback when oriented within a range that changes with distance. It also modifies feedback based on device orientation. This reduces cognitive load and power usage compared to redundant user inputs. The device also generates tactile outputs indicative of AR planes, data sharing, and objects hovering over the display.

Source

Accurately positioning virtual content in mixed reality scenes relative to the physical environment and across multiple users and sessions. The technique involves identifying content from multiple sources based on device location, then accurately placing it in the scene using a combination of coarse and fine localization. This allows seamless integration of virtual and physical objects. The fine localization uses techniques like analyzing images of physical objects and matching 3D models.

Source

Changing display rendering modes based on multiple regions of a physical environment to improve user experience in augmented reality and other extended reality applications. The display adapts color characteristics of virtual content and the physical environment to provide a more immersive experience. It detects color characteristics of both the virtual content and the surrounding physical environment, and changes the display mode to match them when certain conditions are met. This prevents color conflicts between the virtual and physical elements.

Source

Improving efficiency of scanning and modeling environments using augmented and virtual reality by providing better user interfaces and scanning methods. The techniques include: 1. Previewing a 3D model of the environment being scanned while scanning, with the preview rotatable independently of the camera view. This allows users to easily view and manipulate the model without constantly realigning it to match the camera view. 2. Displaying visual cues over the camera feed to indicate unscanned areas between scanned sections. This helps users efficiently fill in gaps and complete the scan. 3. Changing the visual properties of graphical objects representing scanned features based on predicted accuracy, to provide feedback to the user. 4. Rotating the preview of the 3D model in response to user input, without requiring them to also rotate the camera view. This allows users to quickly reorient the preview to match their perspective.

Source

Displaying indicators in extended reality views that show content being used on other devices, positioned based on the other device's location, to provide intuitive access to cross-device content. For example, if a user views a website on their phone, an HMD view may contain an indicator representing the website near the phone's depiction, to indicate the website is open there.

Source

Compact and efficient image projector for AR/VR applications that uses a 1D scanning system to project a sequence of vertical line images. A scanning mirror reflects the lines as they are generated by a 1D image source. A pupil expander then expands the reflected lines into parallel replicas that fill the eye box. This allows a compact projector with high resolution, efficiency, and low power by focusing on vertical lines instead of a full 2D image. The lines are generated by emitters on a photonic integrated circuit that can modulate color and intensity.

Source

Head-mounted device with adjustable optical module positions for accommodating different user interpupillary distances. The device has actuators and optical module guide structures to allow the optical module positions to be adjusted. This allows customizing the headset to fit different users' eye spacing without needing separate headset models.

Source

Refining 3D surface representations of physical environments using a second 3D surface representation with geometric primitives to improve the accuracy and smoothness of the initial 3D point cloud or mesh. The method involves adjusting points in the initial representation based on corresponding points on the geometric primitives. This reduces noise and provides a better representation of flat surfaces like tables.

Source

Enhancing online shopping and customer support experiences by providing real-time communication with a remote salesperson through video chat, allowing the salesperson to demonstrate products and guide the user through the shopping experience. The user's device can also show a view of their environment with products added to see how they would look in the user's space. This provides an interactive, guided consumer experience with computer-generated reality capability.

Source

Head-mounted display (HMD) system that provides content optimized for low-light environments using onboard sensors like IR, depth, and ultrasonic. The HMD senses the environment in low light conditions using these sensors and generates customized content like enhanced graphics, overlays, and indicators based on the sensor data. This allows the display to adapt and enhance the user experience in dark environments where visibility is limited. The sensors also allow the HMD to detect objects and provide augmented reality features even in low light.

Source

Detecting eye gaze direction of users of electronic devices like head-mounted displays using low-power sensors to initiate actions like launching experiences when the user's gaze is approximately oriented towards a target area. An illuminator directs light towards the eye to produce a reflection. A sensor aligned with the gaze direction detects the reflection's properties. If the gaze is approximately towards the target area, it indicates the user's gaze and initiates the action. This allows efficient gaze detection without continual full eye tracking and reduces power.

Source

Extended reality (XR) system that allows users to carry XR content into the physical world by associating it with a portable companion device. When the companion device is detected by the XR system, it retrieves and displays the associated content. This allows users to transport XR content like photos, documents, or digital currency outside of the XR system and access it later. The companion device has a unique identifier that is obtained by the XR system when it is detected. This allows the XR system to associate the companion device with specific content.

Source

Modifying the visual appearance of objects in augmented reality (AR) or virtual reality (VR) based on their real-world locations. The method involves determining the visual properties of physical elements associated with the object's theme. Then, it finds the physical elements at the object's location and uses their visual properties to modify the object's appearance. This allows tailoring the AR/VR objects' look to fit their real-world surroundings.

Source

Low-latency, block-based video encoding for wireless VR/AR systems. The encoding method aims to transmit high-resolution video over wireless links with low latency and reliability constraints. It uses block-level rate control to estimate quantization parameters for individual blocks in a frame instead of frame-level or strip-level estimation. This allows accurate rate allocation between blocks while meeting bandwidth constraints. The block-level rate control involves calculating optimal quantization parameters for each block to minimize distortion given a rate bound. It estimates rate and distortion curves for blocks based on DCT histograms and wavelet synthesis. This enables efficient, hardware-friendly block-level rate control for VR/AR video encoding with low latency and error rates.

Source

Head-mounted device like VR/AR headsets that effectively manage heat and noise for user comfort. The device has a flow channel with a moveable support member inside. The support member has sections with different profiles that generate tonal noises at different frequencies when air moves over them. This distributes the tonal noise across a range to mask it among broader noise. The sections also have varying cross-sections to draw heat away from components. A blower moves air through the channel to cool components while generating broadband noise.

Source

Reducing user inputs for adding items to selection queues in virtual environments. The method involves compositing affordances with virtual objects representing physical items. When the affordance is activated, the physical item is added to the selection queue without needing to navigate back to the item's page. This reduces unnecessary user inputs. It also allows viewing virtual representations of wearable items on avatars to see how they fit without needing size charts.

Source

Intuitive user interfaces for virtual and mixed reality systems that provide more efficient and intuitive interaction for users. The interfaces include features like preventing users from interacting with shared objects when another user has control, displaying visual indications when objects are unavailable, moving objects to maintain distances when users approach, and releasing objects back to users with rotated orientations. These interfaces provide timely feedback, reduce unnecessary visual clutter, and avoid confusion when multiple users share virtual environments.

Source

Private conversation feature in virtual reality that allows users to engage in one-on-one conversations within a shared virtual environment without others seeing or hearing the private conversation. It involves detecting when users are looking at each other and initiating a private conversation cloak that mutes their audio and hides their avatars from other viewers. When they exit the private conversation, their audio is unmuted and avatars are visible again.

Source

Head-mounted displays with facial interfaces that have sensors to monitor physiological conditions of the user's face. The facial interface engages the user's facial region to support the display. The sensors in the interface detect parameters like heart rate, sweat, or facial expressions. The display system combines this facial data with other sensors to make decisions and perform actions. It allows personalized experiences based on facial biometrics. The display can adapt content, lighting, or comfort based on facial conditions. The facial sensors provide additional insight beyond just the head-mounted display's sensors. The system can also opt-in or opt-out users from providing facial data.

Source

Improved user interfaces for computer systems that provide computer-generated reality experiences to users, with the goal of making interaction more efficient and intuitive. The interfaces reduce the number, extent, and nature of user inputs by providing feedback that helps users understand the connection between inputs and device responses. This reduces cognitive burden, wastes less energy, and enables more immersive experiences. Examples include avatars with variable appearance based on certainty of body pose, and augmented reality with superimposed virtual objects changing based on user activity.

Source

Automatically selecting a viewing perspective in virtual or augmented reality environments without requiring user input. The technique involves using saliency values derived from the environment to determine the optimal perspective. Saliency values represent the importance or attention-grabbing potential of objects and parts of objects in the scene. By analyzing these saliency maps, the system can intelligently select a viewpoint that maximizes saliency and presents the most engaging and informative perspective to the user. This reduces the need for user-initiated perspective switching and enhances the VR/AR experience.

Source

A method for presenting content in extended reality (XR) environments that allows users to interact with virtual objects at different depths. The method involves dynamically adjusting the depth of virtual user interfaces based on the user's position. If the user is close to a virtual object, the interface for interacting with that object is presented nearby. If the user is far away, the interface is presented further away. This allows immersive XR experiences where virtual objects appear at natural distances, improving the sense of presence.

Source

User interfaces for virtual and mixed reality systems that improve efficiency and intuitiveness of interaction by leveraging spatial relationships and visual effects. The interfaces modify object appearances based on their arrangement relative to the user's viewpoint, scroll objects towards boundaries, and transition objects between 2D and 3D representations. These techniques provide feedback and cues that match real-world interactions, reducing the number and complexity of user inputs.

Source

Collaborative head-mounted devices that share views and data to improve perception, mapping, and processing compared to individual devices. Headsets have cameras to capture scenes. They exchange data with other headsets to determine object characteristics from multiple angles. This provides better object recognition, hand tracking, surface mapping, and digital reconstruction.

Source

Transitioning content in views of three-dimensional (3D) environments using alternative positional constraints to improve user experience and reduce motion sickness when viewing 3D environments using devices like head-mounted displays. The technique involves detecting user interactions with a content object in a 3D environment and then changing the positional constraint for displaying the object. When the user interacts with the object, it transitions from being world-locked (fixed to a specific location) to being device-locked (moved closer to the user) or less world-locked (rotated towards the user). This provides a hybrid localization-based heads up display (HUD) system between world-locked placement of content and head-locked placement.

Source

Method for navigating and exploring unknown extended reality (XR) environments by gradually building a navigation mesh based on the virtual agent's sensory perception. The method involves determining sensory perception regions for the agent based on its location, orientation, and sensory acuity. Then, generating a portion of the navigation mesh covering those regions. As the agent moves, new perception regions are calculated and mesh portions are added. This iterative exploration and mapping process builds the navigation mesh without requiring a priori knowledge of the XR environment.

Source

Compact, power-efficient, and high-performance optical system for displays in electronic devices like head-mounted displays. The system uses a pancake-shaped, integrated design with a miniature optical element that reduces size and weight compared to conventional systems. The pancake design integrates the display's light source, collimating lens, and imaging optics into a single compact module. This eliminates the need for separate components, saving space and power. The integrated pancake optics provide good optical performance for displaying images.

Source

Enhancing multi-user communication in extended reality (XR) environments by providing tools to generate and share virtual modifications across devices in a multi-user session. The tools allow users to create virtual items in their own physical environment and have them accurately appear in the other users' shared XR environment. This consistency is achieved by specifying the spatial relationship between the virtual item and real objects in the environment where it was created. This avoids inconsistencies that could arise from users generating virtual items in different physical locations.

Source

An optical system for foveated displays that reduces the size and power requirements while still providing high resolution in the central region. The system uses separate low and high resolution displays, a waveguide, coupling prism, and steering mirror. The low resolution display sends light into the waveguide through the prism. The high resolution display sends light directly into the waveguide. The steering mirror reflects the high resolution light back into the waveguide. This allows the high resolution section to be positioned accurately at the fovea using the mirror. The coupling prism and waveguide collimate and guide the light to the eye.

Source

Low-power eye tracking for VR/AR headsets that reduces power consumption of eye tracking systems in head-mounted displays (HMDs) like VR headsets. The methods involve using sensors on the HMD to detect relative movement between the head and eyes, allowing reduced-frequency 3D reconstruction and frame rate of eye cameras. This reduces power compared to constant 3D reconstruction. The sensors can be head odometers at the sides/back of the HMD or eye odometers near the eyes.

Source

Improving comfortability of virtual content during head movements in extended reality (XR) systems. The methods involve lazy follow and blurring for head yaw/pitch motions, and scene origin locking for head roll motions, to reduce computational burden while maintaining comfort. For yaw/pitch, the virtual content moves at a slower rate than the head to prevent jitter. For roll, the content stays fixed when roll is low but moves with the head when roll exceeds a threshold. This avoids head-locked content appearing jittery. Blurring or fading is used when roll exceeds a higher threshold.

Source

Improving user interfaces for virtual and augmented reality systems that reduce the number, extent, and nature of inputs from users to create more efficient and intuitive human-machine interfaces. The interfaces involve techniques like reducing visibility of virtual content to allow physical objects to break through, displaying indicators for physical objects when virtual content is present, and dynamically adjusting virtual object prominence based on physical object salience. These methods help users understand the connection between provided inputs and device responses and conserve power.

Source

Enhancing playback of captured audio-visual (AV) content by decoupling and dynamically adjusting the audio based on context. The method involves capturing AV content with separate audio and video streams. The audio is segmented into labeled portions. During playback, the labeled audio is realigned, modified, or adjusted based on user actions and context in an extended reality (XR) environment. This allows decoupling and manipulating the audio independently of the video for enhanced playback experiences.

Source

Compact and efficient optical system for near-eye displays like virtual reality headsets that allows viewers to see both virtual and real-world images. The system aims to improve display design by reducing size, power consumption, and optical performance issues. It involves a novel optical layout with compact components and coatings to enable high-resolution, low-power, and lightweight near-eye displays for virtual and augmented reality applications.

Source

Detecting touches between objects using thermal imaging to provide a robust, contactless touch interface for applications like AR. The method involves capturing thermal images of an object, finding temperature patterns indicative of a touch, and using those patterns to detect touches between objects with different temperatures. By monitoring thermal energy changes, it can distinguish touches from temperature variations due to motion. This allows reliable touch detection without modifying objects or using cameras for pose tracking.

Source

Enhancing extended reality communication experiences by presenting virtual representations of remote participants in a way that matches their capabilities. When a user requests to see a virtual representation of a remote participant in an extended reality environment, the system obtains the participant's capability (e.g., video or audio only). It then presents the virtual representation accordingly, using different visual characteristics based on the capability. For example, if the participant can only do audio, the virtual representation would have a static image. This allows customizing the virtual representation to match the participant's actual capabilities.

Source

Head-mounted displays with multiple cameras for sensing the environment outside the display field. The cameras have overlapping fields of view that cover 360 degrees horizontally. Images from the cameras are processed to detect subjects in the extended field of view. This allows sensing objects, people, or events outside the display's own field of view. The sensed data can be used to generate graphical content, provide video passthrough, or enhance headset functionality like motion tracking or hand gesture recognition.

Source

Optical system for displays that improves the appearance and performance of near-eye displays in devices like head-mounted displays. The system uses a polarizing beam splitter, prisms, and waveplates to separate and recombine the polarizations of unpolarized display light. This avoids bulky polarization components and reduces overall size. The separated light is then coupled into the waveguide using an input coupler. The output coupler couples the recombined light out. The polarization splitting and combining improves display contrast and brightness compared to unpolarized displays.

Source

Wearable controller for head-mounted devices like VR headsets that uses internal sensors to determine the wearer's head position relative to the device without external trackers. The controller has alternating-current (AC) magnetometers, a direct-current (DC) magnetometer, and an accelerometer. It uses the AC magnetometers to measure the headset's AC field and the DC magnetometer and accelerometer to measure orientation. This data lets the controller wirelessly transmit head position to the headset for controlling it without external trackers.

Source

Head-mounted display system with facial interfaces designed to improve comfort and stability while wearing the display. The facial interfaces engage the forehead and temple regions of the user's face to support the display. The interface converts force applied to the forehead into inward force on the temples to stabilize the display. It also has adjustable lateral forces to customize temple pressure. The forehead engagement is fixed while the lower facial engagement allows movement to prevent pressure on the eyes.

Source

Reducing the user inputs needed to organize virtual objects in an extended reality (XR) environment by associating a virtual object with a region based on user gaze instead of large gestures. The method involves detecting a gesture to move a virtual object, then finding the user's focus location in the region using gaze input. The virtual object is then moved to that location in the XR environment. This lets users place virtual objects precisely where they want using natural gaze instead of large gestures.

Source