Apple's Conversational AI Breakthroughs

Efficiently generating word and phrase predictions for intelligent automated assistants like digital assistants. The method involves using a neural network system with three separate neural networks to generate context-relevant text predictions. The first neural network extracts the context, the second determines text predictions, and the third checks relevance. When confidence scores exceed a threshold, the predictions are provided. This allows context-aware text completion with words and phrases instead of just words.

Source

Controlling a virtual agent's actions based on eye behavior of the user and scene information to provide a more intuitive and natural way to interact with virtual agents. The system uses eye tracking to monitor eye movements and updates the virtual agent's appearance or actions accordingly. For example, if the user's gaze shifts to an object, the virtual agent may move towards it or focus its gaze on it. The scene information is also considered to determine appropriate responses. This allows the virtual agent to respond intuitively to the user's eye behavior without requiring explicit user input or hand movements.

Source

Determining whether a speech input is intended for an intelligent digital assistant. The method involves analyzing various factors like textual representation, acoustic features, user gaze, etc., to score the likelihood that the speech input is for the assistant. If the scores indicate high confidence, the assistant processes the input, else it's treated as regular user input. This prevents misinterpretation of spontaneous speech as assistant commands.

Source

Automatic registration of voice-activated shortcuts for application features during installation or after receiving certain types of data. When installing an app, the installation file is scanned for voice shortcuts. These shortcuts are registered with the device's vocabulary engine. When the user says a shortcut, the app executes the associated action. This enables immediate access to app features through voice commands without manual setup.

Source

Providing suggested user inputs for triggering digital assistant tasks. The method involves receiving a user input requesting tasks, analyzing the context of the request, and generating a textual representation of an utterance for performing a task. This textual representation is then displayed as an affordance over the user interface to suggest what the user could say to the digital assistant to execute the task. This helps users discover and remember tasks they can request from the digital assistant.

Source

Non-verbal audio responses from digital assistants to natural language inputs, where the response is adjusted based on task context. If criteria are not satisfied, the response includes both an audio indication of the task and a verbal response. But if criteria are satisfied, only the audio indication is provided without the verbal response. This allows silent confirmation when appropriate, like when a long-running task completes, to avoid repetitive verbal feedback. The criteria could be task-specific, like completion status, or user preference.

Source

Intelligent automated assistant that provides different response modes to user requests based on the type of domain in the request. The assistant selects a response mode from a set of options, each corresponding to a set of domains. For example, when a request is in a finance domain, the assistant might provide detailed text responses with financial data. But for a request in a weather domain, the assistant might just say "It's going to rain today" and not provide text. This allows tailoring the response type (e.g., audio vs text) and affordances (e.g., buttons vs lists) to match the user's intent and expected response format.

Source

Word prediction for devices like smartphones that improves accuracy by integrating multiple contexts. The system receives a plurality of words and obtains two contexts: one based on the received words and another based on the received words plus an additional context. It calculates separate word probabilities using language models for each context. Then it combines these probabilities using weights to generate a final prediction. This allows better handling of context-specific outliers and multilingual code switching compared to simple language models.

Source

Processing natural language requests using natural language input and input from sensors to improve understanding of user intent. The method involves parsing both the natural language input and sensor data into nodes of a structure. The nodes are analyzed together to determine user intent. By combining natural language understanding with sensor data analysis, it allows a digital assistant to better understand user requests that cannot be fully determined from natural language alone.

Source

Providing suggested subsequent user actions during a conversation between a user and a digital assistant to increase efficiency by anticipating and presenting options. The digital assistant determines possible next actions based on the current request, selects the most likely one, and suggests it to the user. This is done by analyzing the request domain and determining parameters for multiple potential subsequent actions. If one action's score is higher than another's, it's selected as the suggested action.

Source

Real-time social intelligence for accommodating social engagements of a user without interruption from a digital assistant. The system determines if the user is engaged in social interactions based on factors like detecting nearby people, user gaze, and speech inputs. If so, it foregoes providing outputs from the digital assistant during that time. This allows users to socially engage without having to manually pause the assistant.

Source

Generating responses from a machine assistant that engenders greater user confidence and reduces unnecessary user inputs by basing the response style on the user's interaction style. The machine assistant analyzes user input to determine the interaction style, like word choice and speech characteristics, and uses that to generate responses that match the user's style. This improves user trust in the assistant compared to more dissimilar responses. By mimicking the user's style, the assistant reduces the need for users to verify the responses, saving resources and improving privacy/safety.

Source

Coordinating intelligent automated assistants across multiple devices to provide seamless, context-aware task performance. The system involves electing a central "context collector" device from a group of devices sharing context information. Devices can delegate tasks to the collector, which provides aggregated context. This allows remote devices to command local devices using the collector as intermediary. Devices can also receive context updates from other devices. The collector is elected based on factors like network strength. This enables a single assistant-like experience across devices without needing to interact directly with each device.

Source

Reducing the length of media item descriptions provided by intelligent assistants based on confidence levels. When a user requests a media item, the assistant identifies the item and its description. It calculates a confidence level for the match. If the confidence exceeds a threshold, the assistant shortens the description and provides the abbreviated version to the user. This reduces verbosity for confident matches.

Source

Resolving ambiguity in user inputs to intelligent assistants when accessing application functionality. The technique involves generating a set of token sequences representing the user input, expanding them into candidate interpretations with actions, and selecting the best interpretation based on metadata. This allows more specific application-focused actions to be preferred over generic language uses.

Source

Facilitating continuous dialog with a digital assistant by allowing robust interactions between a user and an assistant using multi-stage speech analysis, dynamic interruption, and correction capabilities. The technique involves multiple stages of speech analysis with specific values considered based on the current context or state. It allows users to interrupt and correct the assistant mid-conversation to ensure accurate understanding. This provides a more effective and seamless way for users to interact with devices compared to traditional digital assistant systems.

Source

Voice triggering for audio devices like headphones to detect virtual assistant commands from a connected device without requiring complex voice recognition models. The headphones determine the connected device type, load a specific voice recognition model for that device, and listen for the associated trigger phrase. This allows headphones to reliably trigger the correct virtual assistant on the connected device without needing to differentiate between trigger phrases from multiple assistants.

Source

Multi-function portable electronic device that supports voice activation without user interaction. The device can have functions like voice communications and media playback. When a call comes in while media is playing, it pauses the media, answers the call, and resumes media after the call ends. This prevents missed calls or interruptions. The device listens for voice commands without user prompting. It recognizes authorized commands based on operational state. This enables context-aware voice control like answering calls or pausing media. The device can also interact with external media systems using commands sent over a connection.

Source

Creating and updating natural language models for intelligent assistants that allows more accurate and efficient interpretation of user requests. The method involves augmenting a received utterance with additional variations that are already known to the model. This expands the coverage of the model without requiring more work to collect new utterances. The augmented utterances are mapped to the original action structure to build the updated model. This allows the assistant to handle variations of previously unseen requests. The model is then shared with other devices.

Source

Automated task trigger system that allows users to configure tasks to be performed based on specific conditions. The system receives user input selecting conditions and tasks, and stored context data. It checks if the context indicates the selected conditions occur, and if so, performs the associated tasks. This allows customization of future task execution based on user-defined conditions.

Source

Enabling digital assistants to understand new commands and interact with new applications without extensive retraining. The method involves leveraging existing lightweight natural language models for initial recognition, then if they fail, using a more complex natural language model. The lightweight models are associated with specific applications and registered when data from those apps is received. This allows the digital assistant to understand commands for new apps without retraining the full model. Other techniques involve using speech recognition and reference resolution to infer application and object references from ambiguous requests.

Source

Recognizing and generating emojis from user speech inputs, especially when using voice recognition, to allow efficient insertion and modification of emojis without requiring keyboard or screen interaction. The technique involves determining an emoji corresponding to a portion of the speech, and then adjusting it based on modifiers in the speech. This allows specific emojis to be requested and included in messages using voice commands.

Source

Using sensor data from devices like smartphones to predict applications and actions based on historical usage in identifiable locations. The method involves training a machine learning model using labeled sensor data from known locations to cluster sensor readings. When new sensor readings are received, the model predicts which location cluster they belong to and recommends the associated application or action. The model is trained using a combination of labeled and unlabeled sensor data. This allows accurate predictions without relying solely on GPS location which can be imprecise indoors.

Source

Controlling autonomous agents like robots in dynamic environments by combining high-quality real-time data with models based on historical data to make more accurate predictions of agent behavior. The system obtains sparse high-quality data about the current state of an agent and the environment, like sensor readings, along with behavior models based on large volumes of historical data. It uses both sources to determine intended actions for the agent and predict future states of other agents. This multimodal data fusion improves prediction accuracy compared to relying solely on real-time or historical data.

Source

Identifying and surfacing trending issues in text streams like emails, chats, reviews, etc. using textual analysis. The technique involves automatically analyzing text streams to proactively identify trending issues as close to the source as possible. Textual analysis like natural language processing, machine learning, and anomaly detection are used to identify issues described in a threshold number of text streams in a certain time period. This allows faster detection and surfacing of trending issues to support teams compared to manual voting escalation. It also flags issues that might otherwise go undetected.

Source

Identifying users of digital assistants in a multi-user environment and providing personalized responses based on user identification. The digital assistant receives speaker profiles of registered users and compares them to speech inputs. If a user's profile matches closely, it provides a personalized response. If no match is close enough, it assumes the input is from an unregistered user and determines if it's a personal request. If so, it finds the user and provides a personalized response. If not, it provides a generic response. This allows accurate user identification and customized responses even in shared devices.

Source

System for playing media on multiple devices using natural language commands. It enables seamless transfer of media playback between devices like phones and TVs. The system involves a digital assistant that receives media playback information from devices and user requests to play media on specific devices. It then sends instructions to the requested device to start playback. This allows users to say things like "play this on that" to transfer media between devices. The assistant also handles scenarios like when authorization is needed to access media.

Source

Call assistance using contact suggestions to improve accuracy of initiating outgoing calls when the user's initial input doesn't match the contact exactly. The system receives an initial name reference from the user, provides a response, then detects an outgoing call event associated with a contact within a certain time. If the contact matches the name reference, an association is stored. This allows the system to associate the user's initial input with the correct contact even if it doesn't perfectly match the contact name.

Source

Providing personalized responses based on semantic context by leveraging user data and history. The method involves receiving a message, analyzing its semantic representation, and identifying similar messages in the user's history. Then, suggested responses from those similar messages are provided back to the user. By mapping messages into a vector space based on context, it allows personalized responses that are tailored to the user based on semantics, recipient identity, historical usage, etc.

Source

Improving digital assistant understanding of ambiguous user requests by leveraging context data and metadata extraction to disambiguate and provide more accurate responses. The method involves determining if a portion of a user's request is ambiguous, collecting context data around that ambiguity, extracting metadata from the context, and using the metadata to disambiguate the request and generate a more appropriate response.

Source

Intelligent system that responds to user intent and desires based upon activity that may or may not be expressly directed at the system. The system acquires 3D sensor data to interpret the scene, user activity, and hand gestures. It infers user desires from scene geometry, user engagement, and gesture recognition. The system adjusts output based on factors like user location, acoustic reflection, and presence indicators. The goal is to improve user experience by anticipating and fulfilling user desires without explicit commands.

Source

Using virtual assistants to assist users in completing tasks during group communication sessions. The virtual assistant can understand and interpret natural language commands and requests made by users during the session, then perform actions and tasks on behalf of the users. This allows automation and streamlining of tasks like scheduling, coordinating payments, etc. within group chats without requiring manual coordination and communication between all participants.

Source

Efficient techniques for managing user profiles on devices like smartphones using voice inputs. The techniques involve initiating playback of media items and switching user profiles using voice commands. When a user requests access to a restricted media item, if their voice matches a authorized profile, playback starts. If the voice doesn't match, it's forbidden. For switching profiles, if the voice matches a profile, it switches. If not, it's forbidden. This reduces cognitive load and saves time compared to manual inputs.

Source

Automatically suggesting media assets like music for a user's multimedia presentations based on contextual analysis of their own media items. The method involves requesting candidate media assets for a set of user media items using a knowledge graph metadata network, receiving metadata for the candidates, determining ranked sets of media assets based on that metadata, and outputting the ranked sets for selection. This leverages machine learning and graph analysis to suggest compatible media assets that enhance the user's content.

Source

Determining and executing task flows from user inputs with intelligent automated assistants like digital assistants. The method involves receiving a user request, determining the natural language representation of the request, finding an associated software process, checking if the request can be executed by that process, and if not, determining transformation instructions to revise the request and execute it with a different process.

Source

Enhanced prediction system using generative adversarial networks (GANs) and distillation to improve on-device prediction accuracy on devices with limited resources. The system leverages GANs to learn a high-accuracy teacher model and a lower-accuracy student model. It uses distillation techniques to transfer knowledge from the teacher to the student. The student generates predictions, which are then reconstructed by the teacher. The reconstructed outputs are compared with the original student outputs to enforce cycle consistency. This encourages the student to learn the same distribution as the teacher. The GAN and distillation training together improves the student model's prediction accuracy.

Source

Disambiguating user requests to intelligent assistants like Siri when the request contains an ambiguous entity, by sending the request to an external device and receiving disambiguation options and semantic information. The local device then determines if it can disambiguate the request based on user data, and if so, performs the task. If not, it requests user input to disambiguate. This leverages external knowledge to improve disambiguation accuracy while respecting user privacy by not sharing their data.

Source

Improving user input interpretation on devices like smartphones by leveraging the displayed content to aid interpretation. The system generates and dynamically updates a representation of displayed objects and relationships. When receiving a user input, it retrieves the representation from that time and uses it to interpret the input better. This allows the device to better understand user requests based on the current display context. The representation includes object representations and relationship representations. It is updated as objects move or new ones appear.

Source

Providing intelligent contextual action predictions by a digital assistant on an electronic device. When a user interacts with displayed content, the digital assistant determines the content type, extracts data items, determines action types, and presents suggestive actions based on that context. This provides personalized and relevant action suggestions compared to generic lists. The suggestions are determined from factors like source app, user input accuracy, engagement history.

Source

Intelligent automated assistant handling of health-related requests. The assistant determines if it is authorized to access requested health information based on user permissions and second user sharing settings. If authorized, it performs tasks like providing output based on the health data. If not authorized, it informs the user it cannot access the health info. This prevents unauthorized access to sensitive health data while still enabling assistants to handle health-related requests.

Source

Suggesting applications and recipients for communication to improve efficiency by reducing the number of actions required to initiate a communication. The suggestions are generated based on historical user interactions, contextual data, and machine learning models. The suggestions are presented on a user interface to select from instead of requiring the user to navigate through lists of apps and contacts. The suggestions are personalized using on-device training with user-specific labels.

Source

Reducing latency in multilingual digital assistants when the input language is incorrectly determined. The method involves displaying multiple recognition results in different languages for a natural language input, allowing the user to select the correct language. This avoids the latency of detecting the incorrect language and then reverting to the correct language. The method also involves determining the likelihoods of the input being in the displayed languages and initiating tasks based on the more likely language.

Source

Continuous dialog with a digital assistant that allows seamless, uninterrupted interactions between a user and a digital assistant without requiring trigger inputs between each request. The system initiates a session window with a variable speech threshold that increases if speech is detected during the session. If the threshold doesn't exceed a limit, the session ends. This allows users to interrupt and continue dialog without repetitive triggers.

Source

Providing personalized suggestions indicating that a task can be performed using a digital assistant on an electronic device based on context data. The suggestions are shown when conditions are met, like user expertise level. The context data is analyzed to determine tasks the assistant can do. Suggestion criteria are checked against the context. If satisfied, a suggestion is presented to the user. This increases relevance and usefulness of the suggestions.

Source

Interactive reading assistant that modifies text content based on reading proficiency and engagement to help users improve their reading skills. The assistant assesses a user's reading ability by analyzing their speech when reading. If the user's reading proficiency is low, the assistant adjusts the difficulty level of the text. It may also distinguish certain words until they are correctly pronounced. If the user is disengaged, it presents easier or more interesting content. The assistant aims to provide personalized reading experiences that encourage users to continue reading and improve their skills.

Source

Accurately and efficiently interpreting spoken requests by digital assistants using a multi-stage technique that determines if the request matches predefined user-defined invocation phrases before resorting to natural language processing. This involves a strict matching stage where the request text is checked for exact matches to the user-defined phrases. If found, the corresponding task flow is executed directly. If not, natural language processing is used. This allows fast, simple invocation matching without relying on complex natural language understanding.

Source

Portable electronic devices like smartphones and tablets with improved voice command capabilities that allow more sophisticated and contextually aware voice control. The devices capture voice commands using a microphone and associate them with contextual information like the current app or media being played. This allows more complex voice commands beyond basic device functions. For example, a user could say "find more like this" while listening to a song to have the device recommend similar songs. The device can then send the voice command and contextual data to remote services for processing and return appropriate actions.

Source

Efficiently updating a language model using adversarial discriminative adaptation to accurately reflect individual user idiosyncrasies. The method involves training a first language model using user data, storing a reference version, then updating a second model using the first version as a constraint. This preserves overall probabilities while adapting to new user data.

Source

Providing personalized suggestions indicating that a task can be performed using a digital assistant on a device, based on context data, to improve relevance and usefulness. The device receives context data, determines tasks the assistant can do based on it, checks criteria, and suggests tasks if satisfied. This leverages context like user activity and knowledge level to provide more targeted and useful task suggestions.

Source

Generating voice shortcuts for multiple tasks associated with a user activity based on the user activity and contextual data. The method involves receiving a user input indicating the activity, determining contextual data for that activity, presenting a set of candidate tasks for the activity, having the user select some tasks, and associating those selected tasks with a voice shortcut for the activity. This allows efficient execution of multiple related tasks during an activity using voice commands instead of navigating interfaces for each task.

Source