Apple's Conversational AI Breakthroughs

Efficiently generating word and phrase predictions for intelligent automated assistants like digital assistants. The method involves using a neural network system with three separate neural networks to generate context-relevant text predictions. The first neural network extracts the context, the second determines text predictions, and the third checks relevance. When confidence scores exceed a threshold, the predictions are provided. This allows context-aware text completion with words and phrases instead of just words.

Source

Controlling a virtual agent's actions based on eye behavior of the user and scene information to provide a more intuitive and natural way to interact with virtual agents. The system uses eye tracking to monitor eye movements and updates the virtual agent's appearance or actions accordingly. For example, if the user's gaze shifts to an object, the virtual agent may move towards it or focus its gaze on it. The scene information is also considered to determine appropriate responses. This allows the virtual agent to respond intuitively to the user's eye behavior without requiring explicit user input or hand movements.

Source

Determining whether a speech input is intended for an intelligent digital assistant. The method involves analyzing various factors like textual representation, acoustic features, user gaze, etc., to score the likelihood that the speech input is for the assistant. If the scores indicate high confidence, the assistant processes the input, else it's treated as regular user input. This prevents misinterpretation of spontaneous speech as assistant commands.

Source

Automatic registration of voice-activated shortcuts for application features during installation or after receiving certain types of data. When installing an app, the installation file is scanned for voice shortcuts. These shortcuts are registered with the device's vocabulary engine. When the user says a shortcut, the app executes the associated action. This enables immediate access to app features through voice commands without manual setup.

Source

Providing suggested user inputs for triggering digital assistant tasks. The method involves receiving a user input requesting tasks, analyzing the context of the request, and generating a textual representation of an utterance for performing a task. This textual representation is then displayed as an affordance over the user interface to suggest what the user could say to the digital assistant to execute the task. This helps users discover and remember tasks they can request from the digital assistant.

Source

Non-verbal audio responses from digital assistants to natural language inputs, where the response is adjusted based on task context. If criteria are not satisfied, the response includes both an audio indication of the task and a verbal response. But if criteria are satisfied, only the audio indication is provided without the verbal response. This allows silent confirmation when appropriate, like when a long-running task completes, to avoid repetitive verbal feedback. The criteria could be task-specific, like completion status, or user preference.

Source

Intelligent automated assistant that provides different response modes to user requests based on the type of domain in the request. The assistant selects a response mode from a set of options, each corresponding to a set of domains. For example, when a request is in a finance domain, the assistant might provide detailed text responses with financial data. But for a request in a weather domain, the assistant might just say "It's going to rain today" and not provide text. This allows tailoring the response type (e.g., audio vs text) and affordances (e.g., buttons vs lists) to match the user's intent and expected response format.

Source

Word prediction for devices like smartphones that improves accuracy by integrating multiple contexts. The system receives a plurality of words and obtains two contexts: one based on the received words and another based on the received words plus an additional context. It calculates separate word probabilities using language models for each context. Then it combines these probabilities using weights to generate a final prediction. This allows better handling of context-specific outliers and multilingual code switching compared to simple language models.

Source

Processing natural language requests using natural language input and input from sensors to improve understanding of user intent. The method involves parsing both the natural language input and sensor data into nodes of a structure. The nodes are analyzed together to determine user intent. By combining natural language understanding with sensor data analysis, it allows a digital assistant to better understand user requests that cannot be fully determined from natural language alone.

Source

Providing suggested subsequent user actions during a conversation between a user and a digital assistant to increase efficiency by anticipating and presenting options. The digital assistant determines possible next actions based on the current request, selects the most likely one, and suggests it to the user. This is done by analyzing the request domain and determining parameters for multiple potential subsequent actions. If one action's score is higher than another's, it's selected as the suggested action.

Source

Real-time social intelligence for accommodating social engagements of a user without interruption from a digital assistant. The system determines if the user is engaged in social interactions based on factors like detecting nearby people, user gaze, and speech inputs. If so, it foregoes providing outputs from the digital assistant during that time. This allows users to socially engage without having to manually pause the assistant.

Source

Generating responses from a machine assistant that engenders greater user confidence and reduces unnecessary user inputs by basing the response style on the user's interaction style. The machine assistant analyzes user input to determine the interaction style, like word choice and speech characteristics, and uses that to generate responses that match the user's style. This improves user trust in the assistant compared to more dissimilar responses. By mimicking the user's style, the assistant reduces the need for users to verify the responses, saving resources and improving privacy/safety.

Source

Coordinating intelligent automated assistants across multiple devices to provide seamless, context-aware task performance. The system involves electing a central "context collector" device from a group of devices sharing context information. Devices can delegate tasks to the collector, which provides aggregated context. This allows remote devices to command local devices using the collector as intermediary. Devices can also receive context updates from other devices. The collector is elected based on factors like network strength. This enables a single assistant-like experience across devices without needing to interact directly with each device.

Source

Reducing the length of media item descriptions provided by intelligent assistants based on confidence levels. When a user requests a media item, the assistant identifies the item and its description. It calculates a confidence level for the match. If the confidence exceeds a threshold, the assistant shortens the description and provides the abbreviated version to the user. This reduces verbosity for confident matches.

Source

Resolving ambiguity in user inputs to intelligent assistants when accessing application functionality. The technique involves generating a set of token sequences representing the user input, expanding them into candidate interpretations with actions, and selecting the best interpretation based on metadata. This allows more specific application-focused actions to be preferred over generic language uses.

Source

Facilitating continuous dialog with a digital assistant by allowing robust interactions between a user and an assistant using multi-stage speech analysis, dynamic interruption, and correction capabilities. The technique involves multiple stages of speech analysis with specific values considered based on the current context or state. It allows users to interrupt and correct the assistant mid-conversation to ensure accurate understanding. This provides a more effective and seamless way for users to interact with devices compared to traditional digital assistant systems.

Source

Voice triggering for audio devices like headphones to detect virtual assistant commands from a connected device without requiring complex voice recognition models. The headphones determine the connected device type, load a specific voice recognition model for that device, and listen for the associated trigger phrase. This allows headphones to reliably trigger the correct virtual assistant on the connected device without needing to differentiate between trigger phrases from multiple assistants.

Source

Multi-function portable electronic device that supports voice activation without user interaction. The device can have functions like voice communications and media playback. When a call comes in while media is playing, it pauses the media, answers the call, and resumes media after the call ends. This prevents missed calls or interruptions. The device listens for voice commands without user prompting. It recognizes authorized commands based on operational state. This enables context-aware voice control like answering calls or pausing media. The device can also interact with external media systems using commands sent over a connection.

Source

Creating and updating natural language models for intelligent assistants that allows more accurate and efficient interpretation of user requests. The method involves augmenting a received utterance with additional variations that are already known to the model. This expands the coverage of the model without requiring more work to collect new utterances. The augmented utterances are mapped to the original action structure to build the updated model. This allows the assistant to handle variations of previously unseen requests. The model is then shared with other devices.

Source

Automated task trigger system that allows users to configure tasks to be performed based on specific conditions. The system receives user input selecting conditions and tasks, and stored context data. It checks if the context indicates the selected conditions occur, and if so, performs the associated tasks. This allows customization of future task execution based on user-defined conditions.

Source