Apple's Large Language Model Breakthroughs

A method for generating and editing text using a digital assistant and/or language model, comprising: receiving a user input requesting text generation; determining whether additional information is required for the generated text; if additional information is required, displaying a request for the information without generating the text; and if additional information is not required, generating the text using the language model and displaying it with placeholders for the additional information.

Source

A method for executing actions using an artificial intelligence engine, comprising receiving a user input, generating multiple actions based on the input using a large language engine, estimating certainty values for each action using an estimation engine, presenting actions with high certainty values to the user, receiving a user selection, and instructing accessory devices to perform the selected action.

Source

A method for providing relevant search results to users based on their unstructured search queries. The method involves using large language models (LLMs) to convert unstructured queries into structured queries, then sending those structured queries to specialized knowledge sources for results. The results are aggregated and filtered by another LLM to produce final search results. This allows leveraging the LLMs' natural language understanding to generate structured queries that can be more precisely processed by the knowledge sources.

Source

A head-mounted display (HMD) system that enables users to request summarized text from their physical environment using natural language input. The system uses computer vision to capture images of the environment, extracts text from the images, sends it to a trained language model, and displays the summarized text back to the user on the HMD. This allows users to request summaries of text they see in the real world using voice commands, without the need to manually copy or photograph the text.

Source

Memory-efficient differentiable weight clustering for large language model compression that enables state-of-the-art performance on constrained devices. The approach employs novel memory compression techniques that reduce the memory footprint of weight clustering by applying uniquification and sharding during the backward pass. This enables significant compression ratios while maintaining accuracy, making it particularly suitable for deploying large language models on mobile devices.

Source

Creating and updating natural language models for digital assistants that allows more efficient and accurate interpretation of user requests. The method involves associating actions with user requests, determining augmented utterances based on the original request, and creating the model by mapping the augmented utterances to the associated actions. This allows the model to handle variations and unknown words in new requests. It also involves sharing learned models between applications.

Source

Grouping and presenting content items by semantically equivalent topics instead of strict grouping by topic tags to prevent confusion when topic tags are different in different languages. The technique involves determining semantically equivalent topic tags across languages and grouping content items based on these equivalent topics. This allows presenting all politics-related content together regardless of whether the topic tag is "politics" in English or "politique" in French.

Source

Efficiently updating a language model using adversarial discriminative adaptation to accurately reflect individual user idiosyncrasies without requiring large amounts of user data. The technique involves training a first language model using user data and then storing a reference version with the overall probability distribution. A second language model is trained using new user data, but constrained by the reference version's probability distribution. This updates the second model with the user's idiosyncrasies while preventing it from diverging too far.

Source

Generating sentence embeddings for natural language inputs to enable improved natural language processing tasks like semantic search, question answering, and text generation. The embeddings capture the meaning of sentences in a vector space. They are generated using language models that convert sequences of words into vectors. The embeddings can be used for tasks like finding semantically similar sentences, matching questions with answers, and pairing images with songs based on descriptions.

Source

Improving text prediction by incorporating user feedback into text prediction models. It uses reinforcement learning techniques like imitation learning to optimize a single language model for text prediction according to user's feedback. The model predicts both the next word and the user's intended action on it. If the predicted action doesn't match the user's actual action, model parameters are updated to better align with the user's behavior. This allows the model to learn user-specific idiosyncrasies and improve text prediction accuracy over time as the user interacts with it.

Source

A prompt for generating generative content can include text, images, drawings, videos, or a combination thereof, with optional parameter values indicating importance. The prompt can be structured with phrasing, style, context, and role specifications. The prompt is processed by machine learning models, including transformer-based neural networks, to generate novel content. The prompt can be multi-modal, incorporating multiple content types, and can include structured instructions.

Source

A method for displaying notifications with summary content, comprising detecting an event corresponding to application content, and displaying a notification including an automatically generated summary based on the content, where the summary includes content not part of the original content. The method further includes determining whether to display the notification based on criteria such as event type, application relevance, and content length.

Source

Computer system that assists users with tasks in 3D scenes by detecting gaze and scene data, generating plans based on semantic information and goal states, and executing actions to achieve those goals. The system can also adapt to changes in the scene by adjusting sensor parameters when the actual state differs from predicted state.

Source

Electronic devices with improved notification systems that automatically generate summaries of application content and group notifications based on relevance criteria. The system detects user events and generates notifications with summaries when content exceeds a predetermined length threshold. It also displays grouped notifications with concurrent representations of multiple notifications, enabling users to interact with individual notifications within the group. The system determines notification relevance based on user input and application content, and suppresses notifications that do not meet relevance criteria when operating in a restricted mode.

Source

A method for providing relevant search results that combines user intent with item relevance. The method generates a query vector based on a user's search query and combines it with a user account vector to establish a combined vector. The combined vector is then used to generate an output vector that is compared to item vectors to determine similarity scores. The items are ordered based on their similarity scores and displayed to the user with corresponding affordances.

Source

Providing relevant search results for search queries using AI techniques that generate stable item embeddings from multiple modalities like metadata, images, videos, and audio to provide personalized search results while respecting privacy. The method involves generating a user vector from their search query and account info, combining it with item vectors from search results, comparing to find matches, ordering by similarity, and displaying affordances. The item vectors are generated using AI models trained on stable input embeddings derived from song metadata, album art, videos, and audio.

Source

Reducing resource consumption for generating 3D environments by providing a user interface that guides users in inputting parameters for the environment instead of directly editing the 3D model. The interface allows previewing and editing a 2D version of the environment before generating the 3D version. This reduces the need to repeatedly invoke the resource-intensive 3D generation process for minor edits.

Source

A method for generating search results by interacting with multiple domains, comprising receiving an unstructured query, identifying domains to route the query to, generating structured queries for each domain using domain-specific models, aggregating results from each domain, filtering the aggregated results, and displaying the filtered results to the user.

Source

Techniques for generating responses to search queries by interacting with multiple domains. The method involves breaking down a search query into sub-questions that can be routed to specific domains for answering. Each domain generates structured queries from its models to access knowledge sources. Results are aggregated and filtered to produce the final response.

Source

Digital assistant that automatically learns user environment and determines relevant tasks through environmental analysis and audio cues. The assistant uses camera and microphone inputs to capture environment descriptions, then analyzes these descriptions in combination with audio clips to identify relevant activities. Based on this analysis, the assistant provides personalized task recommendations and audio cues to support user needs.

Source