Apple's Large Language Model Breakthroughs

A method for generating and editing text using a digital assistant and/or language model, comprising: receiving a user input requesting text generation; determining whether additional information is required for the generated text; if additional information is required, displaying a request for the information without generating the text; and if additional information is not required, generating the text using the language model and displaying it with placeholders for the additional information.

Source

A method for executing actions using an artificial intelligence engine, comprising receiving a user input, generating multiple actions based on the input using a large language engine, estimating certainty values for each action using an estimation engine, presenting actions with high certainty values to the user, receiving a user selection, and instructing accessory devices to perform the selected action.

Source

A method for providing relevant search results to users based on their unstructured search queries. The method involves using large language models (LLMs) to convert unstructured queries into structured queries, then sending those structured queries to specialized knowledge sources for results. The results are aggregated and filtered by another LLM to produce final search results. This allows leveraging the LLMs' natural language understanding to generate structured queries that can be more precisely processed by the knowledge sources.

Source

A head-mounted display (HMD) system that enables users to request summarized text from their physical environment using natural language input. The system uses computer vision to capture images of the environment, extracts text from the images, sends it to a trained language model, and displays the summarized text back to the user on the HMD. This allows users to request summaries of text they see in the real world using voice commands, without the need to manually copy or photograph the text.

Source

Memory-efficient differentiable weight clustering for large language model compression that enables state-of-the-art performance on constrained devices. The approach employs novel memory compression techniques that reduce the memory footprint of weight clustering by applying uniquification and sharding during the backward pass. This enables significant compression ratios while maintaining accuracy, making it particularly suitable for deploying large language models on mobile devices.

Source

Creating and updating natural language models for digital assistants that allows more efficient and accurate interpretation of user requests. The method involves associating actions with user requests, determining augmented utterances based on the original request, and creating the model by mapping the augmented utterances to the associated actions. This allows the model to handle variations and unknown words in new requests. It also involves sharing learned models between applications.

Source

Grouping and presenting content items by semantically equivalent topics instead of strict grouping by topic tags to prevent confusion when topic tags are different in different languages. The technique involves determining semantically equivalent topic tags across languages and grouping content items based on these equivalent topics. This allows presenting all politics-related content together regardless of whether the topic tag is "politics" in English or "politique" in French.

Source

Efficiently updating a language model using adversarial discriminative adaptation to accurately reflect individual user idiosyncrasies without requiring large amounts of user data. The technique involves training a first language model using user data and then storing a reference version with the overall probability distribution. A second language model is trained using new user data, but constrained by the reference version's probability distribution. This updates the second model with the user's idiosyncrasies while preventing it from diverging too far.

Source

Generating sentence embeddings for natural language inputs to enable improved natural language processing tasks like semantic search, question answering, and text generation. The embeddings capture the meaning of sentences in a vector space. They are generated using language models that convert sequences of words into vectors. The embeddings can be used for tasks like finding semantically similar sentences, matching questions with answers, and pairing images with songs based on descriptions.

Source

Improving text prediction by incorporating user feedback into text prediction models. It uses reinforcement learning techniques like imitation learning to optimize a single language model for text prediction according to user's feedback. The model predicts both the next word and the user's intended action on it. If the predicted action doesn't match the user's actual action, model parameters are updated to better align with the user's behavior. This allows the model to learn user-specific idiosyncrasies and improve text prediction accuracy over time as the user interacts with it.

Source

A prompt for generating generative content can include text, images, drawings, videos, or a combination thereof, with optional parameter values indicating importance. The prompt can be structured with phrasing, style, context, and role specifications. The prompt is processed by machine learning models, including transformer-based neural networks, to generate novel content. The prompt can be multi-modal, incorporating multiple content types, and can include structured instructions.

Source

A method for displaying notifications with summary content, comprising detecting an event corresponding to application content, and displaying a notification including an automatically generated summary based on the content, where the summary includes content not part of the original content. The method further includes determining whether to display the notification based on criteria such as event type, application relevance, and content length.

Source

Computer system that assists users with tasks in 3D scenes by detecting gaze and scene data, generating plans based on semantic information and goal states, and executing actions to achieve those goals. The system can also adapt to changes in the scene by adjusting sensor parameters when the actual state differs from predicted state.

Source

Electronic devices with improved notification systems that automatically generate summaries of application content and group notifications based on relevance criteria. The system detects user events and generates notifications with summaries when content exceeds a predetermined length threshold. It also displays grouped notifications with concurrent representations of multiple notifications, enabling users to interact with individual notifications within the group. The system determines notification relevance based on user input and application content, and suppresses notifications that do not meet relevance criteria when operating in a restricted mode.

Source

A method for providing relevant search results that combines user intent with item relevance. The method generates a query vector based on a user's search query and combines it with a user account vector to establish a combined vector. The combined vector is then used to generate an output vector that is compared to item vectors to determine similarity scores. The items are ordered based on their similarity scores and displayed to the user with corresponding affordances.

Source

Providing relevant search results for search queries using AI techniques that generate stable item embeddings from multiple modalities like metadata, images, videos, and audio to provide personalized search results while respecting privacy. The method involves generating a user vector from their search query and account info, combining it with item vectors from search results, comparing to find matches, ordering by similarity, and displaying affordances. The item vectors are generated using AI models trained on stable input embeddings derived from song metadata, album art, videos, and audio.

Source

Reducing resource consumption for generating 3D environments by providing a user interface that guides users in inputting parameters for the environment instead of directly editing the 3D model. The interface allows previewing and editing a 2D version of the environment before generating the 3D version. This reduces the need to repeatedly invoke the resource-intensive 3D generation process for minor edits.

Source

A method for generating search results by interacting with multiple domains, comprising receiving an unstructured query, identifying domains to route the query to, generating structured queries for each domain using domain-specific models, aggregating results from each domain, filtering the aggregated results, and displaying the filtered results to the user.

Source

Techniques for generating responses to search queries by interacting with multiple domains. The method involves breaking down a search query into sub-questions that can be routed to specific domains for answering. Each domain generates structured queries from its models to access knowledge sources. Results are aggregated and filtered to produce the final response.

Source

Digital assistant that automatically learns user environment and determines relevant tasks through environmental analysis and audio cues. The assistant uses camera and microphone inputs to capture environment descriptions, then analyzes these descriptions in combination with audio clips to identify relevant activities. Based on this analysis, the assistant provides personalized task recommendations and audio cues to support user needs.

Source

Dynamic prompt creation for intelligent assistants that enables real-time task execution through contextual understanding. The system analyzes user requests, determines potential assistant actions, generates prompts with the action and request, and determines responses based on both. This approach enables seamless task execution while maintaining user privacy by dynamically selecting appropriate assistant actions based on context.

Source

Generating word and phrase predictions for text completion in digital assistants that provides efficient and intelligent provision of context-relevant text predictions. The method involves using a neural network system to determine text predictions based on both the text being completed and the context surrounding it. This allows the digital assistant to provide context-aware text predictions of varying lengths, improving completion accuracy compared to just predicting words based on the text alone. The neural network system has separate networks for extracting the context, determining text predictions, and assessing relevance.

Source

Continuous dialog with a digital assistant that allows more natural and seamless interactions between users and digital assistants. The system uses a multi-stage speech analysis process to handle user follow-up speech, where initial values are analyzed first and then second values are analyzed based on the context. This allows the system to better understand and respond to user requests and follow-up statements. The system also enables users to interrupt and correct the assistant if needed. The multi-stage speech analysis is done by the digital assistant itself, not just relying on speech-related cues.

Source

Allowing asymmetric retraining of upstream and downstream machine learning models without affecting consistency of output. The technique involves specifying constraints on the input data distribution that allow the downstream model to be retrained independently without requiring retraining of the upstream model. The constraints preserve the semantics of the values while allowing the distribution to change over time. This allows the downstream model to adapt to new data without affecting the accuracy of the upstream model's predictions.

Source

Neural processor circuit for performing binary comparison and reduction operations in a neural network accelerator without software control. The circuit has a neural engine to perform convolutions and a separate planar engine with a binary comparator and filter. The binary comparator applies Boolean operations to output tensors to generate conditional tensors. The filter reduces dimensions of the conditional tensors to generate channel-wise values. This allows implementing conditional operations and reducing tensor sizes directly in hardware, avoiding software for these tasks.

Source

Intelligent detection and correction of grammatical errors in user input using a neural network trained using unsupervised learning. The network takes a set of words with a grammatical error and a reference set of words, transforms the error set using the network, and reconstructs the reference set. Comparing the transformed and reconstructed sets determines if the error set is grammatical. The network is trained without labeled examples by generating error sets, transforming them, and comparing. This allows efficient training of a grammar checker using unsupervised data.

Source

A neural processor with multiple modes for processing large tensors efficiently. The processor has both neural engine circuits for convolution operations and planar engine circuits that support a reduction mode for aggregating tensor values. The planar engines reduce large tensors in multiple cycles and accumulate results in buffers. This allows processing tensors larger than the engine capacity. The reduction mode also has optimized post-processing circuits for common reduction operations.

Source

Prediction system using generative adversarial network (GAN) and distillation technology to improve prediction accuracy on devices with limited computational resources. The system involves leveraging GAN and distillation techniques like knowledge distillation and probability density distillation to enhance prediction performance. It uses a GAN framework where a student model learns from a teacher model and a cycle consistency loss is added to constrain the mappings. The student output distribution is fed back to the teacher and vice versa for distillation. The GAN-distillation combined approach provides better prediction accuracy compared to using GAN or distillation alone.

Source

Federated learning technique that uses a second prediction technique on local devices to identify ground truth data for updating the centrally-stored machine learning model. The second prediction technique is more accurate but less real-time than the first technique used on the model. By generating predictions using both techniques on local data, ground truth can be determined without requiring user input. This improves federated learning for scenarios like image classification where ground truth is not readily available.

Source

A neural processor circuit for efficient context switching between tasks in neural network processing. The circuit has a data processor with a buffer to store output data. A task manager sends configuration data to the data processor during context switch. The configuration includes masks to transfer outgoing task data from the buffer to external memory and fetch incoming task data. This allows swapping out unrelated intermediate outputs between tasks without copying all data.

Source

End-to-end training of a machine learning node that interfaces with a fixed function node whereby the machine learning node parameterizes its usage of a logical representation (e.g., software emulation) of the fixed function node. The training involves generating candidate results from the machine learning node using the fixed function node emulation, comparing them to the actual results, and modifying the machine learning node's parameters to reduce the error. This allows optimizing the machine learning node's interaction with the fixed function node for better performance or resource efficiency.

Source

Transformers, a type of neural network architecture for natural language processing and other applications, that replace the attention mechanism with a more efficient gating and pooling operation. The attention mechanism in transformers has quadratic time and space complexity with respect to the sequence length. The proposed attention-free transformers have linear time and space complexity. They replace the softmax nonlinearity with a rectified linear unit (ReLU) function to first aggregate the keys and values along the context dimension, then interact with the query. This eliminates the need for the expensive dot product attention computation. The attention-free transformers can be trained and computed with constant memory and time complexity per step, making them more efficient and scalable for long sequences.

Source

Analyzing and validating language models trained using user data that is inaccessible due to privacy restrictions. The method involves determining if predictions made by a language model trained using user privacy preserving techniques are attributable to that training process. This is done by comparing the predictions to those of a baseline model trained without user data. If the difference is within a certain range, it indicates the predictions are likely due to the user privacy training.

Source

Lookup table based activation functions for neural networks that provide higher precision than a single lookup table for a given hardware implementation. The input range of the activation function is split into smaller subranges, each estimated accurately using a dedicated lookup table. Multiple lookup tables are combined to provide the desired precision level for a specific machine learning application. This allows leveraging the limited hardware lookup table size to improve activation function approximation accuracy compared to using a single table.

Source

Training neural networks on devices like mobile phones with multi-processor training that leverages the device's neural processor for the forward pass and CPU/GPU for the backward pass. The technique involves running the computationally intensive forward pass on the neural processor which is optimized for inference, and then running the backward pass on the CPU/GPU which is better for gradient computations. This allows leveraging the strengths of both processors to train neural networks efficiently on resource-constrained devices like mobile phones.

Source

Efficiently executing machine learning models on devices with limited volatile memory by loading and executing model segments concurrently instead of waiting for full model loading. The model is stored in non-volatile memory and segmented into pieces. Segments are loaded into a fixed-size volatile memory buffer while processing another segment. This allows parallel processing and memory loading/unloading without busy waits.

Source

Accelerating matrix multiplication between a row-major input matrix and a column-major weight matrix in deep learning networks. The technique involves transposing a column of the weight matrix, storing it in a register, and dot-producting it with the input row. This allows using consecutive memory accesses for both input and weight instead of interleaving them. The transposed column is then discarded. This avoids the transpose operation overhead and reduces memory access latency when the input matrix size is large.

Source

Adaptive loss alignment (ALA) technique for improving generalization performance of machine learning models by dynamically adjusting the loss function during training. The ALA technique uses reinforcement learning to iteratively optimize the loss function parameters to directly match the evaluation metric on the validation set. This helps align the loss function to the actual performance metric over iterations, reducing the gap between training and generalization error. The technique learns to adapt the loss function while also optimizing the model weights using gradient descent. This allows optimizing the evaluation metric directly instead of just the loss.

Source