Apple's Innovations in Large Language Models

Memory addressing technique for distributing a large memory address space across multiple memory devices in a computer system. The technique uses hashing to select the memory device based on subsets of address bits at multiple levels of granularity. This allows flexible mapping of addresses to devices while optimizing performance by distributing accesses across devices. It also enables dynamic disabling of devices for maintenance while preserving frequently accessed data.

Source

Hardware accelerators with power-gated local memory to reduce power consumption. The accelerator and memory can be powered down between iterations to save power. However, reusable data like constants and instruction words are stored in non-volatile memory while volatile memory holds varying data. This allows initializing the non-volatile memory once then reusing it without needing to load it each iteration. The volatile memory is powered off between iterations.

Source

A neural processor with a combination of neural engine circuits for specialized neural network computations and planar engine circuits for more general computations. The planar engine circuits can handle a larger number of inputs than the neural engine circuits, and they can efficiently process combined input data by separating and duplicating values. This allows compact storage of multiple inputs and reduces computation cycles for operations involving many sources.

Source

Hardware accelerators in digital systems with local memories that can be power gated to reduce power consumption. The local memory is divided into independently powerable sections. The accelerators receive instructions that specify the amount of memory needed for that instruction. The power control circuit powers on the necessary sections for each instruction while powering off the unused sections to save power. This allows using smaller, lower-leakage local memories while still providing enough memory for each instruction.

Source

Efficient multi-layer convolution algorithm for vector processors that enables maximizing utilization of the vector processing unit (VPU) for convolution operations on multi-channel input. The algorithm selectively processes blocks of the output in accordance with their width to match the VPU's width. For blocks that are multiples of the VPU width, it processes each channel separately in the VPU. For smaller width blocks, it processes multiple channels simultaneously in the VPU. This ensures full utilization of the VPU's data paths. The algorithm involves dividing the output into blocks, determining block width, and assigning blocks to threads that process them using optimized vector instructions for the width.

Source

Generating word and phrase predictions for text completion in digital assistants that provides efficient and intelligent provision of context-relevant text predictions. The method involves using a neural network system to determine text predictions based on both the text being completed and the context surrounding it. This allows the digital assistant to provide context-aware text predictions of varying lengths, improving completion accuracy compared to just predicting words based on the text alone. The neural network system has separate networks for extracting the context, determining text predictions, and assessing relevance.

Source

Providing suggested user inputs for triggering digital assistant tasks to help users discover and request useful functions. The digital assistant analyzes user requests for tasks and device content to determine text representations of utterances that can be used to perform the tasks. These suggested utterances are then displayed as affordances on the user interface to allow easy selection and execution of the tasks. The affordances provide a way for users to discover and request tasks they may not know how to ask for, making the digital assistant more accessible and efficient.

Source

Continuous dialog with a digital assistant that allows more natural and seamless interactions between users and digital assistants. The system uses a multi-stage speech analysis process to handle user follow-up speech, where initial values are analyzed first and then second values are analyzed based on the context. This allows the system to better understand and respond to user requests and follow-up statements. The system also enables users to interrupt and correct the assistant if needed. The multi-stage speech analysis is done by the digital assistant itself, not just relying on speech-related cues.

Source

Allowing asymmetric retraining of upstream and downstream machine learning models without affecting consistency of output. The technique involves specifying constraints on the input data distribution that allow the downstream model to be retrained independently without requiring retraining of the upstream model. The constraints preserve the semantics of the values while allowing the distribution to change over time. This allows the downstream model to adapt to new data without affecting the accuracy of the upstream model's predictions.

Source

Creating and updating natural language models for digital assistants that allows more efficient and accurate interpretation of user requests. The method involves associating actions with user requests, determining augmented utterances based on the original request, and creating the model by mapping the augmented utterances to the associated actions. This allows the model to handle variations and unknown words in new requests. It also involves sharing learned models between applications.

Source

Compiler for neural processors that reduces data fetch and read operations between memory external to the neural processor and the neural engine inside the processor. The compiler stores input, output, and intermediate data in a local data buffer inside the neural processor rather than accessing system memory. This allows more efficient processing by avoiding external memory reads/writes most of the time.

Source

Neural processor circuit for performing binary comparison and reduction operations in a neural network accelerator without software control. The circuit has a neural engine to perform convolutions and a separate planar engine with a binary comparator and filter. The binary comparator applies Boolean operations to output tensors to generate conditional tensors. The filter reduces dimensions of the conditional tensors to generate channel-wise values. This allows implementing conditional operations and reducing tensor sizes directly in hardware, avoiding software for these tasks.

Source

Grouping and presenting content items by semantically equivalent topics instead of strict grouping by topic tags to prevent confusion when topic tags are different in different languages. The technique involves determining semantically equivalent topic tags across languages and grouping content items based on these equivalent topics. This allows presenting all politics-related content together regardless of whether the topic tag is "politics" in English or "politique" in French.

Source

Intelligent detection and correction of grammatical errors in user input using a neural network trained using unsupervised learning. The network takes a set of words with a grammatical error and a reference set of words, transforms the error set using the network, and reconstructs the reference set. Comparing the transformed and reconstructed sets determines if the error set is grammatical. The network is trained without labeled examples by generating error sets, transforming them, and comparing. This allows efficient training of a grammar checker using unsupervised data.

Source

A neural processor with multiple modes for processing large tensors efficiently. The processor has both neural engine circuits for convolution operations and planar engine circuits that support a reduction mode for aggregating tensor values. The planar engines reduce large tensors in multiple cycles and accumulate results in buffers. This allows processing tensors larger than the engine capacity. The reduction mode also has optimized post-processing circuits for common reduction operations.

Source

Prediction system using generative adversarial network (GAN) and distillation technology to improve prediction accuracy on devices with limited computational resources. The system involves leveraging GAN and distillation techniques like knowledge distillation and probability density distillation to enhance prediction performance. It uses a GAN framework where a student model learns from a teacher model and a cycle consistency loss is added to constrain the mappings. The student output distribution is fed back to the teacher and vice versa for distillation. The GAN-distillation combined approach provides better prediction accuracy compared to using GAN or distillation alone.

Source

Federated learning technique that uses a second prediction technique on local devices to identify ground truth data for updating the centrally-stored machine learning model. The second prediction technique is more accurate but less real-time than the first technique used on the model. By generating predictions using both techniques on local data, ground truth can be determined without requiring user input. This improves federated learning for scenarios like image classification where ground truth is not readily available.

Source

A neural processor circuit for efficient context switching between tasks in neural network processing. The circuit has a data processor with a buffer to store output data. A task manager sends configuration data to the data processor during context switch. The configuration includes masks to transfer outgoing task data from the buffer to external memory and fetch incoming task data. This allows swapping out unrelated intermediate outputs between tasks without copying all data.

Source

Efficiently updating a language model using adversarial discriminative adaptation to accurately reflect individual user idiosyncrasies without requiring large amounts of user data. The technique involves training a first language model using user data and then storing a reference version with the overall probability distribution. A second language model is trained using new user data, but constrained by the reference version's probability distribution. This updates the second model with the user's idiosyncrasies while preventing it from diverging too far.

Source

End-to-end training of a machine learning node that interfaces with a fixed function node whereby the machine learning node parameterizes its usage of a logical representation (e.g., software emulation) of the fixed function node. The training involves generating candidate results from the machine learning node using the fixed function node emulation, comparing them to the actual results, and modifying the machine learning node's parameters to reduce the error. This allows optimizing the machine learning node's interaction with the fixed function node for better performance or resource efficiency.

Source