The Apple M5 chip features a 16-core Neural Engine, a crucial component often overlooked in standard benchmarks, yet it represents a significant, long-term investment by Apple.
Apple M5 Chip highlighting the 16-core Neural Engine
[
00:00:33
]
Apple's vertical integration, controlling chip design, memory architecture, OS, and developer frameworks like Core ML, allows for deep optimization, such as unified memory and embedded neural accelerators within the GPU. Since its introduction in 2017 with 2 cores, the Neural Engine has grown to 16 cores, achieving 38+ trillion operations per second in the M5, alongside GPU accelerators. This expansion reflects Apple's strong commitment to on-device AI, prioritizing privacy, cost-effectiveness, and low latency by processing AI workloads directly on the device. The industry trend, mirrored by companies like Qualcomm (Snapdragon X2 Elite with 80 TOPS) and Intel (Lunar Lake with 48 TOPS), also indicates a widespread shift towards integrating dedicated AI hardware. This ongoing investment in local AI compute aims to support future applications that will leverage powerful on-device machine learning capabilities, even as software development catches up to the hardware's potential.
The Underrated Neural Engine in Apple's M5 Chip [00:00]
The new MacBook Pro's M5 chip features a 16-core Neural Engine that is often overlooked in traditional performance benchmarks like Geekbench scores and render times. [00:00]
Apple has consistently expanded the Neural Engine's capabilities since 2017, dedicating more transistors to this component in each generation. [00:32]
This trend is not exclusive to Apple; major chip companies like Qualcomm and Intel are also heavily investing in dedicated AI hardware, signaling a broader industry shift towards on-device AI. [00:36]
Apple M5 Chip highlighting the 16-core Neural Engine
[
00:00:33
]
How Apple Transitioned from Intel to Custom Silicon [02:12]
In 2020, Apple began its transition away from Intel processors, releasing the M1, its first custom silicon for Mac. [02:14]
Initially viewed as a gamble, Apple's custom silicon program has proven highly successful, with the M-series chips now powering all Macs. [02:28]
Traditional PCs have separate memory pools for CPU and GPU, requiring data copying between them, which consumes time and power. [03:25]
Apple's M-series chips feature unified memory, allowing the CPU, GPU, and Neural Engine to share a single pool of memory, eliminating the need for data copying and improving efficiency. [03:37]
Traditional PC memory architecture with separate CPU and GPU memory
[
00:03:30
]
This architectural decision is possible because the teams designing the chip and the OS memory scheduler work in tandem. [03:44]
The Neural Engine in the M5 chip is a 16-core dedicated component that benefits from Apple's vertical integration. [03:53]
It resides on the same die as other components, accessing the unified memory pool, and runs software developed with Apple's Core ML framework. [03:59]
Apple M5 Chip diagram showing CPU, GPU, and 16-core Neural Engine
[
00:04:05
]
A significant new feature in the M5 is the embedding of neural accelerators directly within the GPU cores. [04:25]
This allows the GPU to natively handle AI workloads in addition to the Neural Engine. [04:37]
M5 Chip with Neural Accelerator within each GPU Core
[
00:04:35
]
Apple claims the M5 delivers over four times the peak GPU compute performance for AI compared to the M4. [04:40]
Apple's claim of M5 GPU delivering over 4x peak AI compute compared to M4
[
00:04:45
]
This coordinated design decision is a direct result of Apple controlling the entire pipeline, enabling them to route AI workloads efficiently between the Neural Engine and GPU. [04:48]
Evolution of Apple's Neural Engine: 2017-2025 [05:21]
Apple's investment in the Neural Engine spans eight years, with continuous generational improvements: [05:21]
2017 (A11 Bionic): Introduced with 2 cores, capable of 600 billion operations per second (B OPS). Primarily powered Face ID and Animoji, with no public developer access. [05:26]
Animoji feature powered by the A11 Bionic Neural Engine
[
00:05:43
]
2018 (A12 Bionic): Jumped to 8 cores, achieving 5 trillion operations per second (T OPS), 9 times faster with a tenth of the power. Apple released Core ML, opening the Neural Engine to developers. [05:46]
2020 (A14/M1): Core count doubled to 16, reaching 11 T OPS. The M1 chip brought an identical 16-core Neural Engine to the Mac for the first time. [06:06]
2025 (M5): Features a faster 16-core Neural Engine plus the new neural accelerators embedded in the GPU. [06:24]
Apple Neural Engine 8-year evolution (2017-2025) showing core count and TOPS increase
[
00:06:30
]
This consistent dedication of expensive chip real estate to the Neural Engine over eight years demonstrates Apple's long-term strategic bet on AI. [06:39]
Every square millimeter given to the neural engine is a square millimeter not given to the GPU or CPU, a trade-off Apple consistently makes. [06:48]
System on a Chip diagram showing allocation of chip real estate
[
00:06:53
]
On-Device AI vs. Cloud AI: Two Philosophies [07:16]
There are two main philosophies regarding where AI compute should reside: [07:17]
Meaningful AI workloads run locally on the user's device. [07:57]
Apple is strongly advocating for this approach. [07:55]
Pros: Private by default (data never leaves the machine), essentially free after device purchase, zero latency for tasks like coding assistants. [08:02]
Cons: Limited by the processing power of the user's device. [08:31]
Flowchart comparing Cloud Inference and On-Device Inference
[
00:08:05
]
The future likely involves a hybrid approach, with some workloads running locally and others in the cloud, with developers making the decision based on specific application needs. [08:40]
Apple Intelligence rewriting an email on-device
[
00:08:13
]
Transcribing voice memos or cleaning up photos.
Apple Intelligence cleaning up a photo on-device
[
00:08:15
]
Photo editing tools running models on the neural engine.
Photo editing tool demonstrating on-device AI
[
00:09:25
]
Health apps analyzing sensitive data locally. [09:31]
Coding assistants running suggestions with zero latency.
Developer using a coding assistant running suggestions locally
[
00:09:38
]
Apple's M5 offers a comprehensive system for on-device AI, including a dedicated Neural Engine, GPU neural accelerators, unified memory, and the Core ML developer framework to automatically route workloads. [08:51]
Key features of Apple M5 for On-Device AI
[
00:09:15
]
This widespread investment signals a collective industry belief in the importance of on-device AI. [07:10]
The chip itself represents a company's true commitment to a technology. [09:57]
Silicon wafer representing the physical commitment to chip development
[
00:10:22
]
Software is beginning to catch up to hardware capabilities, with initiatives like Apple Intelligence, Microsoft Copilot Plus, and Intel Lunar Lake's AI PC marketing. [10:32]
Intel Lunar Lake running Stable Diffusion AI image generation
[
00:10:40
]
Many neural engines currently spend time waiting for tasks, indicating a significant opportunity for developers to build new applications leveraging this on-device compute. [10:47]
The continuous expansion of AI engines in each generation suggests a long runway for new and innovative applications that will leverage this on-device compute. [11:08]
The M5 chip is not just a powerful laptop component but a strategic piece in an industry-wide bet that AI belongs on the device, under user control, prioritizing security and privacy. [12:06]
