How to Run Local AI Models: Hardware and Software Recommendations

c't 3003

Summary:

This video discusses the practicality and setup of running AI models locally.

Local AI Advancement: Local AI models have evolved significantly, now offering competitive performance, sometimes even surpassing commercial cloud providers like ChatGPT for specific tasks, especially coding.
Hardware Essentials: Memory bandwidth is paramount for AI inference. GPUs with high VRAM (e.g., NVIDIA RTX 4090/3090) offer superior speeds compared to CPUs. Multi-GPU setups (like three used RTX 3090s) can provide significant VRAM (72GB) and performance for larger models, despite NVIDIA no longer supporting NVLink on newer consumer cards. AMD Strix Halo and Apple M3 Ultra also show promising token generation rates for their cost.
Software for Local LLMs: LM Studio is recommended as a user-friendly platform for downloading, configuring, and running local language models (LLMs) on Windows, macOS, and Linux. It allows for easy adjustment of GPU offload and context length.
Model Selection and Quantization: Many "open-source" models are technically "open-weight" models, meaning their parameters can be downloaded and run locally. Quantization helps reduce model size and increase speed by reducing the precision of parameters. Users can select different quantization levels to fit their GPU memory.
Practical Applications: Local LLMs excel in tasks like RAG (Retrieval Augmented Generation), where users can chat with their private documents without uploading them to the cloud. They can also be integrated into code editors like VS Code for programming assistance.
Limitations: Local models typically don't have built-in internet search capabilities, which can affect accuracy for general knowledge queries compared to advanced cloud LLMs.

Introduction [0:00]

Local AI models have shown significant improvement.
- These models can now generate code at incredible speeds directly on a local machine.
- Sometimes, the results are even better than those from commercial providers like ChatGPT or Claude.
The video aims to guide users, especially those new to local AI, through the necessary hardware and software.
- It answers questions such as:
  - What computer hardware is needed for local AI?
  - Which AI models are effective?
  - Are local models as good as ChatGPT or Claude now?

Hardware [1:59]

Memory Bandwidth is Crucial for AI Performance [3:59]
- The speed of AI inference (called "inference" by the AI community) is largely dependent on memory data transfer rate.
- A table illustrates different memory types and their maximum data rates:
  - DDR5-5600 (Dual Channel / 2x64 Bit): ~90 GByte/s (Used in mainstream CPUs from AMD and Intel).
  - LPDDR5X-7500 (128 Bit): ~120 GByte/s (Used in Macs and iPads with standard M4).
  - LPDDR5X-8000 (256 Bit): ~256 GByte/s (Used in AMD Strix Halo Ryzen AI Max + 395, NVIDIA DGX Spark).
  - LPDDR5X-8533 (512 Bit): ~546 GByte/s (Used in MacBook Pro with 16 cores, end 2024).
  - GPU VRAM (GDDR6) (256 Bit): ~640 GByte/s (Used in Radeon RX 7970 XT).
  - GPU VRAM (GDDR6X) (384 Bit): ~1000 GByte/s (Used in NVIDIA RTX 4090, RTX 3090).
  - GPU VRAM (GDDR7) (512 Bit): ~1800 GByte/s (Used in NVIDIA RTX 5090).
  - HBM3E (6x1024 Bit): ~4800 GByte/s (Used in NVIDIA H200 (Server-GPU)).
Impact of VRAM on Speed [4:24]
- If a language model fits entirely into the GPU VRAM, it runs significantly faster.
- Example: Mistral Small 3.2 model on an RTX 4090 achieved over 40 tokens/sec.
- Running the same model on a CPU (DDR5-5600, 90 GByte/s) resulted in only 3 tokens/sec.
- The performance scales approximately with memory bandwidth: 11x bandwidth resulted in 13x token generation speed.
Challenges for Large Models and Multi-GPU Setups [5:13]
- Large language models (e.g., GPT-OSS 120B with 63GB) require substantial VRAM.
- A single RTX 5090 (32GB) is insufficient for such models.
- NVIDIA no longer supports NVLink for consumer GPUs beyond the RTX 3090 series.
  - NVLink allows multiple GPUs to share memory, effectively pooling VRAM.
- DIY Workstation Solution [6:01]
  - Enthusiasts build custom workstations using workstation motherboards and multiple used RTX 3090 graphics cards.
  - Each RTX 3090 has 24GB VRAM, allowing a 3x RTX 3090 setup to achieve 72GB VRAM.
  - Used RTX 3090s can be found for around 700 euros each.
  - This setup, though complex to cool, offers excellent token generation speed (over 50 tokens/sec) for models like GPT-OSS 120B.
NVIDIA DGX Spark vs. Consumer Hardware [6:40]
- NVIDIA's DGX Spark is a compact AI supercomputer with 128GB unified memory.
- Benchmarks (from external sources) show that for token generation, the DGX Spark (38.55 tokens/sec for GPT-OSS 120B) is outperformed by a 3x RTX 3090 setup (124.03 tokens/sec) and even Apple M3 Ultra (70.79 tokens/sec).
- AMD Strix Halo also performs comparably to the DGX Spark in token generation (34.13 tokens/sec) at a lower price point (~1800 EUR).
- Note: DGX Spark excels in prompt processing (compute-intensive stage) and has lower power consumption.
Affordable Local AI Options [8:35]
- For general LLM usage, Macs with Apple Silicon are a good alternative due to their unified, fast memory.
- A MacBook Pro with an M3 Max chip (over two and a half years old) can achieve 40 tokens/sec for GPT-OSS 120B.
- For text-based AI tasks, AMD graphics cards can also be a viable option, though NVIDIA cards are generally better for image/video generation due to CUDA.
- The speaker uses an AMD Ryzen 9 7950X3D CPU with 128GB DDR5 RAM and an RTX 4090 GPU, achieving 14 tokens/sec for GPT-OSS 120B when offloading 13 out of 36 layers to the GPU. This shows that even with less aggressive offloading, acceptable speeds can be achieved.

Software [11:16]

Evolution of Local Language Models [11:22]
- Initially, local AI models were "unusable" (e.g., Open Llama 3B provided nonsensical answers).
- DeepSeek (from China), released earlier in the year, was the first locally runnable LLM that could compete with cloud-based US models, but it required over 700GB of (preferably fast) memory.
- Recent advancements have made even small models perform well, especially for coding.
LM Studio: A User-Friendly Platform [12:39]
- LM Studio is a free application for Windows, macOS, and Linux that simplifies running local AI models.
- It allows users to download various models (e.g., GPT-OSS, Qwen, Gemma, DeepSeek) and manage their parameters.
- LM Studio is currently the speaker's preferred platform over others like Oobabooga's Web UI or O-Llama due to its ease of use and ability to adjust GPU offload and context window size.
  - O-Llama started incorporating cloud support and accounts, which detracts from the local-first philosophy.
Demonstrating Local AI Capabilities [13:33]
- Coding Example: A prompt to create an HTML webpage displaying world times in Hanover, New York, and Tokyo.
  - Qwen3 4B (2.5GB model from Alibaba) generated clean HTML/CSS/JavaScript.
  - The generated webpage featured a visually appealing design with an animated starry background (added with a follow-up prompt).
  - This small model, running on a standard GPU, produced a better visual result than larger cloud models like ChatGPT with GPT-5 Thinking, Google Gemini 2.5 Flash, or Claude Sonnet 4.5.
- PDF Analysis (RAG) [17:17]
  - LM Studio supports RAG (Retrieval Augmented Generation), allowing users to upload documents (PDF, DOCX, TXT, CSV) and chat with them locally.
  - This maintains data privacy as the documents are not sent to cloud servers.
  - Example: Uploading a VDE Tec Summit program PDF and asking for Jan-Keno Janssen's presentation schedule yielded a quick and accurate response.
  - More complex queries, like analyzing the gender ratio of speakers from the document, also worked effectively, demonstrating the model's ability to process and extract information from local files.
Understanding Model Quantization [15:13]
- The number of parameters in a model doesn't always directly correlate with its file size (in GB).
- This is due to quantization, a process of reducing the precision of model parameters (e.g., from 32-bit floating-point to 8-bit or 4-bit integers).
- Quantization reduces memory footprint and increases inference speed, but excessive quantization can degrade model quality.
- LM Studio displays different quantization levels (e.g., Q3_K_L, Q4_K_M, Q6_K, Q8_0) for each model, allowing users to choose one that balances performance and quality based on their hardware.
Additional Model Features in LM Studio [19:37]
- LM Studio displays icons indicating special capabilities of models:
  - Hammer icon: Signifies "Tool Use," meaning the model can interact with external tools or systems (e.g., browsing the internet via Model Context Protocol or controlling software like Blender).
  - Yellow eye icon: Signifies "Vision," meaning the model can process and understand image inputs (e.g., Google's Gemma 3 (12B) successfully "roasted" a person from a thumbnail).
Integration with Code Editors [20:34]
- Local LLMs can be integrated into code editors like Visual Studio Code via extensions like "Continue" or "Code Llama" as an LM Studio provider, offering local AI assistance for programming tasks.
Conclusion [20:50]
- The world of local AI is vast and rapidly evolving, offering exciting possibilities.
- The speaker invites viewers to suggest future video topics covering more advanced local AI concepts.