Devstral: A Review of Mistral AI's Local Coding LLM, Its Strengths, and Weaknesses

Gary Explains

Introduction to Devstral [00:00:00]

• Devstral is a new large language model (LLM) from Mistral AI, specifically designed for programming and software engineering tasks.
• Technical Specifications [00:00:07]
  • It is a 24 billion parameter model.
  • With quantization, it can run locally on machines with about 14 to 16 gigabytes of RAM.
  • This local execution is feasible on GPUs such as an Nvidia RTX graphics card with 16GB VRAM or more.
  • The video demonstrates it running at approximately 17 tokens per second on an RTX 4060Ti with 16GB VRAM.
• Origin and Core Features [00:00:43]
  • Devstral is an agentic LLM, built through a collaboration between Mistral AI and All Hands AI.
  • It is a fine-tuned version of Mistral Small 3.1.
  • It boasts a long context window of up to 128,000 tokens.
  • As a coding agent, it is text-only, with the vision encoder removed during fine-tuning to focus parameters solely on coding tasks.
  • It is designed for local deployment and on-device use, making it an ideal choice for developers.
  • It can interact with local codebases and provide fast resolution to issues when used with coding platforms like OpenHands, although this video tests it as a standalone LLM for prompt-reply interactions.

1. Rust Programming Task [00:02:07]

• Prompt Description [00:02:16]
  • The task was to write a Rust program that performs a sequence of operations:
    1. Ask the user for their name.
    2. Find the SHA256 hash value of the user's name.
    3. Find the MD5 hash value of the SHA256 value obtained in step 2.
    4. Search the resulting MD5 hash for the letter 'A' (case-insensitive) and print true or false.
  • The goal was to test Devstral's ability to follow complex, sequential instructions.

    

• Execution and Result [00:03:00]
  • Devstral successfully generated the complete Rust program, including the necessary Cargo.toml dependencies.
  • The provided code compiled and ran perfectly when tested.
  • It correctly processed the input name, calculated both hash values, and accurately determined if the letter 'A' was present in the MD5 hash.

    

2. C Bug Fixing Task [00:03:51]

• Problem Statement [00:03:56]
  • Devstral was given a piece of C code containing an integer overflow bug.
  • The bug occurs when calculating the average of two random integers (num1 and num2) by adding them first (num1 + num2) and then casting the sum to a float before dividing. If the sum of num1 and num2 exceeds the maximum value of an int, an overflow occurs before the cast to float, leading to an incorrect result.
• Devstral's Solution [00:04:38]
  • Devstral accurately identified the overflow issue as occurring during the addition of num1 and num2.
  • It provided a corrected version of the C code.
  • The fix involved casting each number to a float before adding them together: ((float)num1 + (float)num2) / 2.0f.
  • This approach ensures that the addition operation is performed using floating-point arithmetic, preventing the integer overflow.
• Result: Successful, Devstral provided a correct and robust fix for the overflow bug.

  

3. Python Programming Task [00:05:15]

• Prompt Description [00:05:17]
  • The request was for a Python program to perform several specific transformations on the phrase "Gary Explains!":
    1. Count the total number of alphabetic characters, ignoring punctuation and spaces.
    2. Multiply this count by 7.
    3. Convert the resulting product to its hexadecimal representation.
    4. Reverse the digits of the hexadecimal number.
    5. Print the final reversed hexadecimal value.
  • This task aimed to evaluate the LLM's capability to execute a series of uncommon, custom steps.
• Execution and Result [00:05:53]
  • Devstral generated the Python code according to the detailed instructions.
  • The code was run successfully in an online Python environment.
  • The output 45 was verified as correct:
    • "Gary Explains!" has 12 alphabetic characters.
    • 12 multiplied by 7 equals 84.
    • 84 in hexadecimal is 54.
    • Reversing 54 gives 45.
• Result: Successful.

4. JavaScript Animation Task [00:06:05]

• Prompt Description [00:06:07]
  • The task was to write a JavaScript program to simulate a yellow ball bouncing within a slowly rotating hexagon.
  • Key requirements included proper collision detection and keeping the ball within the shape. The use of the Three.js library was suggested.
• Initial Attempt Failure [00:06:24]
  • Devstral opted to use Three.js and provided both HTML and JavaScript files.
  • Upon execution, the screen remained black, and the browser console showed multiple "Cannot access 'velocity' before initialization" errors.

    

• Second Attempt with Debugging [00:07:01]
  • The specific error message from the browser console was given back to Devstral, asking it to fix the issue.
  • Devstral apologized and provided a revised version of the JavaScript code.
  • The updated code, however, still failed to meet the requirements:
    • It displayed a rotating shape that was not a hexagon.
    • It showed multiple balls, but they were not yellow, did not bounce, and were not contained within the rotating shape, moving off-screen.

      

• Comparative Performance [00:07:33]
  • A larger, more advanced online LLM was able to successfully generate a program that animated 21 multi-colored balls bouncing correctly within a rotating hexagon, highlighting Devstral's limitations in this domain.

    

• Result: Largely unsuccessful, indicating significant weaknesses in handling complex interactive graphics and physics with JavaScript frameworks.

5. C++ Windows Application Task [00:07:46]

• Prompt Description [00:07:48]
  • The final task required a Windows application developed exclusively with Win32 API calls (no higher-level frameworks).
  • The application needed a text input area, a status bar to display the word count of the entered text, and must be implemented in C++.

    

• Devstral's Attempt and Failure [00:08:11]
  • Devstral provided C++ code aimed at fulfilling the requirements using Win32 API.
  • However, the code failed to compile, producing numerous errors related to undeclared identifiers for standard Win32 API macros and functions (e.g., WM_CREATE, WS_OVERLAPPEDWINDOW, DefWindowProc).
  • The presenter noted that even after multiple iterations of asking Devstral to fix the compilation errors, it was unable to produce a working solution during prior testing.
• Comparative Performance [00:08:40]
  • A larger online LLM was able to successfully generate a functional C++ Windows application that correctly counted words using only the Win32 API.

    

• Result: Unsuccessful, demonstrating considerable difficulty with low-level, platform-specific API programming.

Conclusion [00:08:58]

• Devstral shows strong promise for generating code and fixing bugs in common programming languages like Rust, C, and Python.
• Its ability to run locally on consumer hardware is a significant advantage.
• However, its performance is inconsistent, struggling with more intricate or specialized tasks such as complex graphical animations (JavaScript with Three.js) and low-level platform-specific API implementations (C++ Win32 API).
• While "delightful" for simpler, well-defined coding problems, its "disappointing" results on more challenging tasks indicate areas for future improvement.