Which AI Model is Best for DevOps? A Data-Driven Comparison of 10 LLMs for Real-World Agent Workflows

DevOps & AI Toolkit

Large Language Models (LLMs) Compared [00:00]

• Introduction to the problem of choosing the right LLM for DevOps, SRE, and platform engineering. [00:07]
  • Traditional choices are often based on gut feelings, benchmarks that don't reflect production, or marketing claims. [00:27]
• Overview of the comparison methodology. [00:38]
  • 10 models from Google, Anthropic, OpenAI, xAI, DeepSeek, and Mistral were tested. [00:39]
  • Real agent workflows were used, including Kubernetes operations, cluster analysis, policy generation, and systematic troubleshooting. [00:47]
  • All tests included actual timeout constraints. [00:54]
• Initial shocking results. [00:56]
  • 70% of models could not finish their work within reasonable timeframes. [01:04]
  • A model costing $120 per million output tokens failed more evaluations than it passed. [01:10]
  • Premium "reasoning" models timed out on tasks that cheaper alternatives handled easily. [01:18]
  • Many widely discussed models failed to deliver reliable results. [01:25]
  • The cheapest model delivered better value than options costing 20 times more. [01:30]

How I Compare Large Language Models [01:54]

• The primary reason for custom comparison is to find models best suited for agents being built. [02:18]
• The secondary and more important reason is to base decisions on data, not feelings or personal experience. [02:24]
  • Distrust of standard benchmarks because models are often trained on them, leading to skewed results. [02:49]
• Custom testing approach involves: [02:56]
  • A set of functional tests validate agent correctness. [02:58]
  • These tests are run for each LLM, and output acceptability is known. [03:03]
  • Failed tests are analyzed, and data (iterations, tokens, prompts, responses, pass/fail status) is added to datasets. [03:11]
  • This data is then used to run actual evaluations and provide scoring. [03:26]
• Difference from standard AI evaluations: [03:36]
  • Most evaluations determine if something worked and how well it worked. [03:38]
  • This custom evaluation focuses on "how well it worked" since functional tests already determine "if it worked." [03:43]
  • Scoring is based on real-world agent performance, not subjective correctness. [03:55]
• Models were compared using the same agent based on the Vercel SDK. [04:05]
  • This means differences in performance from specific agents like Claude Code are ignored, focusing purely on the LLMs themselves. [04:13]
• Focus on Kubernetes-based tasks. [04:35]
  • Tests are based on Kubernetes, which is well-understood by all models, ensuring results are comparable. [04:38]
  • Results are fine-tuned for DevOps, SRE, and ops-type work. [04:46]

LLM Evaluation Criteria and Test Scenarios [05:01]

• Five key dimensions for evaluation: [05:20]
  1. Performance: Quality scores measuring how well each model handles different types of agent instructions. [05:25]
  2. Reliability: Can the model complete evaluation sessions without crashing? Measured by participation rate and completion success. [05:35]
  3. Consistency: How predictable is the model's performance across different tools? A consistent model excels across areas rather than failing in some. [05:47]
  4. Cost-performance: Quality score relative to pricing per million tokens. Raw performance is less meaningful if costs are exorbitant. [06:00]
  5. Context Window Efficiency: How well models handle large context loads. Some scenarios send over 200,000 tokens to models. A massive context window doesn't guarantee good performance if not effectively utilized. [06:12]
• Timeout constraints are critical. [06:33]
  • If a model cannot deliver results in a reasonable time, it's not useful in production. [06:35]
  • Real-world agent workflows have time budgets (e.g., 5 minutes for quick pattern creation, 45 minutes for cluster analysis). [06:40]
  • "Evaluation failed" means the timeout was exceeded, not necessarily a crash. [06:50]
• Five evaluation scenarios: [07:02]
  1. Capability Analysis: Endurance test with ~100 consecutive AI interactions over 45 minutes to discover and analyze every Kubernetes resource in a cluster. [07:11]
     • Goal: Sustain performance without degrading over extended workflows, demonstrating deep Kubernetes knowledge. [07:28]
     • 70% of models completely failed this, timing out. [07:55]
  2. Pattern Recognition: Multi-step interactive workflows within tight 5-minute timeouts. Involves expanding trigger keywords, abstracting requirements into reusable patterns, and creating templates stored in a vector database. [08:02]
     • Goal: Handle rapid, interactive refinement workflows, quick answers for pattern matching. [08:41]
  3. Policy Compliance: Schema-to-schema analysis of cluster resources within 15 minutes. Tasks involve expanding policy triggers into resource categories and generating complete, syntactically correct Kyverno policies with CEL validation expressions. [08:58]
     • Goal: Proactive governance, integrating security/compliance directly into AI recommendations. Balance depth of analysis with time constraints. [09:22]
     • 30% of models completely failed due to insufficient context windows for schema complexity. [09:47]
  4. Recommendations: Manifest generation under extreme context pressure within 20 minutes. Context load is brutal, with up to 50 large Kubernetes schemas (100-200k tokens). [09:57]
     • Goal: Transform user intent into production-ready YAML manifests via targeted clarification. Process massive schema context accurately. [10:14]
     • 50% of models failed manifest generation within timeout. [10:56]
  5. Remediation: Investigation loop for troubleshooting, running up to 30 iterations to gather data and determine root cause/remediation steps. [11:04]
     • Model receives issue description and available tools, decides which kubectl commands to run, executes them, adds output to context, and iterates. [11:15]
     • Goal: Intelligent, systematic investigation; understand cross-resource dependencies; provide root cause and remediation with risk assessment. [12:09]
     • Performance variance was extreme (2.5 seconds to >22 minutes for same investigation). [12:51]

AI Model Benchmark Results [13:23]

• 7 out of 10 models (70%) failed to complete Capability Analysis within 45 minutes, timing out. [13:27]
• Some models showed disastrous reliability issues. [14:31]
  • GPT-3.5 Pro: Only 52% participation rate, failed more tests than it passed, including pattern recognition and recommendations. Cost: $68 per million tokens (input $15, output $120). [14:34]
  • Mistral Large: Only 65% participation rate, completely failed remediation investigations. Cost: 2.7x cheaper than GPT-3.5 Pro for input, 1.8x cheaper for output. [15:33]
  • DeepSeek Reasoner: Struggled with pattern recognition and endurance, showing 70% overall performance. Cost: 20 cents input, 20 cents output per million tokens, making it very cheap but with performance issues. [15:48]
• Cost vs. Quality Analysis: [16:13] [Image 1]
  • Grok 4 Fast Reasoning: Dominated cost-performance, achieving a good score at 35 cents per million tokens (20 cents input, 50 cents output). [16:24] However, it failed the Policy Compliance test, indicating it's not suitable for generating Kyverno policies despite its speed. [16:50]
  • Gemini 2.5 Flash: Offers balanced value at 30 cents input and $2.50 output, with solid 78% overall performance and no critical failures. [17:16]
  • GPT-3.5 Pro: High cost with poor performance. ($15 input, $120 output per million tokens). [17:55] The cost is so high it skewed the chart, and it failed over half the evaluations. [18:00]
• Tool-Specific Performance Patterns: [18:27]
  • Capability Analysis: Significant performance spread (90% down to 64%). [18:40]
  • Pattern Recognition: Dramatic variance, with some excelling while GPT-3.5 Pro scored zero. [18:58]
  • Policy Compliance: Ranges from 83% to 40% (Grok 4 Fast Reasoning scored 40%). [19:13]
  • Recommendations: 42x speed difference between fastest and slowest. [19:28]
  • Remediation: Widest gap, from complete failures at 0% to near-perfect scores at 95%. [19:47]
• The context window size alone does not guarantee performance; efficiency in utilizing the space is key. [20:08]
  • Claude Haiku (200,000 tokens) scored 93% for processing massive context loads, outperforming models with 5x larger windows. [20:38]
  • Grok 4 Fast Reasoning (2 million tokens) failed tasks within timeouts, demonstrating that large capacity doesn't guarantee effective processing. [21:11]

AI Model Rankings and Recommendations [27:34]

• Important note: Rankings are specific to Ops/DevOps/SRE/Software Engineering tasks. Models scoring low here might excel in other areas like creative writing. [27:41]
• Bottom Tier (Avoid): Models scoring below 70% overall or below 80% reliability. [28:09]
  • 10. GPT-3.5 Pro: The most expensive ($15 input, $120 output) and worst performer. Failed over half of evaluations, scored zero in pattern recognition and recommendations. Not recommended. [28:10]
  • 9. Mistral Large: Marketed as Europe's answer to OpenAI. Failed all remediation investigations. Overall performance is embarrassing. [28:40]
  • 8. DeepSeek Reasoner: Hyped model, trained cheaply, good benchmark scores, but cannot handle real agent workflows. Below 70% overall. You can game benchmarks but not production. [29:01]
  • 7. Grok 4 Fast Reasoning: Its name implies speed and reasoning, but it failed complex analysis and systematic thinking tasks, with a dangerous 40% policy compliance score. Not fast or good at reasoning for these tasks. [29:23]
  • Bottom line: These models have critical failures making them unsuitable for production use, or at minimum, require extreme caution. [29:55]
• Mid-Tier (Production-ready with caveats): Models between 70-80% overall and at least 80% reliability. [29:56]
  • 6. Gemini 2.5 Pro: Solid mid-tier performer, dependable with some tool-specific weaknesses. [30:03]
  • 5. Grok 4: Elon Musk's "safe" option. Boring but reliable. [30:18]
  • 4. GPT-5 (without Pro): Similar to a reliable Toyota Camry of AI models. Consistent and predictable. [30:32]
  • Bottom line: These are dependable models with some tool-specific weaknesses, but nothing catastrophic. They will get the job done, just not always at the highest level. [30:47]
• Top Tier (High-performing, production-ready): Models consistently scoring above 85% overall, with significantly higher reliability and consistency. [30:52]
  • 3. Gemini 2.5 Flash: Best value for money, scoring 78% overall with no catastrophic failures. Excellent performance at a reasonable price. [30:54]
  • 2. Claude Sonnet: Enterprise-grade reliability. Achieved 98% reliability across all evaluations, never failed a single test. The highest reliability in the comparison. Pricier than Haiku, but worth it if failure is not an option. [31:16]
  • 1. Claude Haiku: Overall winner. Best raw performance across most scenarios at a reasonable price ($1 input, $5 output per million tokens). Achieved 87% overall, highest overall score, and leads in 4 out of 5 tool categories. Remarkable efficiency despite a smaller context window. [31:47]
  • Recommendations: Start with Claude Haiku for most work. Switch to Claude Sonnet when maximum reliability is paramount and budget allows. Use Gemini 2.5 Flash when budget is a primary concern. Avoid the bottom-tier models unless their specific weaknesses are irrelevant to your use case. [32:32]