This video offers a reality check on the adoption and impact of Large Language Models (LLMs) in software engineering by 2025, contrasting executive hype with the nuanced experiences of engineers across different sectors. AI dev tool startups heavily use LLMs, with Anthropic's Claude Code and Windsurf reporting high percentages of AI-written code. Big Tech, like Google and Amazon, is deeply integrating LLMs into their custom development stacks, preparing for a massive increase in code volume and leveraging API-first cultures. AI startups show varied adoption, with some actively sharing AI tips and others finding LLMs slower for novel problems. Experienced independent engineers are increasingly enthusiastic, citing improved productivity and a fundamental shift in programming. Despite high usage rates (50-62% weekly), time savings are moderate (3-5 hours/week), and team-level utility is still evolving. LLMs are seen as a step-change in software development, comparable to shifts from assembler to high-level languages.
Overview of LLM Usage Across Different Organizations
[
00:19:28
]
Introduction: Executive Hype vs. Ground Reality [00:00]
The speaker highlights a significant disconnect between enthusiastic headlines from tech CEOs regarding AI's impact on coding and the actual, more nuanced, reality experienced by engineers.
Executive Claims:
Microsoft CEO: 30% of all code written by AI [00:00:21].
Anthropic CEO: All code will be AI-generated in a year, or 90% in 3-6 months [00:00:38].
Google Chief Scientist: AI could perform at a junior coder's level in a year [00:00:53].
Engineer Experiences (Reality Check):
An autonomous AI agent (Devon, $500/month) introduced a bug that cost $700 extra in events [00:01:14].
Microsoft Copilot agents failed spectacularly to land fixes in complex .NET codebases, often adding broken tests [00:01:41]. Microsoft was transparent about these failures, unlike other startups.
The speaker's goal is to provide a "temperature check" by gathering insights from various types of software engineers on their actual AI tool usage.
Companies actively selling AI development tools internally use them heavily, demonstrating high adoption rates.
Anthropic:
Engineers rapidly adopted Claude Code (CLI-based AI coding tool) internally, with 90% of Claude Code's product being written by Claude Code itself [00:04:17].
Public usage of Claude Code increased 40% on launch day and 160% in less than a month [00:04:33].
Anthropic open-sourced Model Context Protocol (MCP) in November (speaker's typo, likely 2023 or earlier, given adoption timeline). MCP allows AI agents and IDEs to connect to various tools (e.g., GitHub, databases) via a common protocol [00:04:51]. Major companies like OpenAI, Google, and Microsoft adopted MCP support by March/April, leading to thousands of active MCP servers [00:05:30].
This diagram illustrates how AI agents and IDEs (MCP Clients) can interact with various tools like databases, GitHub, and Google Drive through the Model Context Protocol (MCP), enabling broader automation.
Windsurf: Reports that approximately 95% of their code is written using Windsurf's agent (Cascade) or passive tabbing (Windsurf Tab) [00:05:52].
Cursor: Estimates 40-50% of their code is written using Cursor [00:06:09].
Big Tech: Google and Amazon's LLM Integration [06:28]
Big tech companies are heavily investing in and increasingly using LLMs, often integrating them deeply into custom internal stacks.
Google:
Everything is custom, including their IDE (Cider/Cider-V), code review tool (Critique), and code search [00:06:42].
LLMs are integrated across their entire stack:
Cider (IDE): Autocomplete and chat with the IDE [00:07:22].
Code Search: LLM support for querying the codebase and retrieving relevant parts [00:07:42].
Usage of AI tools was low a year ago but has rapidly increased [00:07:55].
Google takes a cautious approach, aiming for high reliability to build engineer trust [00:08:08].
Other internal AI tools include Notebook LM (chat with documents), LLM prompt playground, and MoMA search engine (knowledge base with LLMs) [00:08:21].
Many internal LLM tools are being built, driven partly by leadership interest and funding opportunities [00:08:40].
A former Google SRE engineer revealed that Google is preparing for a 10x increase in lines of code making it to production, beefing up infrastructure, deployment pipelines, code review, and feature flagging [00:09:08].
Google's Preparation for 10x Lines of Code
[
00:09:19
]
A quote from a former Google SRE engineer stating that Google is preparing for a tenfold increase in lines of code making their way into production, indicating significant internal changes to infrastructure and tooling.
Amazon:
Almost all developers use Amazon Q Developer Pro, which is particularly effective for AWS-related coding and general programming [00:09:41]. Engineers are surprised by its lack of external recognition.
Internally hosted Claude is used extensively for writing tasks, including PR/FAQ documents (Amazon's 6-pager press releases) and performance feedback [00:10:09].
Leveraging its "API-first" culture (mandated by Jeff Bezos in 2002), Amazon is rapidly adding MCP support to internal tools and websites [00:10:37]. This facilitates widespread automation of tasks like ticketing, emails, and internal systems, which developers are finding highly beneficial [00:11:43]. This suggests Amazon might become "MCP-first" by 2025 [00:12:06].
Amazon's Internal Automation with MCP Support
[
00:11:49
]
Key points detailing how most internal tools and websites at Amazon now have MCP support, enabling widespread automation of tasks like ticketing and emails, which developers appreciate.
The experience with LLMs in AI startups not focused on AI dev tools is more varied.
incident.io:
Initially an on-call platform, now transitioning to AI-first [00:12:22].
Their team massively uses AI to accelerate work, sharing tips and tricks in Slack [00:12:40].
Engineers successfully use MCP for well-defined tickets, allowing agents to generate a first pass of code [00:12:49].
A new favorite trick is prompting LLMs to ask for options (e.g., different coding approaches, error explanations) [00:13:13].
Claude Code, despite being recently released, is used regularly by their entire team [00:13:44].
Incident.io Engineer's Tip for Prompting LLMs
[
00:13:19
]
A Slack message showing an engineer's "new favorite trick" of prompting LLMs to ask for options, such as different code implementations or explanations for errors, to leverage LLM creativity and problem-solving.
A Biotech AI Startup (Unnamed):
Uses AI/ML models to design proteins, with extensive automated numerical/ML pipelines [00:14:00].
Despite experimenting with several LLMs, none have stuck because it's still faster to write correct code manually than to review and fix issues in LLM-generated code [00:14:18].
This company feels they might be in a "weird niche" where LLMs aren't as effective, especially for novel software development [00:14:38].
Seasoned Software Engineers: The Resurgence of Excitement [15:14]
Experienced independent software engineers, who are passionate about coding, are finding renewed excitement and increased productivity with LLMs.
Armin Ronacher (Creator of Flask):
Initially skeptical, he now prefers being an "engineering lead to a virtual programmer intern" (AI agent) [00:15:20].
Claude Code "got shockingly good," helping him overcome the hurdle of not accepting LLMs [00:16:02].
The tools' ability to run code and get feedback helps avoid model faults (hallucinations) [00:16:16].
The title of Armin Ronacher's blog post, "AI Changes Everything," reflecting his profound shift in perspective on AI's impact on software development.
Peter Steinberger (Creator of PSPDFKit):
Reports a "spark returns" feeling, having not been "this excited, astounded, and amazed by technology in a very long time" [00:16:28].
Believes LLMs represent a tools "inflection point" where they "just work" [00:17:03].
Languages and frameworks matter less because switching is incredibly simple with LLMs [00:17:12].
A capable engineer can achieve 10-20x the output compared to before [00:17:22].
Peter Steinberger's Enthusiasm for AI
[
00:17:28
]
A tweet from Peter Steinberger expressing that his tech friends are in awe of AI and find it "mind-blowing technology," leading to difficulty sleeping from excitement.
Birgitta Böckeler (Distinguished Engineer at Thoughtworks):
Views LLMs as a new tool that allows specifying things in an unstructured way and can be used on any abstraction level (from low code to Assembly) [00:17:49].
Considers this a "lateral move," not just another abstraction layer, making LLMs highly exciting [00:18:18].
Simon Willison (Creator of Django):
Characterizes current LLM coding agents as truly functional, capable of running in loops, executing compilers and tests, and achieving goals [00:18:39].
The model improvements in the last six months have transformed them from "fun toy demos to useful on a daily basis" [00:19:16].
Open Questions and The Future of Software Development [19:45]
Despite the excitement, several critical questions remain about LLM adoption and impact.
1. Why are founders and CEOs more excited than engineers? [19:56]
Founders (like Zach Lloyd of Warp) observe that designers and PMs are more enthusiastic adopters than senior engineers, who struggle to consistently use AI dev tools [00:20:05]. This might be due to financial incentives and the vision for the future.
2. How mainstream or niche is LLM usage across devs? [20:35]
A DX study of 38,000 developers found that the median organization has 5 out of 10 devs (50%) using AI tools weekly [00:20:53]. Top companies show 6 out of 10 devs (62%) usage [00:21:07]. This is significant growth in a short time, but not universal daily adoption.
Data from a DX study of 38,000 developers, indicating that 50% of devs in median organizations use AI tools weekly, with top companies reaching 62%.
3. How much time do LLMs actually save, when used? [21:34]
While some engineers report 10-20x output increase, the DX study found that LLMs save an estimated 3 to 5 hours per developer per week [00:21:42]. The question remains what developers do with this saved time.
DX Study: Hours Saved Per Developer/Week
[
00:21:48
]
A bar chart from a DX study illustrating that LLMs save developers an estimated 3 to 5 hours per week.
4. Why does it work so much better for individuals than teams? [22:00]
Laura Tacho, CTO at DX, notes that these tools are excellent for individual developers but "not yet that good at the organizational level" [00:22:04].
Overall Summary:
CEOs and founders of LLM companies are enthusiastic, aligning with their business goals [00:22:14].
Big tech is investing heavily and experimenting with AI tools [00:22:20].
The most significant observation is that experienced, independent software engineers are finding great success with LLMs and want to use them more [00:22:27].
A Step-Change in Software Building:
Martin Fowler predicts LLMs will change software development to a degree similar to the shift from assembler to high-level programming languages [00:22:44]. The key distinction is that LLMs not only raise the level of abstraction but also introduce non-deterministic tools into programming [00:23:16].
Martin Fowler's quote stating that LLMs will change software development to a similar degree as the shift from assembler to high-level programming languages.
Kent Beck (52 years programming) states he's "having more fun programming than I ever had in 52 years" due to LLMs [00:23:24]. He compares the impact of LLMs to historical shifts like microprocessors, the internet, and smartphones [00:24:20]. LLMs have dramatically shifted what is "cheap" and "expensive" to do in software development, requiring experimentation [00:24:44].
Kent Beck's Observation on Programming Enjoyment
[
00:23:38
]
Kent Beck's striking statement: "I'm having more fun programming than I ever had in 52 years," attributing his renewed enjoyment to LLMs.
Conclusion: It's time to experiment more with LLMs to understand what works and what doesn't, adapting to the changing landscape of software development [00:24:54].
