AI Coding Assistants: A Look Ahead — Evgeny Kolesnikov, Platform Engineering Night (Category AI)

A great talk by Evgeny from Yandex Infrastructure, sharing thoughtful ideas about the development of AI copilot tools. He gave it at Platform Engineering Night at T-Bank. I have already covered my colleagues’ talks from that event: Igor Maslov’s “AI and Platform Engineering” and Denis Artyushin’s “Developing Your Own AI Coding Assistant: Sprint or Marathon?.” They described our approaches to integrating AI into the SDLC :) It is interesting to compare those with Evgeny’s ideas, which I have tried to set out below.

1. The reality of development According to the statistics presented, developers write code for only 40–120 minutes a day and commit an average of just 40 lines daily. The main problem is not typing speed but the complexity of thinking at three levels:

• Mental model: what we want to do.
• Semantic model: how we will do it.
• Syntactic model: the code itself. AI mostly helps at the last stage, which explains its limited impact.

2. Developers’ working modes There are two main modes:

• Flow: the state where code seems to fly from your fingers. Flow is also a component of the DevEx framework; I reviewed its whitepaper.
• Exploration: looking for information in documentation or online, and talking with AI. Understanding these modes is crucial to using AI tools effectively.

3. What developers want from AI Evgeny identifies these expectations:

• Delegate routine work, such as writing unit tests, to AI.
• Communicate in natural language, then refine the request through prompts.
• Get deterministic results from nondeterministic generative AI. Google published a whitepaper literally called “What Do Developers Want From AI?” I reviewed it, then recorded a Research Insights podcast episode with my colleague Kolya Bushkov in which we discussed it.

4. Business priorities Businesses want shorter time to market, lower costs, and predictability. Yet claims that “90% of code will be written by AI” usually focus on cost cutting. This often means making existing teams more productive, rather than firing 90% of programmers. Evgeny cited Dario Amodei’s remarks behind that quotation; I discussed that appearance before.

5. The difficulty of measuring effectiveness Treat claims such as “55% higher productivity” critically. Productivity is an ill-defined term affected by many factors. There is no single way to measure the benefits of AI tools precisely. I have given talks a couple of times on “Why Work on Developer Productivity in a Large Company?”

6. LLM ≠ Product Using the latest language model does not guarantee product success. UX/UI, good prompting, and workflow integration are often more important than the particular model.

7. The right metrics Measure NPS and CSAT alongside retention—60–70% for Yandex’s SourceCraft—as well as cycle time, lead time, and effects on business metrics. The user-happiness metric combines acceptance and rejection of suggestions.

8. Less hype is good Interest in AI fell in some areas during 2023–2024, and that is healthy: developers are assessing tools’ capabilities and limitations more realistically, leading to more effective use.

9. The future: from generation to agents Development is moving from generative models toward agents. Agents tackle tasks proactively, but remain very unreliable. The next step is to make them more reliable and predictable. Deeper integration with company infrastructure brings greater gains.

Evgeny’s conclusion is that AI assistants are definitely useful, but we need to understand their limits and integrate them properly into our work rather than chase hype.

#AI #Software #Engineering #Architecture #Agents