AI assistants when working with code. Evgeny Kolesnikov - Platform Engineering Night (AI column)

Cool. speech Evgenia from the Yandex Infrastructure team, in which he shares deep thoughts about the development of AI copilot tools. Zhenya spoke at Platform Engineering Night in T-Bank. I told you about my colleagues from there:AI and Platform Engineeringfrom Igor Maslov andDeveloping your own AI assistant for code: sprint or marathon?Denis Artyushin. The guys talked about our approaches to integrating AI into SDLC) and it is interesting to compare the thoughts of those reports with the ideas of Zhenya, which I have tried to explain below.

1. Reality of development The developers write the code for everything. 40 minutes 120 Minutes a day, on average, only 40 lines of code a day. The main problem is not the speed of printing, but the complexity of thought processes that go on at three levels.

• Mental model - what do we want to do?
• Semantic model - how do we do it?
• Syntactic model - the code itself AI now helps mainly in the latter phase, which explains the limited effect. 2. Developer modes of work There are two main regimes:
• Flow - flow state, when the code "flying from under the fingers". Interestingly, in DevEx, the Flow framework is one of the components, by the way, I did survey whitepaper about him
• Exploration - search for information in documentation, the Internet, communication with AI Understanding these modes is critical to the effective use of AI tools. 3. What developers want from AI According to Evgeny, the expectations of engineers are Transfer to AI routine operations, such as writing unit tests
• Communicate in natural language with subsequent clarification through prompts Obtain deterministic results from nondeterministic genAI Interestingly, Google had a whitepaper with that name.What Do Developers Want From AI?- I'm his. done before, and then recorded an episode of the podcast "Research Insights" with my colleague, Kolya Bushkov, where we stripper whitepaper 4. Business priorities Businesses want time-to-market reduction, cost reduction, and predictability. But it's usually all about cutting costs when they say,90Percent of the code will be written by AI.” But often that doesn't mean getting fired. 90The percentage of programmers increases the productivity of existing teams. Evgeny gave the example of Dario Amodei with his theses from the quote above - and I handler performance before 5. Measuring effectiveness Be critical of numbers like “increased productivity” 55%. Productivity is a vague term that depends on many factors. There is no single way to accurately measure the benefits of AI tools. I wonder if I've been a couple of times. performer thematicallyWhy Do Developer Productivity in a Large Company?" 6. LLM Product Using the latest language model does not guarantee product success. UX/UI, proper promotion and workflow integration are often more important than choosing a particular model. 7. Correct metrics It is worth measuring NPS, CSAT in conjunction with retention (at SourceCraft from Yandex 60-70%)Cycle time, lead time and impact on business metrics. The user happiness metric is an integral indicator of the acceptance / rejection of prompts. 8. Reducing hype is a good thing. Za. 2023-2024 Interest in AI has fallen in some areas and that’s a good thing – developers are beginning to realistically assess the capabilities and limitations of AI tools, leading to more efficient use. 9. The Future: From Generation to Agents Development is now moving from generative models to agency models. Agents proactively solve problems, but so far are extremely unreliable. The next step is to make agents more reliable and predictable. The deeper the integration of AI into a company’s infrastructure, the greater the gain.

Evgeny believes that AI assistants are definitely useful, but it is important to understand their limitations and properly integrate into the workflow, rather than chasing after hype.

#AI #Software #Engineering #Architecture #Agents