[2/2] AI in software engineering at Google: Progress and the path ahead (AI column)
Finishing. story About using AI in development, I want to talk about what they have achieved and what next steps they see.
- In the end, the guys at Google learned the following: They achieved the greatest effect with UX, which naturally fits into user workflows. In all of the above examples, the user is offered an offer that takes him to the next step in the workflow with a single tab or click. Experiments requiring the user to remember to activate the function did not scale. They observe that with AI-based suggestions, the author of the code is increasingly becoming a reviewer, and it is important to find a balance between the cost of the review and the added value of that sentence. It is usually a matter of compromise through targeted acceptance rates. Fast iterations with online a/b experiments are key, as offline metrics are often just rough indicators of user value. By bringing AI tools to internal tuning, authors benefit from the ability to easily run and iterate, measure usage data, and directly ask users about their experiences through UX research. High-quality data obtained are crucial for improving model quality.
- The interaction funnel looks something like this.
- Event of SWE acting in tooling
- Trigger for ML prediction
- Confident ML Prediction
- Generated ML Prediction
- Discovery of ML Suggestion
- Applied (it's a targeted action) 8 ) The authors note that the autocompletion of code based on ML gave a powerful boost in the development of software. Of course, these models can be improved further, but the authors see the prospects of expanding assistance for testing, understanding and support of code, and they see two trends. Human-computer interaction has shifted to natural language as an accepted modality, that is, there is a transition to using the language as an interface for software development tasks, as well as a gateway to meet the information needs of software developers, and all this is integrated into the IDE. Automation of larger ML-based tasks (from diagnosis of the problem to correction) They began to show signs of feasibility. These capabilities are driven by innovations in the use of agents and tools that enable systems that use one or more LLMs as a component to perform a larger task. The authors suggest teaming up to create benchmarks to assess these next-generation capabilities. They are mentioned as examples.
- SWEBench Solving issues on Github
- Benchi from Cognition AI - and Cognition AI launched Devin (end-to-end software agent)The one who tries to solve e2e problems
#Software #AI #ML #Engineering #Processes #DevEx