Developing your own AI assistant for code: sprint or marathon? (AI column)
We recently passed Platform Engineering Nightwhich was opened by Igor Maslov with a report onAI and Platform EngineeringThe main thoughts from which I am already told. But there were a number of performances at the Confession. One of which was Denis Artyushin.We talk about how we develop our own assistant. NestorUnder which the umbrella combines copilot things for SDLC (But first thing is copilot for IDE code.). The main thoughts of the speech were approximately as follows:
1. What is the motivation for creating your own assistant? Denis began with an explanation, and why do your assistant, where there were a few highlights. Security solution, which is difficult to comply with when sending code to external vendors Sanctions risks, which are clamping in the current difficult geopolitical conditions The need for integration with the internal infrastructure that has already been implemented 2. How to approach the question: as a sprinter or as a styer? First, Denis told about the rapid experiments in the sprinter mode - so the guys launched commit messages, where from the idea to the finished prototype passed one Friday evening. The scheme was simple: take diff, add examples, form a promt and give to the model. And then he said that code completion requires a serious infrastructure with A/B tests, analysis of user behavior and search for similar code in the code base, that is, you need a systematic and measured approach. 3. What does Nestor do? The main features are now quite classic
- Autocompletion of code Chat with search by internal documentation
- Generation of commit messages
- Prototype of code editing
- Test generation (now in development) 4. What are the assistant's usage metrics? They are already impressive: WAU: 5k, MAU: 9.5k. In some professions, penetration has exceeded 60% and acceptance rate has risen to 30% Go (It is time to switch to go, because assistants on it best assist 😉) 5. How did you search for documentation? In fact, you can see about the search in the report of Yegor Prokhorenko, about which I have already told. But Nestor went something like this: structured documentation → brief descriptions → embedding → search base. They also used the Russian-language TP Pro model for better response quality. 6. How to solve the problem of scaling UI in the plugin for IDE? Initial chats on TypeScript/CSS and Swing were not scalable. Switched to React and Compose made it easy to add new functionality. 7. How to balance themes in the development team? Sprints help test hypotheses and keep Olympians motivated in the team (A significant part of the Nestor team is programming Olympiads.). But marathons are needed to create a quality product. 8. What quality metrics do you pay attention to? Metrics are different.
- Adoption Nestor
- Acceptance rate (stripped) Internal benchmarks for searching documentation
- User feedback and suggestions for improvement. 9. What development plans? There are many plans, but now they seem so.
- Full code editing
- Test generation Agent platform for all departments of the company Integration into other ecosystem products (data platform HelicopterObservability platform Sage)
If you sum up and answer the question in the title of the report, then creating your own assistant requires a combination of quick experiments. (sprint) for hypothesis testing and system operation (marathons) for a quality product.
#AI #Software #Engineering #Architecture #Agents