What Do Developers Want From AI?

Alexander opens another reading from the Developer Productivity for Humans series — a couple of papers from it have already been discussed on the podcast — and brings in his T-Bank colleague Kolya Bushkov. Kolya works in the R&D centre: his area is called R&D of engineering productivity, and the method is to apply artificial intelligence and machine learning to raise that productivity, while also measuring productivity itself and the demand for particular features, because you cannot manage what you do not measure. The paper, as Alexander describes it, dates from the summer of 2024 and is not technical but rather conceptual: everyone is rolling tools into the SDLC, yet asking developers themselves where it would be more useful to embed them is often forgotten, so the authors take a step back. Kolya liked it overall, obvious points included, but what stands out is how metaphorical the text is: metaphors, metaphors, metaphors.

The first of them is the faster horse attributed to Henry Ford; the paper notes the line is most likely not his, though the substance matters more than the authorship: neither developers nor other participants in product surveys are always ready to say what they want, and changes that are too large are better designed around the users of the future. Alexander preferred the parallel with electricity: once everyone talked about electrification, now the stage is the same — let us add it here, let us add it there — and in the end AI will accompany us everywhere. The authors are from Google, whose interests come first, but much of it transfers to other large technology companies, even one like T-Bank. The local goal of the paper is not disruptive products built on a grand vision, but local changes: ask engineers where it hurts, cut the routine and lift developer experience, which in Kolya's view pulls productivity along with it.

The authors' method in the paper is a dual one: quarterly surveys plus logs of how engineers actually use the tools. That puts perception — what it feels like when an assistant helps with review or offers tests — next to an objective metric such as the share of accepted suggestions; interviews then explain why someone dislikes an assistant, either because it suggests the wrong things or because the person is against change in principle. At T-Bank the same practice is in place, enough data has been collected to lean on, and the hybrid of surveys and objective data is the best available; a formula surfaces in the conversation: objectivity is well-organised subjectivity. Behind papers like this one, Alexander notes, sits a research team of about 152 people — he does not remember the number exactly — that optimises rather than revolutionises: twice as much infrastructure speeds a build up not twofold but by some 15 to 20 per cent, and from there it becomes an economics problem.

Next comes the three-level model: assisting existing capabilities, extending them, and full delegation of the work to the machine, explained in the paper through cars. An anti-lock braking system brakes the way a human in an emergency cannot, and that is the analogue of code completion: inline and block suggestions already work decently at T-Bank and across the market, and the scenario predates the GPT moment, starting on models of the GPT-3 generation. That is where we are now, gradually moving to the second level — like blind spot monitoring and parking sensors, which compensate for physical limits of human perception. For code assistants that means review and suggestions inside the CI pipeline, which the company is experimenting with right now. Review is a large task with many parameters, so the aim has to be picked: where models are needed, where static analysers and linters suffice, and where the call is better left to the human, who is not removed from the process but whose involvement is minimised.

Alexander recalls a printed Google paper in which review comments are still written by people, but the author of the change receives, along with the comment, a suggestion of how to implement it: rename this, use that instead. Something similar exists at T-Bank — an analyser leaves the comment about a vulnerability and a model proposes the fix — although Alexander could not recall the product's current name. Kolya agrees these are probably among the most mature solutions in the code review category, yet his team works from a more recent paper by another large technology company: it offers a taxonomy of review topics with four top-level directions, one of them vulnerabilities in code; they decide to attach the link to the episode. Vulnerabilities are named in the paper as an important blocker for adoption: generated code turns out more vulnerable, while people are more confident that it is sound. Finding a vulnerability is currently harder than fixing it: models cope with the fix, and the bet is on them getting smarter.

All of this, Alexander adds, works where the guard rails of mature processes exist: a static analyser that parses syntax trees rather than text, and tests of every kind that a generated version can be run through to confirm nothing got worse. Formal checks have a ceiling, Kolya replies: part of what humans look at in review cannot be formalised into a linter, so a model that beats an analyser on a large sample is worth putting in its place, even at the price of interpretability — though nobody will roll anything out without offline metrics. The top three productivity problems at Google are technical debt, poor documentation and learning a new platform; debt is too poorly defined to be tackled whole, and a reading with Dima Gaevsky covered how Google decomposed it and over three or four years moved it in the surveys from first place to a middling level. With documentation the trouble is not search but that the best descriptions come from those who integrated with a system rather than from its authors: hence the bet on a chat over an indexed knowledge base that can refuse to answer, turning logged questions into a documentation plan.