Measuring Developer Experience With a Longitudinal Survey

Artyom runs the product and project office in Yandex's city services — taxi, market, grocery, delivery, food — and owns the developer platform as a product, whose goals are development speed and quality and lower cognitive load. He calls the survey one of the most important instruments for practically any big tech company: it shows how developers feel about current processes and what state tools, platforms and engineering culture are in. He deliberately draws a line against HR surveys on satisfaction and mood: the topics overlap, but the conversation has to be about the engineering side. Alexander adds that Google's experience here is phenomenal — the surveys have run since 2018, the methodology kept improving, and the trend data told them a great deal about covid and remote work, and about how AI wedged itself into engineering practice. A survey is also quick to set up: Google has long articles on pulling activity metrics out of logs, and that looks far harder.

Hence a starter pack for anyone starting out. Artyom, who is preparing his own talk on Developer Experience and platforms, suggests simply running a survey — in Yandex Forms or any other tool: it costs little, and the results can be thrown at a modern neural network for help with the analysis. Google's questions, as he recalls, are either public or easy to find online, and they suit most organizations; then you can see whose pain is where and prioritize the work. The other side is the duty to show a reaction: he has seen companies take items into the backlog without ever linking them to the survey, and people will not work the connection out themselves. It has to be said explicitly — out loud, on a dashboard, in a presentation, in a post on the internal portal. Two more observations: towards the end people click down the middle of the scale, and a free-text field is a must — engineers, unlike managers, willingly complain in it.

The paper itself, by Alexander's reckoning, is from 2023 or 2024, so the team has some six years behind it, and it lays out an end-to-end process with the duration of every stage. It works like this: each quarter a new pair owns the survey — the researcher drives the evolution of the research programme, the engineer the automation of infrastructure, builds and data processing. The pair documents its hypotheses about changes, because overly long surveys are answered badly: to add a question you first have to drop one that no longer matters. The pipeline runs from defining question changes through their implementation, verification, launch, and wrap-up to reporting. Methodologically, Alexander stresses, they are meticulous: constructs and cross-checking questions keep answers from contradicting each other and stop a question being read differently by everyone — the last part of Accelerate is written about exactly this. Wording, moreover, makes it easy to nudge a result in a pleasant direction.

Artyom disagrees on one point: he likes the idea of rotating owners, but has not worked out how to fit it at home and sees no value in it. He prefers a product manager who owns the survey, keeps improving it and defends it: when someone comes asking to add their own questions, the owner explains that it is too narrow and interesting only to one unit — it can be run separately, but not dragged into the big shared survey. Alexander counters that behind the survey stands an internal team of several dozen researchers, whose handling of top-manager requests is described in Software Engineering at Google, and that rotation exists to externalize knowledge: the methodology is written down, so the survey does not live in one head, and each newcomer arrives without the effect of a worn rut. Artyom agrees that it is a question of maturity and team size: after two or three years the process runs on rails anyway. Alexander also likes the sampling: three cohorts, each answering once every three quarters, offset so that they never land in the same season.

Artyom warns against copying the scheme blindly: with a headcount of some hundred and fifty you will probably have to survey everyone, and the frequency should be synchronized with the rhythm of major changes. If a noticeable improvement lands once every six months, then ask once every six months; otherwise people answer for a second time that code review is as bad as ever, and start getting annoyed. He calls such changes wow changes: run the survey after the platform has qualitatively changed the experience, not to lift a rating but so that people actually notice the difference. Artyom picks up a point from their previous meeting with Alexander: ask about scenarios — merge a pull request, roll changes out to production, resize a database — because no engineer holds a scenario that reads as bringing up a Kubernetes cluster. Ratings of individual tools are skewed: one bad tool drags the whole process down, and in a large company there are always neighbouring teams with their own SLAs. Alexander adds that tool owners see only their own patch of ground and do not think about the end-to-end path.

That is why the paper moved from questions about specific tools to five general themes: outcomes, strategy and product direction, how teams work, tools and resources, and workflow and execution. Tech debt lived inside the last one — the term itself, Alexander recalls, is disliked by Dima Gaevsky, who owns their internal development platform. The problem showed up in the big surveys, then came focused studies, a breakdown by types of debt and by parts of the codebase, and over several years the answer moved from tech debt getting in the way to it no longer doing so. Artyom advises against making a cult of big surveys, though: the trend is a product one — one to five stars and a comment box right after the scenario, because after the fact a person only remembers that something was bad or good, not what exactly went wrong. He illustrates it with his coffee machine: after the second one broke three times his next will be a different brand, though he understands that a share of units come out defective. Alexander's closing point is a second leg built on logs: hours to a first commit on his own company's platform fell from 51 in 2021 to 9 in 2024.