Public System Design Interview at ArchDays 2023

Alexander sets the format straight away: he states the task, Nikita asks clarifying questions and records the answers, then designs the solution, and the remaining half hour goes to the review — what worked, what did not and what is worth improving. The task is a system for running A/B tests on web and mobile: some in the audience may be familiar with such systems and have used Google Firebase with its flags and remote config, and now such a foreign product has to be replaced by an in-house one. The functional requirements: an analyst creates an experiment with parameters and a list of metrics and checks its statistics regularly; client applications receive through an SDK not the general list of experiments but only those a particular user has landed in, and ship analytics events back. The tests are not a coin toss but smart targeting by audience segments: men only, clients holding certain products, people who visited certain pages. Alexander admits the task is not an accident: the group's A/B platform is developed inside his own division and he watches it closely.

The non-functional requirements are short but firm: a 200-millisecond answer so tests never block a page or a screen from loading; a user must stick to an experiment so the button does not flash green, red and yellow at every visit; and the results must support a statistically significant conclusion. Nikita presses for numbers and Alexander gives them: thousands of audience segments, with one identifier falling, by his reckoning, into about a hundred of them; some 30 million monthly active users, around 10 million daily active in the mobile apps and roughly 3 million on the web; a target ceiling of a thousand simultaneously active experiments; and, say, a hundred analysts in a large organisation. An experiment normally lives one or two weeks, sometimes a month, sometimes it ends in a couple of days. A statistics lag of 10–15 minutes, or even an hour, is acceptable for the first version. Alexander parks automatic conclusions as a bonus question, while a queue for conflicting experiments and a role model are simply put out of scope.

Nikita starts not with boxes but with scenarios. The analyst creates an experiment, watches a table of statistics, ends it, and if the hypothesis won, files a task to roll the winning variant out to everyone with a link to the experiment attached, so it does not look as if he personally decided to repaint all the buttons. The user is passive: he opens the app and all the magic happens without his knowledge. The application fetches its list of experiments by identifier at start-up and lives with it; the app itself is deliberately dumb and merely applies the configuration — reordering elements on a screen or steps in a form, or changing the product description text. No new analytics pipeline is built: the existing telemetry simply gains a mark saying which experiment the event happened in. Web and mobile are not split apart, but since a mobile app can go offline and accumulate events, two separate upload endpoints remain.

Then the boxes appear: an API gateway with a public and a private part — the analyst's endpoints have no business facing the outside world, and Alexander accepts that as said out loud; an experiment service with its database; separate statistics collectors for web and mobile, because they have to scale differently; a queue; an analytical database; and a service that decides whether a user belongs to an experiment. Its name emerges in dialogue: Nikita admits he cannot think of one and is handed a ready name — the decision service, much better that way. One box is still missing: a segment store the decision service queries by user ID for a collection of segments, while the experiment list itself is small and rarely changes, so it can be cached in memory. Nikita picks technologies by load: PostgreSQL for experiments, Kafka partitioned by user identifier so ordering need not be rebuilt from timestamps, ClickHouse for analytics — he honestly says it is the only analytical database he really knows — and Cassandra for segments, a key-value store with high cardinality by user.

The last question Alexander has time for is stickiness. Nikita suggests keeping the user-experiment pair in a fast key-value store: 30 million users at a couple of bytes each add up to mere megabytes. Alexander shows another move: the assignment can be written into the segments themselves — create a segment meaning "landed in the test variant of this experiment" — and no separate entity is needed. That takes the conversation back to Nikita's own remark about things being computable: given the experiments and the user's segments, it can be precomputed. And if the list of experiments were fetched not once at start-up but on every page load, as happens on the web, online segments would be needed: a processor sitting next to the event stream would count real-time behaviour and write the result into the same store, so it could be taken into account when deciding and used to launch chains of experiments.

In the review Alexander praises the approach itself: do not dive into the thick of it, work out what the system does and which APIs it exposes for which scenarios, split it into conceptual boxes and only then talk about technologies — assembled from the company's standard platform bricks. They did not manage to draw the endpoint through which the analyst reads his table, to estimate the load, or to reach automatic conclusion of an experiment with a notification to its author — which, he says, happens often in real interviews too. The floor fills in the details: segments are held in the commercial key-value store Aerospike, which is awkward in current conditions, and the team recently wrote an RFC comparing Cassandra and FoundationDB and weighing a sharded stateful service of their own. One challenge from the audience: the candidate named technologies but not their competitors; Alexander agrees that he normally asks why exactly this one, and recalls that in an earlier generation of the platform the speed layer of a lambda architecture ran on Cassandra, worked poorly and was rewritten on ClickHouse.