[2/3] System Design. Preparing for a complex GenAI interview (Category SystemDesign)

Continue. topicIt is worth discussing the framework from 7 This is important because it is easy to fail in many ways, such as Respond like a regular backend system design interview and barely talk about data, models, generation quality, hallucinations and safety Talk only about LLM, RAG, embeddings and fine tuning, but forget that it all has to work like a production system: with delays, cost, monitoring, access control, fallbacks and normal operation.

And a good framework can help you not fall into these extremes. Below are the steps of such a framework.

1️⃣ Clarifying requirements First we need to understand what we are building. “Chatbot,” “picture generator,” or “AI assistant” are too broad. A good candidate specifies: who is the user, what input and output, whether personalization is needed, whether memory is needed, what languages and modalities are supported, what latency budget, how many users, whether it is possible to err, what privacy and security requirements, how critical hallucinations are. This is similar to regular System Design, but with AI-specific questions: can you use user data, is RAG necessary, is fine-tuning necessary, what safety restrictions there are on the input and output.

2️⃣ Framing the problem as an ML task Next, the product problem should be translated into ML-formulation. For example, Gmail Smart Compose isn’t just about “helping write emails.” This is text generation: at the entrance of the already typed part of the letter, at the exit - a short likely continuation. A RAG system is not just a document chatbot. This is retrieval-augmented question answering: user query → search for relevant chunks → context collection → response generation → verification and references to sources. In this step, it is important to show that you distinguish between generation, transformation, retrieval, ranking, summarization, captioning, translation and multimodal tasks.

3️⃣ Data preparation In GenAI, data is part of the quality of the system. We need to discuss where data comes from, how to clean it, how to delete personal information, how to filter NSFW and toxic content, how to fight bias, how to make chunks of documents, how to build embeddings, how to version data and indexes, how to respect access rights. This is particularly critical for RAG. If retrieval gets the wrong context, even a good LLM will generate a confident but useless response.

4️⃣ Model development Now we can discuss the model. But not in the format of "take the largest model." For Smart Compose, a small and fast decoder-only Transformer model may be more important, because a hint should appear almost instantly. For Google Translate, encoder-decoder Transformer is more logical because it is the task of converting from one language to another. In general, a good answer involves explaining compromises.

5️⃣ Evaluation This is one of the most important steps in these tasks. In ordinary ML-tasks, you can often talk about accuracy, accuracy, recall. But in GenAI, things are more complicated: a good answer may not have a single ground truth. Therefore, it is necessary to separate: offline assessment, online assessment, evaluation of people, product metrics, system metrics, safety metrics.

6️⃣ Overall ML system design That's the central point: put the whole system together. At this point, it becomes clear that GenAI System Design is not only ML, but also normal engineering: services, queues, storage, caching, access rights, observability, rollback, A/B tests and planning capacity.

7️⃣ Deployment and monitoring After starting GenAI, the product must be constantly monitored: latency, token use, cost per request, GPU utilization, timeout rate, safety filter trigger rate, hallucination signals, user feedback, retrieval quality, drift in data, degradation after changing the model or version of the prompt, prompt injection attempts and abuse. It is this layer that distinguishes the production AI system from a beautiful demo.

And last post We'll look at the problems in the book.

#SystemDesign #AI #GenAI #Architecture #Engineering #ML #Interview #Software