Tanenbaum, PagedAttention and the Foundation That Doesn't Get Old (Books column)

I was dealing with an interesting paper.Efficient Memory Management for Large Language Model Serving with PagedAttention" from the creators of vLLM. In 2023 In the year, the authors looked at how modern LLM inferencing engines work with memory, and found an almost criminal picture for the present time. (The efficiency killer was KV-cache.).

The systems pre-reserved a continuous memory portion based on the maximum possible sequence length. But real answers are usually shorter, queries grow dynamically, and memory fragments. According to the authors’ measurements, only useful data were used. 20,4–38,2% KV-cache memory.

The authors proposed PagedAttention: divide KV-cache into logical blocks and display them into physical memory blocks that do not need to lie in a row and are allocated as needed. In fact, they transferred to LLM serving long familiar operating systems ideas of virtual memory and page organization.

And then I remembered reading Modern Operating Systems by Andrew Tanenbaum and Herbert Bose with great interest. It was like 2017 year. I had been a technical manager at Tinkoff for a year, but I didn’t want to lose my technical skills. Therefore, on weekends he came to the office: first to work out in fitness, then to work out technical topics. One of these topics is operating systems.

The main value of the book is not that after 1100 You can write your own core pages. It provides an engineering insight into how a computer divides limited resources between multiple competing programs while maintaining the illusion that each has its own processor, memory, files, and devices.

The book consistently parses: Processes and flows, planning, interprocess interaction and synchronization; Address spaces, virtual memory, page organization and page replacement algorithms;

• File systems, data storage and I/O; Interlocks and ways to prevent, detect or experience them; Virtualization and cloud, multiprocessor systems and security;
• UNIX, Linux, Android and Windows device 8.1 Real examples of these principles.

After such a book, you begin to see the same architectural moves in completely different systems: an additional level of indirectness, separation of logical resource from physical, isolation, sharing, planning, and tradeoffs between productivity, delay, and fairness. PagedAttention is a great example. The task is new, the iron is different, the stakes have risen, and a useful abstraction has come from the operating systems textbook.

Of course, the fourth edition is a time capsule. Windows 8.1 Not a modern system, but specific details of Linux, Android, security and hardware need to be supplemented with up-to-date documentation. And this is definitely not a quick Linux administration workshop. But the fundamental chapters on processes, memory, file systems, resource competition, and virtualization age much more slowly than specific APIs.

Therefore, I recommend Tanenbaum to engineers, architects and technical managers. Especially for those who, like me back then, have already moved away from daily coding, but do not want to lose the ability to understand why the system slows down, rests on memory, is blocked or poorly divides resources.

#Books #Engineering #Software #Architecture #OS #AI #DistributedSystems