Why AI doesn’t work without an engineer (AI column)

I saw it. show-off Kirill Mokevin’s “Organized Programming” podcast, where he invited an interesting guest, Andrei Tatarinov, to talk about AI not only in terms of concepts, but also practical applications. Andrew himself is an experienced engineer. (ex-Yandex, ex-Google) with a background in machine learning. After working for hire, he founded his own team, The Age. 8" (Epoch 8), which specializes in machine learning problem solving and helps startups implement AI into real products (By the way, in the blog on Andrey’s website it is interesting to read the cases of implementing AI solutions.). Below are the main points of the discussion that lasted about 2.5 hours:)

1. AI as ‘magic API challenges’ Andrey notes that the complex themes of machine learning have become classic API calls to services. For most developers, modern AI is not independent training of models from scratch, but the skillful use of ready-made solutions through APIs. 2. Limits of exponential growth Andery believes that we are close to a possible saturation in the development of AI. He compares the current situation with space technologies: after rapid growth, there is a plateau that requires qualitatively new approaches. The rapid change of leaders on benchmarks indicates an approach to the theoretical limit of quality. 3. LLM Technical Architecture Inference and Running: Neural networks are a sequence of matrix multiplications and nonlinearities. Choosing a runtime is critical for efficiency, especially for larger models. Language models generate tokens one by one, triggering inferens repeatedly to get a full response. This explains the streaming nature of the output. 4. Practical challenges of implementation Fine tuning and prompt engineering are technologies for the masses, as learning from scratch has a “barrier price tag” for conventional companies. And it is recommended to start with writing the best prom and testing it. Separately, Andrey noted compatibility problems - customization of models often leads to incompatibility with various rantimes. Conversion between runtimes is similar to cross-compilation of programs. In general, there are a lot of runtimes and for LLM there are in addition to the general view of PyTorch there are llama.cpp, Ollama, Llamafile, etc. 5. RAG and Vector Knowledge Bases RAG (Retrieval Augmented Generation) It solves the problem of volume limitations. The system uses a vector knowledge base to search for relevant information and saturate the industry with target data. Very popular technique at the moment 6. Limitations of Modern Models Even with clear rules, models can produce inaccurate information due to their probabilistic nature. This is especially critical for specialized tasks such as recommendation systems in e-commerce. The amount of information that a model can absorb on a pretrain is limited by the number of hyper-parameters. Retraining a model sometimes leads to forgetting knowledge in other areas that were learned during training. 7. The Future of AI Agents MCP Standard (Model Control Protocol) It provides opportunities for creating business assistants. However, the limiting factor remains the ability to multistep reasoning. Andrei is skeptical about the possibility of creating a universal assistant that would intelligently solve multi-stage tasks in the near future.

Based on the results of communication, a number of practical tips can be identified for solving problems using LLM Start with promt engineering instead of trying to train a model from scratch Use RAG to handle large amounts of specialized data Test quality iteratively, as in classic ML Consider the limitations of rentimes when planning architecture. Realistically evaluate the capabilities of current AI solutions

#AI #Software #Architecture #Metrics #Engineering #ML #Agents