CodeFest Russia: Where does iron for neurons go? (AI column)

Interesting. speech Valentina Mammadov from Sber, where he conducted an analysis of the iron market for AI. He began by talking about the market realities that

• ChatGPT is now 5The most popular website in the world
• NVIDIA controls. 92Percentage of the GPU market for data centers
• H100 costs $30,000And the stack 8 These cards are $250,000

If you compare the consumer and server segments,

• RTX 4090 It costs $2 and H100 is about 30k. 15 with similar peak productivity
• RTX memory bandwidth 4090 1 TB/s and the H100 already has 3 TB H100 optimized for BF16 precision computation, a special type for neural networks

But the main problem is not even the iron, but the ecosystem for development. NVIDIA has been working on this for the last time. 20 years and they have great frameworks, fixed bugs and this is all Flash Attention on AMD came a year after NVIDIA Switching to alternatives = risks and loss of development time

In terms of budgetary alternatives, Apple M4 Pro laptop comparable to RTX 4090 for many ML problems

• Apple M cluster for $20,000 It can run DeepSeek v3 at speed. 20 token But consumer GPUs can’t be clustered effectively for learning (This option is only available in NVIDIA’s industrial maps.)

Main conclusions of the report are as follows: NVIDIA's monopoly rests on software, not hardware - CUDA's ecosystem is critical

• But competitors are growing - they are haunted by NVidia's margin on industrial cards, which is estimated to reach 57% ||(Truly during the gold rush, the shovel seller wins.)|| Among the competitors is Cerebra. ($25M per chip)AWS Trainium and Google TPU v7 ||(According to the author, TPU chips for Google are two times cheaper than using NVidia cards.)|| For personal inferencing or even for small businesses, RTX is enough. 4090 For inferencing more suitable cluster from Apple M

#AI #Engineering #Software #ML #Hardware #Future #DevEx