How Amazon and Google view CI/CD in an entirely different way (Category Architecture)
Very interesting. article The different approaches of the two major companies to building their CI/CD processes. The author worked together 15 years in engineering teams that dealt with CI/CD infrastructure: first 11 years at Amazon, then 4 I went to Google for a year and then I went back to Amazon. And in this article, he explains how the two companies’ schools of thought differ in building their CI/CD processes, focusing on the next trio. - Pre-submit This phase refers to the developer experience before the code submit. In fact, the author analyzes what checks can be screwed into your workspace or as part of the code review process.
- Post-submit This phase refers to the developer experience after the code submit. How change merges, how code is deployed in prod-like test environments and run checks there, and how code is rolled out.
- Testing The author in the article talks about integration and end-to-end testing, not modular testing. Modular testing is trivial to run anywhere, but integration testing requires the candidate code to be deployed on the system being tested and linked to dependencies, which adds an exponential level of complexity to the infrastructure.
Google and Amazon host their code differently. Google uses monorepo and all code is stored in one place. Amazon uses the concept of microrepos, where the code of individual services is stored in separate repositories.
And this difference causes a different approach to tools and focus on different stages of the developer experience.
In short, Google It’s cool in pre-submit checks, as hundreds of thousands of engineers live in a shared repository where there are no separate brunches. Blast radius in submite problem code is very large, so they invest a lot in different cool things on pre-submit. (Here is another article by the author where he talks about it. run end-to-end integration tests from a local dev environment, or a code review, against ephemeral, hermetic test environments). But the post-submit experience is quite different - code does not roll to the product very often. The point here is that the changes of different engineers accumulate in batches, which run more tests. This is because a single line of code can affect a large number of deployments, so running all the checks on each commit is not exactly possible. But the payoff is a long wait for the code to be delivered to production.
In Amazon, the code lives micro-repeat, where the blast-radius of change is limited to your repo. (at least on the pre-submit part). That is, if you make something strange, you will easily harm only your colleagues. 2-pizza-team. And then include post-submit checks that will not allow this change to roll out. Therefore, Amazon has weak pre-submit automation, but post-submit is much better than Google. The difference is that changes in the code are on sale within hours, not days.
The author says that Google and Amazon have chosen different approaches to combat complexity. Monorepo guys from Google works at the expense of a large team, which automates pre-submit and makes smart selectivity of tests, allows them to run tightly and so on, but the code is delivered to the market for a long time. But if major codebase changes are required, they are easier to make in one monorepe. Microrepos guys from Amazon is easier to change and roll out a new feature code on the product, without fear of big problems. But the problem with updating dependencies in different repositories is pain. As a result, the author of the article notes that it is possible to support the Amazon approach with a much smaller team. Both have a right to life, and in fact set a school of thought for their adherents:) The author prefers the approach of Amazon.
Well, it’s really cool to combine a cool experience on pre-submit and post-submit, but it’s often too expensive, so with limited infrastructure resources, teams focus on what’s most important in their environment.
#CI #SRE #Architecture #Software #Infra #QA