Jepsen (Distributed Systems Safety Research)

Today I decided to remember jepsen, clojure libraryIt allows testing of distributed systems. Why now? Because this week I was doing a review of a report for ArchDays on emulating failures. As part of the review, I thought about the Jepsen library, and we talked a little bit about Kyle Kingsbury’s approach to testing warranties and failures in distributed systems. Overall, I like Kyle’s approach and find it useful to use, so I decided to write this post:)

Kyle's great. told on his approach 2018 Year at the GOTO Conference (I recommend watching the video - it is made with humor and a lot of information.). Nana his website A brief description of the work of the library Jepsen analyses generally consist of running operations against a distributed system in a dedicated cluster, introducing faults into that cluster, and observing whether the results of those operations are consistent with some model. This introduces various sources for error: bugs, bounds on the search space, and the problem of induction. Jepsen’s design also limits its use as a performance benchmark.

On the same site, rigidityCompliance with which is checked in tests. Plus. section with the results of the analysis of different bases, which accumulated more 20 behind 10 years of research.

But if we go back to the performance 2018 Kyle first talks about the complexity of stateful services, or rather databases, giving a metaphor with a burning pile of tires, over which applications are deployed that provide APIs to clients who pretend that everything is fine. He goes on to talk about the types of problems in distributed systems and the results of research on popular products and the problems found in them.

And at the end of the graduation it comes down to practical tips about choosing a database/queue for your product to do the following: Read the documentation and look for the guarantees provided by this decision If it just says strong consistency, ACID, strict, and nothing else, then the authors of the database may not understand what it means. (Or hide real guarantees for marketing bullshit) Look at formal guarantees and specifications Think about those invariants and guarantees that are important to your system. (Balance between safety/consistency and performance)

• Think about failure models (failure modes) specifically
• process colors (kill -9)
• machine failures -- clock skew
• pauses on gc/io Separation of the network (network partition) (iptables -j DROP)
• Test the end-to-end system, not just the base or queue - this will allow you to check the performance of user scripts entirely Not be a perfectionist and stop testing the system at a good enough level.

#DistributedSystems #QualityAssurance #SystemDesign #Engineering #Software #Architecture #SoftwareArchitecture #SoftwareDevelopment #SRE