AWS re:Invent 2023 - [LAUNCH] Achieving scale with Amazon Aurora Limitless Database (DAT344)

AWS re:Invent 2023 - [LAUNCH] Achieving scale with Amazon Aurora Limitless Database (DAT344)

Continuing yesterday themeI recommend you to watch interesting. speech from the guys at AWS about a meaningful upgrade to their Aurora database, which we're somehow debate in the bonus edition of the "Code of Architecture" whitepaper from 2017 years. Now the guys have added scaling capabilities to this database. And here the focus was on scaling write load, since with the read load, this base was already well. If you briefly describe the changes, the meaning is something like this.

1. Adding semantics sharding Inside the Aurora itself - for this users have opportunities

• Mark the tables as sharded and specify the sharding key - if the transactions fall on one shard, then this is as fast as possible, if not, then distributed transactions are used two-phase commit. Hash-range partitioning is used for sharding. Mark part of the tables as reference - such tablets are used as reference books and they are laid out on each shard (They should be used if you write loads on these tablets small.)

2. The sharded base is supported by semantics like read commited and repetable read
3. This semantics works inside the Aurora cluster, so common things like dump or point-in-time recovery work as expected from a cluster without shards.
4. Under the hood, it all works by adding the shard group concept inside the aurora cluster, where there are routers for distributed transactions. (Distributed transactions are coordinated and the overall result is collected.) and data tricks (Local query parts are executed here, local scheduler works, indexes are used, and so on.). Here, too, an additional compute redundancy parameter appears, which allows the compute parts of the shards to better survive failures.
5. To implement repetable read guys use the concept of bounded clocks, which is similar to the concept. TrueTime from Spanner. The point of this concept is to use time stamps from different devices to organize transactions. The problem is that the clock on different machines time can slightly deviate, so the application needs several parameters: current time. (approximate), earliest possible time, latest possible time. And then the application must wait a little in order not to accurately put the transaction a time stamp from the past. (Otherwise, it can disrupt the order of transactions in the cluster.). These parameters are obtained from the EC2 TimeSync service with an accuracy of the order of microseconds. The presentation itself explains how this works for local and distributed transactions, as well as for performing aggregate requests.
6. At the end of the report, the authors talk about how routers and shards interact, as well as how to measure the effectiveness of queries, which requests receive the maximum benefits from parallelizing the execution of requests on different shards. (creation of indexes, vacuum, aggregates)
7. For now, this limitless version of Aurora is available through a preview application

Anyway, this talk is quite interesting. I would like to note that in order to benefit from these innovations, it is necessary to be able to design the data model correctly and understand how to more effectively decompose data in order to get into specific shards as often as possible and to get less into distributed transactions.

#Software #Architecture #DistributedSystems #SystemDesign #Engineering #Databases