Brief overview of the T-Bank data platform (Data column)
I read something interesting. article Talk about our data platform. To summarize a long enough article, T-Bank’s data platform has evolved. 18 years, following industry-wide trends. The company gradually moved away from the classic concepts of data warehouses according to Inmon and Kimbell in the direction of Data Lake, and then to modern Lakehouse architectures. The platform is now serving more 17 thousands of users and processes more than 144 Millions of requests per month, which requires constant development of scalability and performance. Current architecture includes 19 Key systems that provide the full lifecycle of data – from collection to visualization and security. Here's how they're grouped.
1. Data collection and transportation
- Data Replication: BODS (legacy) Chrono for packet and streaming replication
- Event Sourcing: SDP (Streaming Data Transfer Platform) Based on the principles of Data Mesh Reverse ETL: Spheradian to return data to operating systems with latency 100 miss 2. Data storage Data Warehouse: GreenPlum as the main database (15 clusters, 1,7 PB) LakeHouse: Spark/Trino + S3 with Multiple Computing Engines Real-Time Analytics: ClickHouse for fast analytics on large tables 3. Processing and transformation
- Streaming Processing: Unicorn (On Apache Flink) nifi
- Workflow Management: TEDI (Apache Airflow) and Moebius for orchestration
- Analytics Tools: Proteus (Apache Superset) for dashboards and Helicopter for collaboration 4. Data management Data Discovery: Data Detective for search and catalog Data Governance: Data Contracts for Data Supply Management Data Observability: DQ Tools for Quality Control and Data Incident Management Data Security: SLH for sensitive data access management
If you want to know more, you can read the article and ask questions in the comments.
#Data #Database #Architecture #Software #Engineering #PlatformEngineering