DevOps Conf · June 14, 2022
Assessing SRE skills
Interviews on system design and troubleshooting
/ Assessing SRE Skills · DevOps Conf 2022
Slide contents
1. Assessing SRE skills
Interviews on system design and troubleshooting
2. Alexander Polomodov
I lead digital ecosystem development at Tinkoff
I curate our System Design Interviews
And I interview for troubleshooting myself
3. From the hiring funnel to how to learn this
What we'll cover
The hiring process and why we check troubleshooting
What the candidate gets out of the section
How the section and a typical task look
Scoring, the contrast with design, and growth
4. 01. What SRE hiring looks like
Hiring stages and candidate levels
5. Troubleshooting is one of three technical sections
Hiring stages against the candidate's level
Pre-interview — a call with a recruiter
Technical sections: System Engineering
…Programming and Troubleshooting
Then the Fit Interview and the offer
6. 02. Why we check this at all
From the company's side and the candidate's
7. Reliability is decentralised — the teams own it
Why we check troubleshooting skills
25M+ customers, and both they and the load grow
Many business lines, platforms and service lines
The products merge into an ecosystem
Engineers own the reliability of their own systems
8. An incident almost like the real one, at a low price
What the candidate gets out of it
Working an incident under near-real conditions
Normally only a production system gives that
Went brilliantly — likely a Senior/Lead offer
Went poorly — we say what to work on
9. 03. What the troubleshooting section looks like
Eight steps and a typical task
10. The candidate is the Lead, the interviewer is on call
What the troubleshooting section looks like
Ground rules: we are both SRE engineers
The system is described, the task is framed
The incident starts, diagnosis runs on hypotheses
Workaround, real fix, then prevention
11. Two data centres, one database, a spike of 504s
What a typical task looks like
Each data centre has frontend, backend and cache
One database serves both data centres
The symptom: the share of 504s grows
Those are gateway timeouts — check the logs
12. 04. How we score the candidate
Six axes and the level read off them
13. Six axes, and the level is read off them
The scoring criteria
Breadth, and how methodical the search is
Whether a workaround mitigated it quickly
Whether a full fix came with a stated algorithm
Whether root cause was found, and what comes next
14. 05. Troubleshooting and system design
How they differ and why one section remained
15. Put out a fire, or build what will not burn
How troubleshooting and system design differ
Troubleshooting starts from a production system
System design starts from requirements only
One is about firefighting, the other about design
And different people pass them
16. System design dropped: candidates averaged Junior
Ideally SRE = troubleshooting + system design
The ideal: fight fires and design systematically
The reality: design was passed at Junior level
So we stopped asking System Design
But we decided to teach it internally
17. 06. Wrap-up and how to get better
Theory and practice for both sections
18. You only get better with theory and practice together
How to get better at system design and troubleshooting
Troubleshooting: SRE Book, tools, postmortems
Plus practice in an SRE team on real systems
Design: principles, system classes, others' architectures
Plus practice as an architect and design katas
19. The questions we covered
From the hiring process to how the two sections differ
What the hiring process looks like
Why we check troubleshooting skills
What the candidate gets out of it
What the section and a typical task look like
How we score, and how the sections differ
20. Links for further study
The sources from the talk's closing slide
An article based on this talk
Google SRE Book, SRE Workbook, a Mobile SRE Report review
«Release It!» and a repository of postmortems
My articles on the System Design Interview and preparing for it
21. Thank you!
polomodov.tech
All slides and links are in the Telegram channel
Alexander Polomodov, Technical Director & Fellow, T-Technologies
@book_cube