Confidential a/b testing (Trustworthy Online Controlled Experiments)

After the start of the holiday, I finished reading. book nIt is a necessary tool for big tech companies in order to evaluate the effectiveness of certain ideas for optimizing websites, applications, ml-models.

This is a really cool book written by three authors:

• Ron Kohavi - Technical Fellow and corporate VP of Microsoft's Analysis and Experimentation (previously director of data mining and personalization at Amazon)
• Diane Tang - Google Fellow, with expertise in large-scale data analysis and infrastructure, online controlled experiments, and ads systems
• Ya Xu - head of Data Science and Experimentation at LinkedIn

This book is in Russian came out In the publishing house DMK Press and it can even be read, checking periodically with the source. The book consists of five parts:

1. Introduction for all explanation of the motivation for conducting experiments, what a full cycle of experiments looks like, how to assess the reliability of the obtained data and how to pump the culture of experimentation and come to the platform
2. Selected topics for all An example of well-known experiments assessing the impact of website speed on business performance (full design of the experiment and analysis of its results)What organizational indicators are available, how to choose the OEC (overall evaluation criteria) To evaluate the effects of experiments, how experiments form institutional memory, and how they can be used for meta-analysis, and how to conduct ethical experiments.
3. Additional and Alternative Methods of Controlled Experiments What to do if an honest a/b test is not performed (Expert assessment, UX research, focus groups, reviews, ...)How to design observational studies to investigate cause-and-effect relationships
4. Platforms for experiments This is a very important section for those who decide to build their own platform. This is about experiments on the client side. (for example, in a mobile application)Toolkit for experiments, how to choose a randomization unit (page/screen, user session, user, company)How to find a compromise between speed/quality/risk in the further development of the experimental platform, how to analyze large-scale experiments
5. Detailed description of the analysis of experiments Here comes the time for statistics and the authors talk about the t-test, p-value and confidence intervals, errors of the first and second kinds. I recommend reading the book.Understanding Statistics and Experimental Design. How to Not Lie With Statistics"that I mean." wrote earlier. Here we are talking about the assessment of variance and increasing the sensitivity of experiments, how and why to conduct a / a testing, what are the restrictive indicators when conducting experiments, like SRM. (sample ratio mismatch)How leakage and interference between options can occur (For example, in experiments on social networks or on e-com platforms.)How to measure long-term effects.

In general, the book is top-of-the-line and I recommend it to those who are deeply immersed in the topic of a/b experiments ... or those who just like statistics:)

#Math #Statistics #PopularScience #Science #ML #Data #Software #PlatformEngineering