Skip to content
#Management

[1/2] Dark data (Dark Data. Why What We Don’t Know Is Even More Important Than What We Do) (Category Management)

#Management #Data #Math #Statistics

I recently read this book by David Hand, Fellow of the British Academy and President of the Royal Statistical Society. I was interested in the subtitle of the book in Russian, “A Practical Guide to Making the Right Decisions in a World of Missing Data.” As a result, I didn't see a practical guide, but I did see a lot of real-life file stories when working with data. Interestingly, I’ve read more than half of the examples in other books, plus I’ve encountered most of the problems in practice. So the book, to me, doesn't contain any wow information, but it's definitely useful for thinking people who plan to make decisions based on data. The bottom line is that decisions are usually made based on what we have. But often to make a decision Equally important are the data we don’t have.. The author calls such data dark and cites their classification as usually incomplete, but seems useful.

In the last part of the book, the author gives tips on how to extract benefits from dark data through the use of data. Randomized controlled trials (hello a/b tests)

  • Simulations. (I recently read a book called “Probability Programming in Practice.”)
  • Data replication (Here's the author talking about bootstrapping.) The Bayesian approach with a priori distribution (Here you can think about the multi-armed bandits.) Confidentiality of data and collection without disclosing sensitive information

And at the very end, the author recalls the joke about the search under the lantern, where it is lighter.

A drunk man looking for something under the light. A police officer approaches him:

  • What are you doing here?
  • Looking for apartment keys
  • Where did you lose it?
  • Over there in the park.
  • Why are you looking here?
  • It's lighter here. And wishes readers to find their own ways to expand the little spot of light and illuminate the surroundings of their data:)

P.S. In the next post, I’ll tell you more about dark data.

#Management #Data #Math #Statistics