The Power of Experiments - Review
Book review - The Power of Experiments: Decision Making in a Data-Driven World by Michael Luca and Max H. Bazerman. Reprinted with Permission from The MIT PRESS. Copyright 2020.
If you are using experiments already: An interesting, but not essential read.
If you are not using experiments: This helps make the case.
I think it’s really useful to synthesise our thinking after a learning exercise. It helps it sink in and makes us re-evaluate the salient themes. That’s why I’m writing book reviews.
When you’re in ensconced in the product world it seems like everybody is talking about experimentation and the power of data insights. Well, this book isn’t really aimed at that group. It’s main purpose is to illustrate just how powerful the experimentation practise is and to encourage the reader to adopt the approach. For that use case, they nailed the naming for this book.
It achieves its goal by highlighting how much has already been achieved across a range of contexts.
The anchor case study is from the UK Government tax collection service. The idea is that no matter how dusty, rigid or old fashioned your organisation is, you can use experiments. From that perspective it works. The team achieved significant results in a context that will surprise many.
Those who know about GDS will be less surprised to find an outcomes oriented technique in use in government.
The authors move on to the most detailed account of the #AirbnbWhileBlack controversy. It’s the most interesting part of the book and for those, like me, who only understood the topic superficially it’s revealing and disturbing. This article dives deeper, but in summary the book explains how researchers uncovered racism in Airbnbs users and what the company did to stop it. The conclusion is, they didn’t really do enough.
The book goes on to talk about some of the more well known experimenters in technology, touching on Facebook and having an interesting debate about some of the ethical considerations behind some of their research. Generally I find myself agreeing with their stance on the topic.
Given my knowledge of the topic, I didn’t come away with much in terms of actionable insights, but I added some more stories to my ‘experimentation is good’ sales pitch.
TL;DR - Great for convincing the skeptical to accept the critical role of experimentation in driving outcomes. Fascinating insight into the #airbnbwhileblack scandal is worth the entrance fee, but prepared to be dismayed by the results. This book is about winning minds, it’s not about actionable insights.
