What drives Success in StartUps? Don’t ask Sam Altman!

By Steven Seggie.

I don’t systematically follow any blogs and instead rely on the people I follow on Twitter to point me in the direction of blog posts that I might be interested in. Last week Burak Büyükdemir linked to a post on Sam Altman’s blog (http://blog.samaltman.com/super-successful-companies) about what makes startup companies successful (when last I looked it had been viewed over 90,000 times in 5 days – that’s a lot of people viewing it). This led to a conversation between Burak and I on Twitter and then to me writing this.

The article was a fairly typical listing and detailing of the behaviors and features of successful startup companies based on the author’s experiences / observations of successful companies. Some of the behaviors / features included an obsession with talent, focus on growth, having nice founders etc. I am sure that these are all important to some degree or another. My quibble is not with the list per se, instead my quibble is with the methodology Sam used to compile his list. In fact, Sam Altman knows the problem with his methodology: in the first paragraph of his post while talking about how he made his list he notes that “.... plenty of non-successful startups do some of these things too.”

That they do, and that’s why it is impossible to tell whether or not the list is of any use. Sam has fallen into the trap of choosing on the dependent variable. He’s made a list of the drivers of successful startups by examining successful startups. You can’t just examine the successful ones, all that does is bias your conclusions. To be fair, he’s not the first person to do this (my favorite is Tom Peters in ‘In search of Excellence’) and I’m pretty sure he won’t be the last.

What Sam has done is tried to explain what makes start-ups successful by examining successful start-ups.  Seems quite logical I guess but no matter how logical it may seem it is still wrong. Let’s say, you examined 20 successful start-ups and came up with a list like Sam’s of things they had in common. If you follow Sam’s logic then you would argue that nice founders, an obsession with talent etc. drive success in startups. Sadly, this conclusion does not follow. As Sam mentioned there are non-successful startups doing these things too.

So how should we do it? How can we find out what drives success in startups? The best way for me to describe how to do it is through a simple example. Suppose we suspect that the niceness of the founders is an important driver of startup success. To test if this is true or not, we would take a random sample of startups, selected  irrespective of how successful or not they are (i.e. randomly). Then we would rate them on the niceness of the founders and see if there is a positive correlation between the niceness of the founders and the success of the startup. If our analysis showed a positive correlation between niceness of founders and success of startup then we could make an argument for niceness of founders driving the success of startups.

So why does this even matter? Sam’s list passed the sniff test, so what’s the problem? The problem is we don’t know if there is a problem. The problem with analysis like Sam’s is that the methodological problems mean we can have little or no faith in his conclusions. And if 90,000 + people are reading his blog post and they are not aware of the methodological issues and the impact on his conclusions then that’s a problem.