A Knowledge Question
You may wonder why Facebook, LinkedIn, Yahoo!, Google, Apache, and Microsoft all have their own versions of virtually the same thing. Apart from Microsoft, none of these companies competes by selling this technology to third parties, so there's no real reason for them to spend countless hours of engineering effort to reinvent something that has been implemented by someone else. It's not as if these implementations are poor or obscure—Hadoop has been in use by Yahoo! for several years, and has been developed as open source for a very long time. Why then did Facebook and LinkedIn need to come up with their own, slightly different answers to the same problem?
I think the reason is twofold:
- These slightly different solutions, when taken in aggregate over the very large numbers of engineers and products, actually multiply to have quite a significant impact. If LinkedIn can tune Voldemort to solve a problem 10 seconds faster (or 10% more efficiently) for the particular ergonomics of LinkedIn's data, that translates to literally millions of hours saved in aggregate. You can't roll your eyes at that.
- Homegrown stacks have a rather large learning and upkeep cost. A new engineer takes a good deal of time to get productive with the tools of a particular large organization such as Facebook. This is very different from an open source stack or a smaller organization, where the technologies used are commonplace and reasonably well-understood by everyone. You might think this fact is to the detriment of these large organizations, but it offers a kind of mindshare lock-in that I think they value quite highly. If you're very comfortable working with all the tools of a particular company, it takes a fair amount of motivation to start over again somewhere else. (Granted, this isn't a huge problem for talented engineers, but I believe that large companies gain a significant advantage by this kind of lock-in.)
But is that all? Is the world so cynical that the only worthwhile solutions that are invented in-house are those that solve some extremely subtle and specific problem? Worse, are they invented purely for the sake of being different from other existing solutions? I certainly don't think so.