3 ms·
Most of the leverage is in the efficiency at exploiting data, and not just in technical terms but also operational economics. This is far more important than th
by jandrewrogers 8y ago
Most of the leverage is in the efficiency at exploiting data, and not just in technical terms but also operational economics. This is far more important than the data itself for a fundamental reason that has not been broadly internalized by the industry at large, and which makes it difficult to build a data moat.
Almost any proprietary data model based on proprietary data can be reconstructed with sufficient fidelity from unrelated external data sources to be competitive with that proprietary data model, at the cost of being somewhat more expensive to engineer all things being equal. I've demonstrated this many times in practice. As a corollary, a company with sufficiently efficient end-to-end data infrastructure could, in principle, commoditize all proprietary data model companies. This company does not exist, yet, and it would require some very specialized talent but every necessary ingredient exists.
This is a realizable endgame due to the reality that virtually all data companies, for good and practical reasons, are strongly incentivized to build on infrastructures that are literally orders of magnitude less efficient than is possible in principle. A company that was purpose-built on exceptional end-to-end data infrastructure engineering could capture much of the revenue in these markets in a surprisingly short time by commoditizing the data model and arbitraging efficiency.
- huac 8y ago> A company that was purpose-built on exceptional end-to-end data infrastructure engineering could capture much of the revenue in these markets in a surprisingly short time by commoditizing the data model and arbitraging efficiency. do you have any examples?
- untilHellbanned 8y agoI was with you at the beginning. Can you ELI5?
- dswalter 8y agoLet's say you have a company that tries to make online checkout more efficient by predicting which payment the user will prefer, and research shows that if the user is first presented with a payment method they use, they are more likely to buy. If I subscribe to your API and get the outputs of your predictions, since I can see the kinds of inputs that you tell me are associated with preferring PayPal, I can approximate your PayPal-preference model. This works for basically any machine learning model-as-service. So if I invest heavily in approximating and then serving models more at lower cost than anyone else, that might be a viable businese.
- jandrewrogers 8y agoI can give it a try. :) There is an assumption that unique data enables unique insights, which can presumably can be monetized in some fashion. As long as no one else has access to your unique data, you have pricing power for the unique insights. There are loads of companies, big and small, trying to execute this business model. A problem with this model is that for practical purposes there are no such things as "unique insights". The only thing unique data grants you is a cheap path to a specific set of insights. For every set of "unique" insights, there is almost always many sets of unrelated data sources that can be analytically combined to deliver the same insights. In the slightly seedy underworld of data model brokers, I've seen some very impressive examples of this. The way these alternative data models make money, despite the creation process being more expensive, is that they are positioned as a cheaper alternative to companies that think "unique data" means they can extract monopoly rent. The explosion in data source availability has slowly made these clever alternatives more prevalent. Currently, the alternative to having access to the unique data is to do significantly more expensive computations on what are typically larger and more diverse data sets. This keeps it from becoming a runaway race to the bottom due to the higher cost. Note that in both cases, the parties are using conventional data infrastructure stacks with their implied limitations. In recent years I've done extensive studies of the cost structures of these types of businesses. It turns out that if you can reduce the end-to-end data infrastructure costs by an order of magnitude for the reconstructive approach then your total costs will be far below the break even point for the conventional "unique data" approach. Furthermore, the computational work required to replicate one high-value data model is substantially reusable for other data models, so the more data models you reconstruct, the lower the marginal cost of reconstructing additional data models this way. Done at scale across enough data models, the amortized cost of reconstructing unique data can be less than using the unique data! People have been idly thinking about what it would take to do this for a few years. Collecting the rare skillsets required to pull it off is a major hurdle for any company. The notion that it is possible to reduce end-to-end data infrastructure costs by (at least) an order of magnitude relative to conventional data infrastructures is well-supported but it raises another question: why can't the "unique data" companies do the advanced engineering required to have such an infrastructure themselves (it isn't something you can do with open source currently)? The simplest answer is that it is difficult to justify extremely expensive engineering efforts outside of an organization's core expertise solely to prevent erosion of the market value of their unique data. Fundamentally, it is a shift to competing on infrastructure instead of data, which is an improbable transition for companies.
- brianchu 8y agoThis is really interesting and thought-provoking, but I'm skeptical. 1. In my experience, each data source and each data format requires a lot of custom work. Each kind of prediction task requires additional custom work. I don't see this going away. Even if a company develops a solid core of reusable engineering infrastructure, it will always need to be adapted to the problem at hand. At this point, this company would seem more like a consultancy, with non-trivial marginal/variable costs. This reminds me of Palantir, which operates this way - core set of tools and infrastructure, consultants implement and apply these tools/infra at each client company with a lot of custom integration work. 2. Assuming this is not a problem and the shared infrastructure is able to generalize enough of the custom work to be feasible, this thesis actually seems like an argument for the big tech companies dominating all data companies. Google, Microsoft, and Amazon have the engineering talent and resouces to develop this hypothetical infrastructure. They also have the internal political will because they can then expose this as cloud APIs. Indeed, it appears they are already attempting this in certain domains. 3. Superior engineering infrastructure is indeed a competitive advantage, but isn't enough of a moat for a single company to dominate this space. Yes, great engineers are hard to find, but there are enough of them for more than one company to feasibly develop this infrastructure, with a lot of money. You can't say the same about trying to buy the social network of Facebook or Instagram.