11 ms·

Why AI Infrastructure Startups Are Insanely Hard to Build

192 points by nextworddev 2y ago

deleted 2y ago
[deleted]
pierre 2y ago
Good article, but what is the alternative? What can you build today as a software engineer that can have impact? Nothing seems to come close to AI / AI infra, even of its hard / risky / a moving landscape.
- hnlmorg 2y ago
  Everything we build has some kind of impact. At risk of getting philosophical, I’d ask yourself what your goals actually are if you feel only AI can have the impact you desire.
  - fhd2 2y ago
    Not sure why this is down voted, that is the key question. Impact means different things to people. Could be: 1. Building a sustainable business and making decent money 2. Building a market leader and making ludicrous amounts of money 3. Advancing the state of the art in technology 4. Helping people with their little daily struggles 5. Solving pressing problems humanity is facing Or many other things I suppose. Now if you believe that AI is eventually going to make anything humans can build now redundant, that'd be a reason to believe nothing else matters in the end I suppose. But even if we get there, there's a lot of road leading to that destination. Any step provides value. Software built today can provide value even if nobody is going to need it ten years from now. And it's not like you could even predict that.
    - mirekrusin 2y ago
      The motive is to get acquired in most cases. It’s obvious and starts to make sense when you see startup that has no feasible monetisation strategy on the horizon, yet they exist and get funding. They’re betting on building infra to be hopefully used in large corp and this is their demo/PoC.
- astronautas 2y ago
  Data infra?
- lionkor 2y ago
  A lot of things, if you're okay with not chasing the next hype bubble
- mlsu 2y ago
  I would almost invert that statement. Sorry if this comes off ranty, but what exactly are people doing in the "AI space" currently that isn't "undifferentiated spam/chatbot" being sold to non-techies who heard about AI on NPR? What are real people using "AI" for that is so insanely valuable today? How much "company Y: same product with a chat window, sparks emoji" do we all need before this thing levels out and we all take a breather on the hype?
  - jdross 2y ago
    personally? - writing and refactoring code. probably 50 times a day now - improving documentation across the company - summarizing meetings automatically with follow ups - drafting most legal work before a lawyer edits (saved 70% on legal bills) - entity extraction and data cleanup for my users
    - mlsu 2y ago
      Put a number on it. How much value of this will they capture from you personally (we'll assume, very very charitably by the sound of it, that you represent an "average" user of AI products) when this market matures? Exactly how much will your employer pay for a meeting summarizer? $10/mo a seat, $20/mo a seat, $50/mo a seat? Could the product sustain a 5x, 10x, 50x price hike that is going to have to happen to recoup the investment being made today?
      - fhd2 2y ago
        Agreed. Even if right now this seems like stuff companies want to throw money at for novelty/FOMO related reasons, I think eventually reality ought to catch up. Probably an unpopular opinion, but I think the most efficient companies of the future will tackle the ironies of automation effectively: Carefully designing semi automation that keeps humans in the loop in a way that maximises their value - as opposed to just being bored rubber stamping the automation without really paying attention.
        per1Peteia 2y ago
- ianpurton 2y ago
  It's fine to be in AI. My takeaway from the article is instead of being a Gen AI startup be a Gen AI startup for a specific use case.
- oivey 2y ago
  Slightly different take than some of the siblings: you can still just build this stuff. If your goal is impact, maybe the best place to do it will be at a cloud vendor or other big corp. If your goal is actually just a big VC exit, then maybe not. If your product is something that can be ripped off in 3 months, then it probably wasn’t going to have a long term impact anyway.
- ehnto 2y ago
  All the same stuff, to be honest. If AI is set to replace human work, well we have had a cheap human labour market for decades and yet we still need software. An LLM can't replace a business itself, which is made up of niche processes, direction and purpose, which we sometimes codify into a SaaS. We'll still need to do all that even if AI replaces some of the human parts of the business.
- jononor 2y ago
  Anything SaaS that solves a painpoints for established industries. Those that have billions of turnaround for decades already, are not good at building tech themselves, and buy solutions/services to run their business. Bonus for low barriers to entry. Agriculture, logistics, real estate, energy, etc.
  - swalsh 2y ago
    I have a theory that the days of established businesses that don't know tech is dwindling. A lot of companies which has adopted tech has started building a small foundation of talent internally. I think you're seeing this trend accelerate with the large tech companies laying people off. I have heard about top grade data science talent landing at some small sized health plan. My companies fastest growing competitor is "internally sourced departments" of the services we provide.
    - nextworddev 2y ago
      You confirm my observation as well. Even motel chains have developers building internal tools these days
    - jononor 2y ago
      Yes computer savvyness is on the rise, and have been for decades, and this will continue. But there are many levels: Ability to be a competent user, and competent buyer, ability to build it themselves. Then there are big difference in buy vs build culture. And preferences for type of solutions that are default buy vs default build. And finally, smaller and medium sized organizations are less likely to have internal teams. All this should be analyzed for the specific market, product and customer segment one targets.
- rsynnott 2y ago
  Define ‘impact’. Does ‘impact’ here mean ‘tickles the fancy of a 2024-era VC’? If so, you may be right. If used in its common meaning, absolutely not; most of this stuff is ~useless.
- _el1s7 2y ago
  If you don't know what to build, you don't build.
- notamy 2y ago
  > What can you build today as a software engineer that can have impact? Quite a bit, if you don’t follow the standard tech hype. Find an industry that isn’t tech-first and you’ll notice that there’s a lot of room for improvement.
threeseed 2y ago
Way too many founders don't understand the impact of competing with cloud vendors. Almost all enterprises have pre-committed budgets for cloud which means unless your product is FOSS it's going to be hard to convince someone to bet their business on it. Especially given that in this fundraising environment there is a 95% chance they won't be around in a year or two anyway. It's going to be a brutal few years especially if we are heading into a period of diminishing returns in terms of LLM accuracy.
- devjab 2y ago
  I’m also sort of curious as to how much of a market research they’ve done if they’re trying to compete with Azure and AWS. Even before the recent LLM rush took off, AI was a thing. In the city of Copenhagen there was a project to digitalise a few million case files (which is 10-100 documents per case file), and how it was done was basically with an intermediary company who knew the training and a cooperation with Microsoft. Yes, I’m dumping down the complexity of it all, but once the training period of half a year was over, Azure made a lot (and I mean a lot) of infrastructure available for not a lot of money and the process completed in a week or so. Since it had to happen and because it was a PoC the same project was also done by real humans. This was the “actual” project and every time deadline and whatnot the AI project had came from how long it would take X humans to do it. I can’t recall how many X was, but it was enough to meet the legal deadline for when these case files had to be digitised and sorted correctly. The human project was the result, and then the AI PoC was later used as a lesson on whether it could be done this way or not. It can, it was more accurate and not more expensive. Anyway… I’m not sure who would’ve been capable of competing with Azure. (Outside the usual suspects). Maybe a company of Hetzner could? But you would need someone who can offer you a massive amount of computing on demand, and the only companies which are going to have that are big vendors. Maybe it’s different with LLMs because the requirement is a continuous thing rather than something you need for a short period of time?
  - deleted 2y ago
    [deleted]
  - jononor 2y ago
BrunoJo 2y ago
Good tips, especially the point about narrowing the scope. At https://Lemonfox.ai https://Lemonfox.ai we started with a LLM, image and speech-to-text API. Now we are only focusing on the speech-to-text API as the other areas are already very crowded and there's a lack of innovation in the speech-to-text space.
- anonylizard 2y ago
  Like how do you plan on competing against multimodals, which keep getting cheaper and clearly can do audio->text? Or existing incumbents like Deepgram? Or just the generic APIs provided by the big clouds.
- cootsnuck 2y ago
  > Now we are only focusing on the speech-to-text API as the other areas are already very crowded and there's a lack of innovation in the speech-to-text space. I'm legitimately wondering how your hosted Whisper API for $0.17/hr is supposed to compete with groq's exact same API that costs $0.03/hr. You may be about to find out how crowded all of the AI infra spaces are. I strongly recommend narrowing your scope far beyond modality. If you've been working with this tech and getting familiar with it then you already have valuable expertise. Pivot now or panic later. If you want to stay in the speech space find what markets are being underserved with speech AI related solutions. Are there pain points there that can be solved by a STT API? If so, build those solutions. You can't compete at the infra layer and I'm not sure why you would want to try if you don't already have something unique about your offering beyond hosting open source models. It's never good if your competition is potentially just a single developer in a company standing up your entire service internally in a week. If you are determined to stay in the AI infra space then you'll need to be tackling a hard problem that companies want solved. Maybe take a look at fine-tuning models. Hard problem and maybe there's a hunger for it. (It's a risky one to tackle too though since it's very possible general/foundational models will maintain a grip on "good enough".)
JohnMakin 2y ago
As someone who has semi-unwillingly worked in infrastructure and infrastructure consulting most of my career - we’ve never even really solved that problem, what on god’s green earth convinced you AI did? I am genuinely curious.
anshulbhide 2y ago
[dead]
dvt 2y ago
Infra has always been a tarpit idea. Google didn't start out as an "infra" company, and neither did Amazon or Facebook. In fact, the few "infra companies" that did start back then (companies like Godaddy) are minuscule compared to the aforementioned. VC pouring money in LLM infra is legitimately crazy to me. It's clear as day that there will be winners of this AI cycle, but, as always, they will be companies that provide actual, real, tangible value. Making shovels works for huge companies like Nvidia or Intel, but it won't work for you. It's sad to see so much capital funneled in frameworks upon frameworks upon frameworks instead of fresh new ideas that could revolutionize the way we interact with our devices. I know it's a bit of a meme, but I'd rather see more Rabbit R1 and less LangChain. Even OpenAI doesn't really have a product. Just throwing data at a bunch of video cards isn't value-generating in itself. We need a Dropbox or a Slack or an Instagram: something people love that makes their life easier or better.
- rapsey 2y ago
  > Even OpenAI doesn't really have a product. They are making a ton of money off subscriptions.
  - nextworddev 2y ago
    Yes. Author here. OpenAI is at near 4bn ARR and growing (api plus SaaS)
    - whiplash451 2y ago
      Source?
      - matwood 2y ago
        https://www.theinformation.com/articles/openais-annualized-revenue-doubles-to-3-4-billion-since-late-2023 https://www.theinformation.com/articles/openais-annualized-r...
  - soneca 2y ago
    Yep. Obviously they have a product. Millions and millions of people that are not software engineers or any very tech-savvy persona use their product. And a lot of those people pay for it. It’s just silly to say they don’t really have a product. Specially given the rest of GP comment, OpenAI seems to be the Google, Facebook of the industry and will be the infrastructure company of it (already is kind of)
mehulashah 2y ago
The title is true. But, the arguments don't hold water for me. 12 years ago, I started a big data company. It looked similar for big data companies when Cloudera raised almost $1B in 2014. Too many people building data warehouses, especially in the cloud. I exited. Who knew that Snowflake and Databricks would emerge against the incumbents. Similarly, there will be winners in the AI infrastructure space. To win, you need to focus on your customers and delight them. Narrowing focus makes a lot of sense. Don't pay attention to the doom and gloom, or you'll never do a startup.
- rapsey 2y ago
  > Who knew that Snowflake and Databricks would emerge against the incumbents. Snowflake is not profitable. I doubt Databricks is. Their market and business is crap.
  - pas 2y ago
    if they are not profitable with these prices ... what the fuck they are doing!? do they just have company coke-athons all day every day?
    - rapsey 2y ago
      It is a sales and engineering heavy business. Very difficult to generate large returns.
      - TimPC 2y ago
        They are becoming like SAP though where initially a company buys one service and it soon finds itself buying every adjacent service from them. If they manage to do that successfully they will be quite profitable.
      - pas 2y ago
        They are bundling open source stuff (Spark and Delta and so on), sure, they have their fancy IDE and whatnot on top, but they can let the community maintain things, scale back R&D, focus on things that matter to existing clients. They have 1.6B revenue, a 50% YoY growth. And still not profitable. Hm, okay, recent acquisition on ML stuff, and of course probably burning hundreds of millions on cloud-GPU-AI shit. Well, I guess as long as they have so big growth it makes sense to invest and raise ... and yeah that probably completely obscures the actual profitability of their core business. (Not to mention that they are probably spending all that money to try to expand their core business. To upgrade their value prop from cloud version of less-dumb-data-pipelines to 1-800-data-4-AI.)
xg15 2y ago
In a nutshell: If there's a goldrush, you get rich by selling shovels. ... unless there are already 200 shovel shops next to each other...
- okanat 2y ago
  and unless some of the shovel shops not have an iron mine, a steel factory and a stamping shop as well. They can sell the stuff 1/5 of the price and you're not getting rich anytime soon.
- iknownthing 2y ago
  at what point does a goldrush become a shovelrush?
- kirubakaran 2y ago
  Yes, Brannan cornered the market before he sold the shovels. Selling shovels is not that profitable if you skip that "Step 1". > he owned the only store between San Francisco and the gold fields — a fact he capitalized on by buying up all the picks, shovels and pans he could find, and then running up and down the streets of San Francisco, shouting 'Gold! Gold on the American River!' He paid 20 cents each for the pans, then sold them for $15 a piece. In nine weeks, he made $36,000." https://en.wikipedia.org/wiki/Samuel_Brannan https://en.wikipedia.org/wiki/Samuel_Brannan
weitendorf 2y ago
Great article, and pretty relevant to what I'm building (cloud developer tooling, including some genai, but also including non-AI tools + an application platform. Email me if interested.). Obviously I'm not nearly as pessimistic about it. Zoom out for a sec and generalize to SaaS in general, not just AI infra (a subset of Saas) - all the arguments listed apply there too, except the data moat (which honestly doesn't matter to tons and tons of AI infra companies. That's more of an AI application problem). Now of course most startups are doing AI at least a bit, but in the past decade we've seen plenty of SaaS vendors compete with incumbents either head on or by carving out their own niche. In fact, two of the companies the author considers "incumbents" are arguably still challengers, but definitely were in this exact situation just a few years ago: Vercel and Databricks. Also, competition from incumbents is hardly a deathknell. There's room for multiple products in some market segments - how many RDBMS companies are there? Competition from a huge incumbent in many ways comes with benefits, because it helps grow the overall market and awareness of the product space, including your own product. I suppose according to this author I'm in the "application layer" even though really I'm in the AI-application-layer-now-but-not-later-layer, software-infrastructure-layer. And that's great because I actually do have experience in that specific application area. But honestly, saying "you ought to have expertise in your domain" is 1) duh 2) in the examples (llamaindex parsing/ocr, langchain llmops + agnetic stuff), there is clearly a big enough twist on doing it "but with AI" that the application/vertical is close to novel. Successful challengers create valuable businesses without prior deep expertise in their domain all the time and I don't really see how this is any different. Basically, you could repeat this for any SaaS business. Starting a company is hard, but I don't know if AI infra is uniquely hard in the ways laid out.
- latchkey 2y ago
  Build bare metal developer tooling for GPUs and you'll get aquihired.
openrisk 2y ago
Yet there is little "AI" specific in this AI infrastructure startup challenge: 1) insane levels of competition towards any goal make relavant minor, secondary, traits that are not obvious before hand. Pure luck becomes more important. 2) excess market concentration (of which the tech sector is maybe the most egregious example) makes any new initiative harder. The more dominant and controlling the incumbents the harder to find a decent sized niche to grow. 3) selling to risk averse enterprizes / organizations is always an uphill battle that requires climbing a mountain of bureaucracy and regulation, only to eventually face random internal politics. In the end the current craze will certainly produce a modified tech landscape. These recurring hypes always overpromise and underdeliver, but a cumulative effect is slowly happening. In such stormy seas its hard to identify an optimal course and strategy. Riding every hype wave may sound silly but might work. On the other extreme, one may seek beacons indicating eventual stable land and try to navigate there. Good luck
swalsh 2y ago
I'm not sure I'm exactly at the edge of things, but I have 2 companies trying to setup regular meetings with me to be a beta customer. Both have promised I can help define a new product, but when I list my real problems... they aren't in the mission. Everyone wants to solve RAG (that's easy, don't need help) or they want to give me a gui I don't need, or wrap open source software like vllm. Or "solve privacy" (which usually comes in the form of masking... which surprise, that works for PII, but not PHI... I need the protected information). Want to solve a real problem, help me create custom benchmarks, clean my data, get my small parameter model to reason better etc.
- benrutter 2y ago
  This rings so true! I think it's natural whenever there's a new technology that a lot of start ups spring up with a vibe of "GenAI is cool, let's do something with that!", which is 100% the wrong way to go about building something. Starting by investing yourself fully into a given problem, and fixing it with the most appropriate tool (might be GenAI, might not) is much more likely to end in something people actually want or need. Doing the reverse, and trying to find an existing problem that matches a solution you've already picked is how you end up with hundreds of companies selling thin API wrappers for ChatGPT.
- tinco 2y ago
  We had the exact same problem before genAI became the next big thing. All the startups were selling generic fine tuning and labeling services both of which are super easy to build, and they didn't even work on our unique super high quality super high resolution 40TB dataset. Our problem was we had a real world problem and real data. All the startups were solving for imaginary problems and had no data.
  - llm_trw 2y ago
    Sounds like you need a consultancy and not a startup to solve your problem.
    - BillyTheKing 2y ago
mschwarz 2y ago
Why does the author claim that Adept was acquired by Amazon? The linked article says they hired away the CEO and key staff.
- justincormack 2y ago
  It was a weirdly structured deal that in effect was an acquisition. The investors were paid off.
physicsguy 2y ago
If I'm an application developer or manager, at any >100 person company, it normally doesn't fall into my remit to go out and pick a new company to contract with to provide services. Typically, it gets harder and harder to do that. Even with LLM stuff, we're contracting that through our existing relationship with Microsoft. When evaluating infra options, it therefore is a huge barrier to entry for most developers if there's a 'good enough' option on one of the main cloud providers
- cageface 2y ago
  I've experienced this too. Any new service that requires more than a credit card number has to go through the legal department to review the contract and that means it goes to the bottom of the pile of similar requests and probably won't even get a look for months.
siva7 2y ago
It bugs me that all we are seeing in the vc-backed startup scene seems to be ai infrastructure startups. We got something close to ai and all people come up with is they want to be the next ai marketplace store or the millionth infrastructure startup that does exactly the same like their competitors. How boring.
- teaearlgraycold 2y ago
  Some of us are working on synthetic data
- nextworddev 2y ago
  Author here. Wait until you see the ChatGPT for Law landscape! /s
  - GoAwayPulomi 2y ago
    This will be as fun of a read in 7 years as reading about IBM Watson's impact on the medical field is now.
- pas 2y ago
  YC funds 200-500 companies each year, there's plenty of interesting ones.
Simon_ORourke 2y ago
I've come across a number of these AI infra start-ups like Scale AI and Zerve and TBH I'm amazed they can do what they do with relatively small teams when you have Meta and Apple somewhat struggling in this area and buying rather than building themselves.
curious_cat_163 2y ago
Articles like this represent reminder that we are in the middle of a hype cycle. [1] I don’t say that thinking that LLMs (really: Transformers and the corresponding scaling of compute around it) don’t represent a step change. I say that because I am very sure that we are going to see a slope of enlightenment that results in products that improve the quality of human life. [1] https://en.m.wikipedia.org/wiki/Gartner_hype_cycle https://en.m.wikipedia.org/wiki/Gartner_hype_cycle
- nextworddev 2y ago
  Author here - the article isnt questioning the value of AI per se, but value capture and competitive dynamics. Internet routers remained valuable but got commoditized, sort of a thing.
ankit219 2y ago
Great article. I am not going to name names, but over the last one year, whenever there is a concept that became popular in Gen AI, thousands of startups pivoted to doing that. Many come from software background where the expectation was that if the code works on one dataset, it would work for everything. You can see this with 1/ Prompt engineering 2/ RAG 3/ and now, after Apple's WWDC, it's adapters. Enterprises I have spoken to says they are getting pitched by 20 startups offering similar things on a weekly basis. They are confused on what to go with. From my vantage point (and may be wrong), the problem is many startups ended up doing the easy things - things which could be done by an internal team too, and while it's a good starting point for many businesses, but hard to justify costs in the long term. At this point, two clear demarcations appear: 1/ You make an API call to OpenAI, Anthropic, Google, Together etc. where your contribution is the prompt/RAG support etc. 2/ You deploy a model on prem/private VPC where you make the same calls w RAG etc. (focused on data security and privacy) First one is very cheap, and you end up competing with Open AI and hundred different startups offering it. Plus internal teams w confidence that they can do it themselves. Second one is interesting, but overhead costs are about $10,000 (for hosting) and any customer would expect more value than what a typical RAG provides. Difficult to provide that kind of value when you do not have a deep understanding and under pressure to generate revenue. I don't fully believe infra startups are a tarpit idea. Just that, we havent explored the layers where we can truly find a valuable thing that is hard to build for internal teams.
- spacecadet 2y ago
  Its rent seeking and grifting. As technology has become easier to get into, a huge number of "startups" are low level people just looking to make noise, get their cut, and bail. Its a bad look, up there with fast fashion. An acquisition here amounts to teams luck surface.
- hobs 2y ago
  Pretty much this, 18 months ago my CEO told me we HAD to get into this space, and I told him that basically our money came from our private product and that the only way our big enterprise customers were going to play game with us was either ironclad agreements that went all the way to openai, or more likely a completely single tenant system, which would cost far more than they were willing to pay. Of course they went with both, and as far as I can tell both are a major disaster post layoffs :)
blitzar 2y ago
Did any of the startups in question actually ever want to build AI infrastructure or did they all pivot from Metaverse to Crypto to AI in the great pivotting of 2022. Given VC's penchant for throwing cash at grifters in the latest hype space is it any suprise that some of the beneficiaries are looking for a quick exit before they have to do any actual work?
- EdwardDiego 2y ago
  Yeah those grifter pivots were amazing. Like professional ice skaters. I'm just surprised the Long Island Iced Tea Corp / Long Blockchain Corp hasn't become the Long LLM Corp yet.
Juliate 2y ago
Gen AI feels more and more like NFTs and blockchains, and overall, a lot like pre-2001-bubble (or more accurately post y2k). A very exciting and expensive solution in search of an actual problem, that will ultimately find its way, commoditised, in a small niche, while adjacent technologies take the lead for productive use-cases.
cootsnuck 2y ago
I work for a foundational AI company. I guess we're technically AI infra. We're inherently "narrowly" focused since our origin (which was well before the recent hype in past 2-3 years). Our customers are really the type of AI infra companies being talked about in this article. And yea, the new ones I work with everyday are often a dime a dozen. A revolving door of small startups trying to make the same general purpose AI infra targeting other traditional "boring enterprise infra" companies. The ones that I'm seeing get the most traction, have the best products, and best chances of success have zeroed in on specific niches and sub-industries. (Think AI infra that helps B2B2B companies where that last "B" is like Roofing companies and the value provided is helping Roofing companies easily and drastically scale their outbound and inbound marketing and sales.) The startups I work with that make me scratch my head are the ones trying to build "disruptive" AI infra that does nothing different, provides nothing special, other than potentially nice UI/UX, and is liable to have their lunch eaten by either natural iterations and improvements of our own services they essentially just white label, or some other incumbent. To me, it's like trying to create a new company to compete against Walmart and Target on groceries because they're too massive scale to win against "a well tailored customer experience" but then forgetting Costco, Aldi's, Trader Joes, and Whole Foods exist. And why would any of those aforementioned companies feel the need to acquire you rather than casually crush you as they go about their business either ignoring you as you wither or taking your good ideas and incorporating them into their own offering? It's not impossible, just has to make sense and even then a certain degree of "the stars aligning" is required. Which is why there inevitably can only be a small group of winners out of this massive sea of hopefuls. And I of course can only shrug my shoulders if asked if the AI infra startup I work at is differentiated, necessary, and lucky enough to be at the finish line with the survivors at the end. (We're finding our PMF and potential road to incumbency mainly with two-ish markets: old and new school enterprise infra and non-tech Fortune 500 type of companies.)
- nextworddev 2y ago
  Author here. Thanks for the perspective. p.s. I do hope AI startups not estimate how hard it is to break into vertical markets which have their own challenges
rgavuliak 2y ago
It's as if all of the AI devex/infra companies are cargo culting the story of how the people that made the most money in the gold rush were the people who sold the tools. The thing is that the tools were well understood and battle tested.
- weitendorf 2y ago
  GenAI applications are so finnicky that it’s easier to build a company around tools-for-AI than a company fitting their archtypical user profile doing AI applications (that’s actually profitable. Most of their customers likely aren’t anywhere close). That’s inverted from the prior SAAS/cloud boom. I too think there are too many shovel chasers but I think it’s also a consequence of what’s easier to ship.
- hibikir 2y ago
  And this isn't just a matter of AI: You see all kinds of companies trying to provide value adds on top of cloud: "We will annotate DNA for you as a service!" When all they do is dockerize the same tools their customers use, and serve as small shims for the least sophisticated customers. The moment said customer grows, they understand they can replace the vendor with less than a week of work. The people making shovels make the money by having strong profit margins, becoming a default vendor, and having a moat. Good luck doing that in AI! And my favorite counter example of selling tools is precisely docker: They built tech used everywhere... yet how much value they captured? It's tge same story all over dev tool space.
htrp 2y ago
Selling shovels is a way to make money, but definitely not at venture scale.
diwank 2y ago
While I agree that AI infra startups are hard to build, I strongly disagree with the idea that they are harder than foundational or application layer startups. I think it boils down to what you know and what resources you can muster. For instance, foundational AI startups are also ridiculously hard to build. You need an insane amount of funding, spend it pretraining models to stay competitive only to find that gains in hardware and model architecture make them obsolete within months plus there's no real guarantee that scaling will keep working. Application layer startups are hard in a very different way, there's an insane amount of competition and new capabilities are emerging every few weeks. I have worked with a few AI girlfriend startups and they are really struggling with keeping apace and warding off ridiculous amount of competition. I think it's really just YMMV. Of course, the deeper you get into the stack, the more monopolizing pressure there is. Is it hard to build AI infra startups? Yes 100%. Will there be very few winners? Yes. Is it harder than foundational or application layer startups? Depends on the founders' strengths. Is it Is it a lost cause? I really don't think so.
- nextworddev 2y ago
  Author here. Yes, I explicitly called out the danger of thinking application layer startups are easier, because it totally depends on the founding teams' backgrounds and interests.
mushufasa 2y ago
Curious for discussion: Does this logic also apply to industry-specific "AI Infra?," where the APIs are wrapping a service that solves a domain-specific problem using AI, rather than general purpose infra technology? And provides those APIs to other businesses within that industry?
sitkack 2y ago
“Insanely Hard” an overused term that is now a cliche. We don’t build ffmpeg wrappers as a service because it is easy.
throwaway2037 2y ago
This is a well-written blog post. Thank you to share. This part: > For AI infra startups to be “venture scale”, they will eventually need to win over enterprise customers. No question. That requires the startups to have some sustainable edge that separates their products from the incumbents’ (GCP, AWS, as well as the likes of Vercel, Databricks, Datadog, etc). On the surface, I agree. But look at a parallel market segment: Cheap cloud hosting. Think: Linode (or any of its competitors). There are a bunch of cheap cloud providers who are more than 10 years old. They didn't all get bought out nor bankrupt by up-starts. Why? They must add just enough value to stay in business. Could we see something similar in the AI infra space? In fact, it looks more logical for the cheap cloud providers to try to build some AI infra -- low hanging fruit, to help with LLM training. (I am sure they already see GPU time.)

192 points by nextworddev 2y ago

deleted 2y ago

[deleted]

Good article, but what is the alternative? What can you build today as a software engineer that can have impact? Nothing seems to come close to AI / AI infra, even of its hard / risky / a moving landscape.

hnlmorg 2y ago
Everything we build has some kind of impact. At risk of getting philosophical, I’d ask yourself what your goals actually are if you feel only AI can have the impact you desire.
- fhd2 2y ago
  Not sure why this is down voted, that is the key question. Impact means different things to people. Could be: 1. Building a sustainable business and making decent money 2. Building a market leader and making ludicrous amounts of money 3. Advancing the state of the art in technology 4. Helping people with their little daily struggles 5. Solving pressing problems humanity is facing Or many other things I suppose. Now if you believe that AI is eventually going to make anything humans can build now redundant, that'd be a reason to believe nothing else matters in the end I suppose. But even if we get there, there's a lot of road leading to that destination. Any step provides value. Software built today can provide value even if nobody is going to need it ten years from now. And it's not like you could even predict that.
  - mirekrusin 2y ago
    The motive is to get acquired in most cases. It’s obvious and starts to make sense when you see startup that has no feasible monetisation strategy on the horizon, yet they exist and get funding. They’re betting on building infra to be hopefully used in large corp and this is their demo/PoC.
astronautas 2y ago
Data infra?
lionkor 2y ago
A lot of things, if you're okay with not chasing the next hype bubble
mlsu 2y ago
I would almost invert that statement. Sorry if this comes off ranty, but what exactly are people doing in the "AI space" currently that isn't "undifferentiated spam/chatbot" being sold to non-techies who heard about AI on NPR? What are real people using "AI" for that is so insanely valuable today? How much "company Y: same product with a chat window, sparks emoji" do we all need before this thing levels out and we all take a breather on the hype?
- jdross 2y ago
  personally? - writing and refactoring code. probably 50 times a day now - improving documentation across the company - summarizing meetings automatically with follow ups - drafting most legal work before a lawyer edits (saved 70% on legal bills) - entity extraction and data cleanup for my users
  - mlsu 2y ago
    Put a number on it. How much value of this will they capture from you personally (we'll assume, very very charitably by the sound of it, that you represent an "average" user of AI products) when this market matures? Exactly how much will your employer pay for a meeting summarizer? $10/mo a seat, $20/mo a seat, $50/mo a seat? Could the product sustain a 5x, 10x, 50x price hike that is going to have to happen to recoup the investment being made today?
    - fhd2 2y ago
      Agreed. Even if right now this seems like stuff companies want to throw money at for novelty/FOMO related reasons, I think eventually reality ought to catch up. Probably an unpopular opinion, but I think the most efficient companies of the future will tackle the ironies of automation effectively: Carefully designing semi automation that keeps humans in the loop in a way that maximises their value - as opposed to just being bored rubber stamping the automation without really paying attention.
      - per1Peteia 2y ago
ianpurton 2y ago
It's fine to be in AI. My takeaway from the article is instead of being a Gen AI startup be a Gen AI startup for a specific use case.
oivey 2y ago
Slightly different take than some of the siblings: you can still just build this stuff. If your goal is impact, maybe the best place to do it will be at a cloud vendor or other big corp. If your goal is actually just a big VC exit, then maybe not. If your product is something that can be ripped off in 3 months, then it probably wasn’t going to have a long term impact anyway.
ehnto 2y ago
All the same stuff, to be honest. If AI is set to replace human work, well we have had a cheap human labour market for decades and yet we still need software. An LLM can't replace a business itself, which is made up of niche processes, direction and purpose, which we sometimes codify into a SaaS. We'll still need to do all that even if AI replaces some of the human parts of the business.
jononor 2y ago
Anything SaaS that solves a painpoints for established industries. Those that have billions of turnaround for decades already, are not good at building tech themselves, and buy solutions/services to run their business. Bonus for low barriers to entry. Agriculture, logistics, real estate, energy, etc.
- swalsh 2y ago
  I have a theory that the days of established businesses that don't know tech is dwindling. A lot of companies which has adopted tech has started building a small foundation of talent internally. I think you're seeing this trend accelerate with the large tech companies laying people off. I have heard about top grade data science talent landing at some small sized health plan. My companies fastest growing competitor is "internally sourced departments" of the services we provide.
  - nextworddev 2y ago
    You confirm my observation as well. Even motel chains have developers building internal tools these days
  - jononor 2y ago
    Yes computer savvyness is on the rise, and have been for decades, and this will continue. But there are many levels: Ability to be a competent user, and competent buyer, ability to build it themselves. Then there are big difference in buy vs build culture. And preferences for type of solutions that are default buy vs default build. And finally, smaller and medium sized organizations are less likely to have internal teams. All this should be analyzed for the specific market, product and customer segment one targets.
rsynnott 2y ago
Define ‘impact’. Does ‘impact’ here mean ‘tickles the fancy of a 2024-era VC’? If so, you may be right. If used in its common meaning, absolutely not; most of this stuff is ~useless.
_el1s7 2y ago
If you don't know what to build, you don't build.
notamy 2y ago
> What can you build today as a software engineer that can have impact? Quite a bit, if you don’t follow the standard tech hype. Find an industry that isn’t tech-first and you’ll notice that there’s a lot of room for improvement.

threeseed 2y ago

Way too many founders don't understand the impact of competing with cloud vendors. Almost all enterprises have pre-committed budgets for cloud which means unless your product is FOSS it's going to be hard to convince someone to bet their business on it. Especially given that in this fundraising environment there is a 95% chance they won't be around in a year or two anyway. It's going to be a brutal few years especially if we are heading into a period of diminishing returns in terms of LLM accuracy.

devjab 2y ago
I’m also sort of curious as to how much of a market research they’ve done if they’re trying to compete with Azure and AWS. Even before the recent LLM rush took off, AI was a thing. In the city of Copenhagen there was a project to digitalise a few million case files (which is 10-100 documents per case file), and how it was done was basically with an intermediary company who knew the training and a cooperation with Microsoft. Yes, I’m dumping down the complexity of it all, but once the training period of half a year was over, Azure made a lot (and I mean a lot) of infrastructure available for not a lot of money and the process completed in a week or so. Since it had to happen and because it was a PoC the same project was also done by real humans. This was the “actual” project and every time deadline and whatnot the AI project had came from how long it would take X humans to do it. I can’t recall how many X was, but it was enough to meet the legal deadline for when these case files had to be digitised and sorted correctly. The human project was the result, and then the AI PoC was later used as a lesson on whether it could be done this way or not. It can, it was more accurate and not more expensive. Anyway… I’m not sure who would’ve been capable of competing with Azure. (Outside the usual suspects). Maybe a company of Hetzner could? But you would need someone who can offer you a massive amount of computing on demand, and the only companies which are going to have that are big vendors. Maybe it’s different with LLMs because the requirement is a continuous thing rather than something you need for a short period of time?
- deleted 2y ago
  [deleted]
- jononor 2y ago

BrunoJo 2y ago

Good tips, especially the point about narrowing the scope. At https://Lemonfox.ai https://Lemonfox.ai we started with a LLM, image and speech-to-text API. Now we are only focusing on the speech-to-text API as the other areas are already very crowded and there's a lack of innovation in the speech-to-text space.

anonylizard 2y ago
Like how do you plan on competing against multimodals, which keep getting cheaper and clearly can do audio->text? Or existing incumbents like Deepgram? Or just the generic APIs provided by the big clouds.
cootsnuck 2y ago
> Now we are only focusing on the speech-to-text API as the other areas are already very crowded and there's a lack of innovation in the speech-to-text space. I'm legitimately wondering how your hosted Whisper API for $0.17/hr is supposed to compete with groq's exact same API that costs $0.03/hr. You may be about to find out how crowded all of the AI infra spaces are. I strongly recommend narrowing your scope far beyond modality. If you've been working with this tech and getting familiar with it then you already have valuable expertise. Pivot now or panic later. If you want to stay in the speech space find what markets are being underserved with speech AI related solutions. Are there pain points there that can be solved by a STT API? If so, build those solutions. You can't compete at the infra layer and I'm not sure why you would want to try if you don't already have something unique about your offering beyond hosting open source models. It's never good if your competition is potentially just a single developer in a company standing up your entire service internally in a week. If you are determined to stay in the AI infra space then you'll need to be tackling a hard problem that companies want solved. Maybe take a look at fine-tuning models. Hard problem and maybe there's a hunger for it. (It's a risky one to tackle too though since it's very possible general/foundational models will maintain a grip on "good enough".)

JohnMakin 2y ago

As someone who has semi-unwillingly worked in infrastructure and infrastructure consulting most of my career - we’ve never even really solved that problem, what on god’s green earth convinced you AI did? I am genuinely curious.

anshulbhide 2y ago

[dead]

dvt 2y ago

Infra has always been a tarpit idea. Google didn't start out as an "infra" company, and neither did Amazon or Facebook. In fact, the few "infra companies" that did start back then (companies like Godaddy) are minuscule compared to the aforementioned. VC pouring money in LLM infra is legitimately crazy to me. It's clear as day that there will be winners of this AI cycle, but, as always, they will be companies that provide actual, real, tangible value. Making shovels works for huge companies like Nvidia or Intel, but it won't work for you. It's sad to see so much capital funneled in frameworks upon frameworks upon frameworks instead of fresh new ideas that could revolutionize the way we interact with our devices. I know it's a bit of a meme, but I'd rather see more Rabbit R1 and less LangChain. Even OpenAI doesn't really have a product. Just throwing data at a bunch of video cards isn't value-generating in itself. We need a Dropbox or a Slack or an Instagram: something people love that makes their life easier or better.

rapsey 2y ago
> Even OpenAI doesn't really have a product. They are making a ton of money off subscriptions.
- nextworddev 2y ago
  Yes. Author here. OpenAI is at near 4bn ARR and growing (api plus SaaS)
  - whiplash451 2y ago
    Source?
    - matwood 2y ago
      https://www.theinformation.com/articles/openais-annualized-revenue-doubles-to-3-4-billion-since-late-2023 https://www.theinformation.com/articles/openais-annualized-r...
- soneca 2y ago
  Yep. Obviously they have a product. Millions and millions of people that are not software engineers or any very tech-savvy persona use their product. And a lot of those people pay for it. It’s just silly to say they don’t really have a product. Specially given the rest of GP comment, OpenAI seems to be the Google, Facebook of the industry and will be the infrastructure company of it (already is kind of)

mehulashah 2y ago

The title is true. But, the arguments don't hold water for me. 12 years ago, I started a big data company. It looked similar for big data companies when Cloudera raised almost $1B in 2014. Too many people building data warehouses, especially in the cloud. I exited. Who knew that Snowflake and Databricks would emerge against the incumbents. Similarly, there will be winners in the AI infrastructure space. To win, you need to focus on your customers and delight them. Narrowing focus makes a lot of sense. Don't pay attention to the doom and gloom, or you'll never do a startup.

rapsey 2y ago
> Who knew that Snowflake and Databricks would emerge against the incumbents. Snowflake is not profitable. I doubt Databricks is. Their market and business is crap.
- pas 2y ago
  if they are not profitable with these prices ... what the fuck they are doing!? do they just have company coke-athons all day every day?
  - rapsey 2y ago
    It is a sales and engineering heavy business. Very difficult to generate large returns.
    - TimPC 2y ago
      They are becoming like SAP though where initially a company buys one service and it soon finds itself buying every adjacent service from them. If they manage to do that successfully they will be quite profitable.
    - pas 2y ago
      They are bundling open source stuff (Spark and Delta and so on), sure, they have their fancy IDE and whatnot on top, but they can let the community maintain things, scale back R&D, focus on things that matter to existing clients. They have 1.6B revenue, a 50% YoY growth. And still not profitable. Hm, okay, recent acquisition on ML stuff, and of course probably burning hundreds of millions on cloud-GPU-AI shit. Well, I guess as long as they have so big growth it makes sense to invest and raise ... and yeah that probably completely obscures the actual profitability of their core business. (Not to mention that they are probably spending all that money to try to expand their core business. To upgrade their value prop from cloud version of less-dumb-data-pipelines to 1-800-data-4-AI.)

xg15 2y ago

In a nutshell: If there's a goldrush, you get rich by selling shovels. ... unless there are already 200 shovel shops next to each other...

okanat 2y ago
and unless some of the shovel shops not have an iron mine, a steel factory and a stamping shop as well. They can sell the stuff 1/5 of the price and you're not getting rich anytime soon.
iknownthing 2y ago
at what point does a goldrush become a shovelrush?
kirubakaran 2y ago
Yes, Brannan cornered the market before he sold the shovels. Selling shovels is not that profitable if you skip that "Step 1". > he owned the only store between San Francisco and the gold fields — a fact he capitalized on by buying up all the picks, shovels and pans he could find, and then running up and down the streets of San Francisco, shouting 'Gold! Gold on the American River!' He paid 20 cents each for the pans, then sold them for $15 a piece. In nine weeks, he made $36,000." https://en.wikipedia.org/wiki/Samuel_Brannan https://en.wikipedia.org/wiki/Samuel_Brannan

weitendorf 2y ago

Great article, and pretty relevant to what I'm building (cloud developer tooling, including some genai, but also including non-AI tools + an application platform. Email me if interested.). Obviously I'm not nearly as pessimistic about it. Zoom out for a sec and generalize to SaaS in general, not just AI infra (a subset of Saas) - all the arguments listed apply there too, except the data moat (which honestly doesn't matter to tons and tons of AI infra companies. That's more of an AI application problem). Now of course most startups are doing AI at least a bit, but in the past decade we've seen plenty of SaaS vendors compete with incumbents either head on or by carving out their own niche. In fact, two of the companies the author considers "incumbents" are arguably still challengers, but definitely were in this exact situation just a few years ago: Vercel and Databricks. Also, competition from incumbents is hardly a deathknell. There's room for multiple products in some market segments - how many RDBMS companies are there? Competition from a huge incumbent in many ways comes with benefits, because it helps grow the overall market and awareness of the product space, including your own product. I suppose according to this author I'm in the "application layer" even though really I'm in the AI-application-layer-now-but-not-later-layer, software-infrastructure-layer. And that's great because I actually do have experience in that specific application area. But honestly, saying "you ought to have expertise in your domain" is 1) duh 2) in the examples (llamaindex parsing/ocr, langchain llmops + agnetic stuff), there is clearly a big enough twist on doing it "but with AI" that the application/vertical is close to novel. Successful challengers create valuable businesses without prior deep expertise in their domain all the time and I don't really see how this is any different. Basically, you could repeat this for any SaaS business. Starting a company is hard, but I don't know if AI infra is uniquely hard in the ways laid out.

latchkey 2y ago
Build bare metal developer tooling for GPUs and you'll get aquihired.

openrisk 2y ago

Yet there is little "AI" specific in this AI infrastructure startup challenge: 1) insane levels of competition towards any goal make relavant minor, secondary, traits that are not obvious before hand. Pure luck becomes more important. 2) excess market concentration (of which the tech sector is maybe the most egregious example) makes any new initiative harder. The more dominant and controlling the incumbents the harder to find a decent sized niche to grow. 3) selling to risk averse enterprizes / organizations is always an uphill battle that requires climbing a mountain of bureaucracy and regulation, only to eventually face random internal politics. In the end the current craze will certainly produce a modified tech landscape. These recurring hypes always overpromise and underdeliver, but a cumulative effect is slowly happening. In such stormy seas its hard to identify an optimal course and strategy. Riding every hype wave may sound silly but might work. On the other extreme, one may seek beacons indicating eventual stable land and try to navigate there. Good luck

swalsh 2y ago

I'm not sure I'm exactly at the edge of things, but I have 2 companies trying to setup regular meetings with me to be a beta customer. Both have promised I can help define a new product, but when I list my real problems... they aren't in the mission. Everyone wants to solve RAG (that's easy, don't need help) or they want to give me a gui I don't need, or wrap open source software like vllm. Or "solve privacy" (which usually comes in the form of masking... which surprise, that works for PII, but not PHI... I need the protected information). Want to solve a real problem, help me create custom benchmarks, clean my data, get my small parameter model to reason better etc.

benrutter 2y ago
This rings so true! I think it's natural whenever there's a new technology that a lot of start ups spring up with a vibe of "GenAI is cool, let's do something with that!", which is 100% the wrong way to go about building something. Starting by investing yourself fully into a given problem, and fixing it with the most appropriate tool (might be GenAI, might not) is much more likely to end in something people actually want or need. Doing the reverse, and trying to find an existing problem that matches a solution you've already picked is how you end up with hundreds of companies selling thin API wrappers for ChatGPT.
tinco 2y ago
We had the exact same problem before genAI became the next big thing. All the startups were selling generic fine tuning and labeling services both of which are super easy to build, and they didn't even work on our unique super high quality super high resolution 40TB dataset. Our problem was we had a real world problem and real data. All the startups were solving for imaginary problems and had no data.
- llm_trw 2y ago
  Sounds like you need a consultancy and not a startup to solve your problem.
  - BillyTheKing 2y ago

mschwarz 2y ago

Why does the author claim that Adept was acquired by Amazon? The linked article says they hired away the CEO and key staff.

justincormack 2y ago
It was a weirdly structured deal that in effect was an acquisition. The investors were paid off.

physicsguy 2y ago

If I'm an application developer or manager, at any >100 person company, it normally doesn't fall into my remit to go out and pick a new company to contract with to provide services. Typically, it gets harder and harder to do that. Even with LLM stuff, we're contracting that through our existing relationship with Microsoft. When evaluating infra options, it therefore is a huge barrier to entry for most developers if there's a 'good enough' option on one of the main cloud providers

cageface 2y ago
I've experienced this too. Any new service that requires more than a credit card number has to go through the legal department to review the contract and that means it goes to the bottom of the pile of similar requests and probably won't even get a look for months.

siva7 2y ago

It bugs me that all we are seeing in the vc-backed startup scene seems to be ai infrastructure startups. We got something close to ai and all people come up with is they want to be the next ai marketplace store or the millionth infrastructure startup that does exactly the same like their competitors. How boring.

teaearlgraycold 2y ago
Some of us are working on synthetic data
nextworddev 2y ago
Author here. Wait until you see the ChatGPT for Law landscape! /s
- GoAwayPulomi 2y ago
  This will be as fun of a read in 7 years as reading about IBM Watson's impact on the medical field is now.
pas 2y ago
YC funds 200-500 companies each year, there's plenty of interesting ones.

Simon_ORourke 2y ago

I've come across a number of these AI infra start-ups like Scale AI and Zerve and TBH I'm amazed they can do what they do with relatively small teams when you have Meta and Apple somewhat struggling in this area and buying rather than building themselves.

curious_cat_163 2y ago

Articles like this represent reminder that we are in the middle of a hype cycle. [1] I don’t say that thinking that LLMs (really: Transformers and the corresponding scaling of compute around it) don’t represent a step change. I say that because I am very sure that we are going to see a slope of enlightenment that results in products that improve the quality of human life. [1] https://en.m.wikipedia.org/wiki/Gartner_hype_cycle https://en.m.wikipedia.org/wiki/Gartner_hype_cycle

nextworddev 2y ago
Author here - the article isnt questioning the value of AI per se, but value capture and competitive dynamics. Internet routers remained valuable but got commoditized, sort of a thing.

ankit219 2y ago

Great article. I am not going to name names, but over the last one year, whenever there is a concept that became popular in Gen AI, thousands of startups pivoted to doing that. Many come from software background where the expectation was that if the code works on one dataset, it would work for everything. You can see this with 1/ Prompt engineering 2/ RAG 3/ and now, after Apple's WWDC, it's adapters. Enterprises I have spoken to says they are getting pitched by 20 startups offering similar things on a weekly basis. They are confused on what to go with. From my vantage point (and may be wrong), the problem is many startups ended up doing the easy things - things which could be done by an internal team too, and while it's a good starting point for many businesses, but hard to justify costs in the long term. At this point, two clear demarcations appear: 1/ You make an API call to OpenAI, Anthropic, Google, Together etc. where your contribution is the prompt/RAG support etc. 2/ You deploy a model on prem/private VPC where you make the same calls w RAG etc. (focused on data security and privacy) First one is very cheap, and you end up competing with Open AI and hundred different startups offering it. Plus internal teams w confidence that they can do it themselves. Second one is interesting, but overhead costs are about $10,000 (for hosting) and any customer would expect more value than what a typical RAG provides. Difficult to provide that kind of value when you do not have a deep understanding and under pressure to generate revenue. I don't fully believe infra startups are a tarpit idea. Just that, we havent explored the layers where we can truly find a valuable thing that is hard to build for internal teams.

spacecadet 2y ago
Its rent seeking and grifting. As technology has become easier to get into, a huge number of "startups" are low level people just looking to make noise, get their cut, and bail. Its a bad look, up there with fast fashion. An acquisition here amounts to teams luck surface.
hobs 2y ago
Pretty much this, 18 months ago my CEO told me we HAD to get into this space, and I told him that basically our money came from our private product and that the only way our big enterprise customers were going to play game with us was either ironclad agreements that went all the way to openai, or more likely a completely single tenant system, which would cost far more than they were willing to pay. Of course they went with both, and as far as I can tell both are a major disaster post layoffs :)

blitzar 2y ago

Did any of the startups in question actually ever want to build AI infrastructure or did they all pivot from Metaverse to Crypto to AI in the great pivotting of 2022. Given VC's penchant for throwing cash at grifters in the latest hype space is it any suprise that some of the beneficiaries are looking for a quick exit before they have to do any actual work?

EdwardDiego 2y ago
Yeah those grifter pivots were amazing. Like professional ice skaters. I'm just surprised the Long Island Iced Tea Corp / Long Blockchain Corp hasn't become the Long LLM Corp yet.

Juliate 2y ago

Gen AI feels more and more like NFTs and blockchains, and overall, a lot like pre-2001-bubble (or more accurately post y2k). A very exciting and expensive solution in search of an actual problem, that will ultimately find its way, commoditised, in a small niche, while adjacent technologies take the lead for productive use-cases.

cootsnuck 2y ago

I work for a foundational AI company. I guess we're technically AI infra. We're inherently "narrowly" focused since our origin (which was well before the recent hype in past 2-3 years). Our customers are really the type of AI infra companies being talked about in this article. And yea, the new ones I work with everyday are often a dime a dozen. A revolving door of small startups trying to make the same general purpose AI infra targeting other traditional "boring enterprise infra" companies. The ones that I'm seeing get the most traction, have the best products, and best chances of success have zeroed in on specific niches and sub-industries. (Think AI infra that helps B2B2B companies where that last "B" is like Roofing companies and the value provided is helping Roofing companies easily and drastically scale their outbound and inbound marketing and sales.) The startups I work with that make me scratch my head are the ones trying to build "disruptive" AI infra that does nothing different, provides nothing special, other than potentially nice UI/UX, and is liable to have their lunch eaten by either natural iterations and improvements of our own services they essentially just white label, or some other incumbent. To me, it's like trying to create a new company to compete against Walmart and Target on groceries because they're too massive scale to win against "a well tailored customer experience" but then forgetting Costco, Aldi's, Trader Joes, and Whole Foods exist. And why would any of those aforementioned companies feel the need to acquire you rather than casually crush you as they go about their business either ignoring you as you wither or taking your good ideas and incorporating them into their own offering? It's not impossible, just has to make sense and even then a certain degree of "the stars aligning" is required. Which is why there inevitably can only be a small group of winners out of this massive sea of hopefuls. And I of course can only shrug my shoulders if asked if the AI infra startup I work at is differentiated, necessary, and lucky enough to be at the finish line with the survivors at the end. (We're finding our PMF and potential road to incumbency mainly with two-ish markets: old and new school enterprise infra and non-tech Fortune 500 type of companies.)

nextworddev 2y ago
Author here. Thanks for the perspective. p.s. I do hope AI startups not estimate how hard it is to break into vertical markets which have their own challenges

rgavuliak 2y ago

It's as if all of the AI devex/infra companies are cargo culting the story of how the people that made the most money in the gold rush were the people who sold the tools. The thing is that the tools were well understood and battle tested.

weitendorf 2y ago
GenAI applications are so finnicky that it’s easier to build a company around tools-for-AI than a company fitting their archtypical user profile doing AI applications (that’s actually profitable. Most of their customers likely aren’t anywhere close). That’s inverted from the prior SAAS/cloud boom. I too think there are too many shovel chasers but I think it’s also a consequence of what’s easier to ship.
hibikir 2y ago
And this isn't just a matter of AI: You see all kinds of companies trying to provide value adds on top of cloud: "We will annotate DNA for you as a service!" When all they do is dockerize the same tools their customers use, and serve as small shims for the least sophisticated customers. The moment said customer grows, they understand they can replace the vendor with less than a week of work. The people making shovels make the money by having strong profit margins, becoming a default vendor, and having a moat. Good luck doing that in AI! And my favorite counter example of selling tools is precisely docker: They built tech used everywhere... yet how much value they captured? It's tge same story all over dev tool space.

htrp 2y ago

Selling shovels is a way to make money, but definitely not at venture scale.

diwank 2y ago

While I agree that AI infra startups are hard to build, I strongly disagree with the idea that they are harder than foundational or application layer startups. I think it boils down to what you know and what resources you can muster. For instance, foundational AI startups are also ridiculously hard to build. You need an insane amount of funding, spend it pretraining models to stay competitive only to find that gains in hardware and model architecture make them obsolete within months plus there's no real guarantee that scaling will keep working. Application layer startups are hard in a very different way, there's an insane amount of competition and new capabilities are emerging every few weeks. I have worked with a few AI girlfriend startups and they are really struggling with keeping apace and warding off ridiculous amount of competition. I think it's really just YMMV. Of course, the deeper you get into the stack, the more monopolizing pressure there is. Is it hard to build AI infra startups? Yes 100%. Will there be very few winners? Yes. Is it harder than foundational or application layer startups? Depends on the founders' strengths. Is it Is it a lost cause? I really don't think so.

nextworddev 2y ago
Author here. Yes, I explicitly called out the danger of thinking application layer startups are easier, because it totally depends on the founding teams' backgrounds and interests.

mushufasa 2y ago

Curious for discussion: Does this logic also apply to industry-specific "AI Infra?," where the APIs are wrapping a service that solves a domain-specific problem using AI, rather than general purpose infra technology? And provides those APIs to other businesses within that industry?

sitkack 2y ago

“Insanely Hard” an overused term that is now a cliche. We don’t build ffmpeg wrappers as a service because it is easy.

throwaway2037 2y ago

This is a well-written blog post. Thank you to share. This part: > For AI infra startups to be “venture scale”, they will eventually need to win over enterprise customers. No question. That requires the startups to have some sustainable edge that separates their products from the incumbents’ (GCP, AWS, as well as the likes of Vercel, Databricks, Datadog, etc). On the surface, I agree. But look at a parallel market segment: Cheap cloud hosting. Think: Linode (or any of its competitors). There are a bunch of cheap cloud providers who are more than 10 years old. They didn't all get bought out nor bankrupt by up-starts. Why? They must add just enough value to stay in business. Could we see something similar in the AI infra space? In fact, it looks more logical for the cheap cloud providers to try to build some AI infra -- low hanging fruit, to help with LLM training. (I am sure they already see GPU time.)