3 ms·
Agree, intervention is needed. Any new entrant sort-of has to prove itself also as crawling sites can be hard, e.g. Cloudflare which fronts an appreciable port
by ricardo81 2y ago
Agree, intervention is needed.
Any new entrant sort-of has to prove itself also as crawling sites can be hard, e.g. Cloudflare which fronts an appreciable portion of the web. Also large sites like Reddit you need to crawl them at 100s of pages a second to keep up to date with content, not something they're generally going to appreciate without a recognisable benefit to them.
- ADeerAppeared 2y ago> Any new entrant sort-of has to prove itself also as crawling sites can be hard It's very hard, getting harder as websites grow more hostile to crawlers. (With more and more putting up login walls to keep poorly written AI crawlers out.) But the real problem is that tech is culturally unused to dealing with monopolists "on even ground". The wider tech community operates on the implied worldview that they can take on anyone by just asking daddy A16Z for a duovigintillion dollars. For 25 nearly-continuous years, you could just outspend to get into even the most hostile markets. That economic climate is just ... gone now. No more free money. You can't just buy your way to the top anymore, the business has to financially work. And most importantly: You can't outspend the incumbents anymore, but they still can outspend you.