4 ms·

At the risk of oversimplifying things from a distance, this site -- especially the free, public-facing part of it -- seems like it would be an ideal candidate f

At the risk of oversimplifying things from a distance, this site -- especially the free, public-facing part of it -- seems like it would be an ideal candidate for a rewrite using static site generator/framework. That, coupled with a bot-aware CDN should keep them online at a reasonable cost for many years to come. Otherwise, I'm very curious to know more about their old and new architecture and what sorts of mitigation/scaling strategies they've started using to keep the site online.

tengada1 3mo ago
Yeah this seems like one of the easiest access models to create an excellent security model for -- basically just static publishing. Sounds like they were in need of a rewrite anyway! It probably seems daunting but to be honest this feels like a weekend's work at this point with LLM assistance. Not to be glib!
- jaredwiener 3mo ago
  But it also destroys the business model behind the site. Sure, you could technically redesign to handle the bot traffic, but if the bot traffic is just taking the data and reducing any need for humans to visit the site, why is he putting in the effort to maintain the site?
  - hyperhello 3mo ago
    Well, why is he? If it's for ad revenue, this problem is surely global to the web. If it's because he likes to do it, it shouldn't matter if one AI or a trillion scrape the site. I feel like we badly need a rethink of the web architecture after DoubleClick anyway. Maybe the name of the game should be to cut down a site's assets very hard and use static hosting for them. This interferes with crummy sites that show a mess of inlined ads every refresh, but now that there are a billion 'poor users' this is no longer feasible.
    - jaredwiener 3mo ago
      Maybe more an inherent problem with these chatbots. LLMs are great, but they aren't producing new information. You still need people for that. But if you cut down any incentive for the people to do that, the LLMs will starve.
  - tptacek 3mo ago
    The business model of the site is apparently private data sales, not ad revenue.
    - jaredwiener 3mo ago
      Bots don't make purchasing decisions.
      - Terr_ 3mo ago
        Some people are already trying to make that happen, but I'm convinced it'll be a net-negative.
        jaredwiener 3mo ago
        Rephrase: Bots SHOULDNT make purchasing decisions.
        fragmede 3mo ago
        Because... ? If I'm operating a site, and I want $X to allow a bot to scrape my site, why shouldn't the bot be allowed to make the purchasing decision to scrape the site? Obviously the bot owner would have their own set of guardrails, but if it allows sites like The numbers to stay up because bots aren't going to look at ads so the old model of displaying ads isn't going to work, I have a hard time seeing that as a bad thing.
        jaredwiener 3mo ago
        I mean it from the consumer perspective.
      - FinnKuhn 3mo ago
        According to the post inquiries for purchasing the data increased as a result though so they might not make the decision, but they do make the suggestion.
RobotToaster 3mo ago
Or just use a cache.
zackmorris 3mo ago
Ya this problem was solved 20 years ago with Varnish cache and Coral cache/CDN. I think what's really going on is that bots expose how underpowered web servers has gotten in recent years. In the 2000s, even poorly-architected PHP sites tended to serve about 200 requests per second, with 1000+ being common for static sites. I remember when Node.js came out and claimed that it could serve more like 100,000 RPS due to its cooperative threading model. But today sites have a remarkable slowness to them, running many hundreds or thousands of database queries due to ORMs and N+1 problems, so that response times can be 500 ms or more and even 1000 simultaneous users stresses servers. What went wrong is that nobody solved stuff like Russian doll caching in a general way separate from the programming language and database. We should have had ways to make dependency graphs using Etag headers as keys with real cache invalidation of dependent data. I went down that rabbit hole 10 years ago using touch events in Laravel with callbacks to handle cache invalidation when class model data was saved to the database. Also a query cache using Redis which I think might have been handled better at the database level anyway. After that experience, I can honestly say that cache invalidation is so difficult to get right that it's effectively an open problem. Meaning that programmings should use a package instead of rolling it by hand, and it should be a major concern from the start (along with sharding by user id or using something like Firebase). Don't get me started on how the web should have been a P2P content-addressable memory anyway. Nearly everything should be available from a nearby edge peer, similarly to BitTorrent. But nobody bothered to solve how to make that work with HTTPS/SSL. I suspect that has to do with early flaws in the browser security model where the whole page has to be behind HTTPS or warnings appear. So it was never clear what was personally identifiable information (PII) or merely public data being served over HTTPS. To really solve that, we probably need real trust networks and maybe even zero-knowledge proofs. Since these problems are so challenging to fix, and big companies can't be bothered to do it since they pulled the ladder up behind them, we're probably stuck with banal "are you human" challenge screens for the foreseeable future.
- withinboredom 3mo ago
  They are indeed quite challenging. We literally wrote a paper about it (https://arxiv.org/abs/2605.09114 https://arxiv.org/abs/2605.09114) ... you almost describe some of our original architecture too! You might find it interesting -- or not. https://getswytch.com https://getswytch.com

tengada1 3mo ago

Yeah this seems like one of the easiest access models to create an excellent security model for -- basically just static publishing. Sounds like they were in need of a rewrite anyway! It probably seems daunting but to be honest this feels like a weekend's work at this point with LLM assistance. Not to be glib!

jaredwiener 3mo ago
But it also destroys the business model behind the site. Sure, you could technically redesign to handle the bot traffic, but if the bot traffic is just taking the data and reducing any need for humans to visit the site, why is he putting in the effort to maintain the site?
- hyperhello 3mo ago
  Well, why is he? If it's for ad revenue, this problem is surely global to the web. If it's because he likes to do it, it shouldn't matter if one AI or a trillion scrape the site. I feel like we badly need a rethink of the web architecture after DoubleClick anyway. Maybe the name of the game should be to cut down a site's assets very hard and use static hosting for them. This interferes with crummy sites that show a mess of inlined ads every refresh, but now that there are a billion 'poor users' this is no longer feasible.
  - jaredwiener 3mo ago
    Maybe more an inherent problem with these chatbots. LLMs are great, but they aren't producing new information. You still need people for that. But if you cut down any incentive for the people to do that, the LLMs will starve.
- tptacek 3mo ago
  The business model of the site is apparently private data sales, not ad revenue.
  - jaredwiener 3mo ago
    Bots don't make purchasing decisions.
    - Terr_ 3mo ago
      Some people are already trying to make that happen, but I'm convinced it'll be a net-negative.
      - jaredwiener 3mo ago
        Rephrase: Bots SHOULDNT make purchasing decisions.
        fragmede 3mo ago
        Because... ? If I'm operating a site, and I want $X to allow a bot to scrape my site, why shouldn't the bot be allowed to make the purchasing decision to scrape the site? Obviously the bot owner would have their own set of guardrails, but if it allows sites like The numbers to stay up because bots aren't going to look at ads so the old model of displaying ads isn't going to work, I have a hard time seeing that as a bad thing.
        jaredwiener 3mo ago
        I mean it from the consumer perspective.
    - FinnKuhn 3mo ago
      According to the post inquiries for purchasing the data increased as a result though so they might not make the decision, but they do make the suggestion.

RobotToaster 3mo ago

Or just use a cache.

zackmorris 3mo ago

Ya this problem was solved 20 years ago with Varnish cache and Coral cache/CDN. I think what's really going on is that bots expose how underpowered web servers has gotten in recent years. In the 2000s, even poorly-architected PHP sites tended to serve about 200 requests per second, with 1000+ being common for static sites. I remember when Node.js came out and claimed that it could serve more like 100,000 RPS due to its cooperative threading model. But today sites have a remarkable slowness to them, running many hundreds or thousands of database queries due to ORMs and N+1 problems, so that response times can be 500 ms or more and even 1000 simultaneous users stresses servers. What went wrong is that nobody solved stuff like Russian doll caching in a general way separate from the programming language and database. We should have had ways to make dependency graphs using Etag headers as keys with real cache invalidation of dependent data. I went down that rabbit hole 10 years ago using touch events in Laravel with callbacks to handle cache invalidation when class model data was saved to the database. Also a query cache using Redis which I think might have been handled better at the database level anyway. After that experience, I can honestly say that cache invalidation is so difficult to get right that it's effectively an open problem. Meaning that programmings should use a package instead of rolling it by hand, and it should be a major concern from the start (along with sharding by user id or using something like Firebase). Don't get me started on how the web should have been a P2P content-addressable memory anyway. Nearly everything should be available from a nearby edge peer, similarly to BitTorrent. But nobody bothered to solve how to make that work with HTTPS/SSL. I suspect that has to do with early flaws in the browser security model where the whole page has to be behind HTTPS or warnings appear. So it was never clear what was personally identifiable information (PII) or merely public data being served over HTTPS. To really solve that, we probably need real trust networks and maybe even zero-knowledge proofs. Since these problems are so challenging to fix, and big companies can't be bothered to do it since they pulled the ladder up behind them, we're probably stuck with banal "are you human" challenge screens for the foreseeable future.

withinboredom 3mo ago
They are indeed quite challenging. We literally wrote a paper about it (https://arxiv.org/abs/2605.09114 https://arxiv.org/abs/2605.09114) ... you almost describe some of our original architecture too! You might find it interesting -- or not. https://getswytch.com https://getswytch.com