5 ms·

Turn entire websites into LLM-ready data

• jshchnz 2y ago
  This is cool, seems like a nice way to easily add context for ChatGPT at the least
  • cpeffer 2y ago
    * Creator here - Thats the goal!
    • donohoe 2y ago
      And you honor or ignore robots.txt?
      • cpeffer 2y ago
        It wasn't in our initial version (we didn't plan on launching today), but we are pushing an update to do so now.
• altdataseller 2y ago
  What user agent does it use and do you obey robots.txt? * apparently not as I tested it with LinkedIn.com which blocks most non-google/bing bots and it crawls it OK
  • super256 2y ago
    > FireCrawl is built to navigate common web scraping challenges, including reverse proxies, rate limits, and caching They probably ignore robots.txt
• thawab 2y ago
  [dead]
• deleted 2y ago
  [deleted]
• deleted 2y ago
  [deleted]
• nyolfen 2y ago
  https://gist.github.com/yoki/fd2a3642b059703529d158de8233bbc9 https://gist.github.com/yoki/fd2a3642b059703529d158de8233bbc...
  • cpeffer 2y ago
    It crawls webpages (finds subdirectories), handles JS blocking with fallbacks to headless browsers, and does this all concurrently. If only that script worked for every website. But, alas, it does not.
  • deleted 2y ago
    [deleted]
• pryelluw 2y ago
  Does this respect any anti AI related scraping rules set forth by website owners?
  • duskwuff 2y ago
    It doesn't even respect general robots.txt exclusions, so... I doubt it.
• padolsey 2y ago
  Do you have pay-as-you-go pricing? I feel that’s always missing from things like this. Cool otherwise.