3 ms·

IMO one of the killer use cases of GPT is reformatting information from any format X to any other format Y, and we're using this superpower in the relatively "b

IMO one of the killer use cases of GPT is reformatting information from any format X to any other format Y, and we're using this superpower in the relatively "boring" space of data extraction: https://kadoa.com https://kadoa.com can turn any website into an API.

nicbou 4y ago
I tried this with a textual list of oublic grilling spots in my city and it got it completely wrong, hallucinating with high deceiving confidence. How do you prevent this?
- kolinko 4y ago
  Did you ask for grilling spots, or did you paste the grilling spots from some database and ask it to summarize/reformat? Parent commenter was talking about the latter. If you did the former, it will hallucinate like crazy, of course.
- hubraumhugo 4y ago
  We use LLMs to semantically understand the website and generate the scrapers code for it, not for the actual data extraction (which would be too expensive anyways). We also have checks in place to verify that the extracted data truly exists on the website.
tamiral 4y ago
love this! signing up
sergiotapia 4y ago
I'm in the market for this. How do you get around scrape blockers? For example Target and Walmart are tricky to get the markup for even with services that specialize in this like scrapfly. Do you guys have indie friendly pricing? I don't have $500/month to spend but could do $20/month.
- davidjones332 4y ago
  Lol $20 won't buy you a turkey sandwich these days
  - quickthrower2 4y ago
    2023 In an airport 2024 In a cafe 2025 In a posh supermarket 2026 In any supermarket
  - sergiotapia 4y ago
    https://scrapfly.io/pricing https://scrapfly.io/pricing https://scrapestack.com/product https://scrapestack.com/product https://www.scrapingbee.com/#pricing https://www.scrapingbee.com/#pricing There's quite a few products in this segment around that price range.
- rustdeveloper 4y ago
  For $20 you can get over 10k requests from scraping fish: https://scrapingfish.com/buy https://scrapingfish.com/buy
kolinko 4y ago
Watch out / warn users about possible mistakes. With long numbers I've seen it modify them slightly.

nicbou 4y ago

I tried this with a textual list of oublic grilling spots in my city and it got it completely wrong, hallucinating with high deceiving confidence. How do you prevent this?

kolinko 4y ago
Did you ask for grilling spots, or did you paste the grilling spots from some database and ask it to summarize/reformat? Parent commenter was talking about the latter. If you did the former, it will hallucinate like crazy, of course.
hubraumhugo 4y ago
We use LLMs to semantically understand the website and generate the scrapers code for it, not for the actual data extraction (which would be too expensive anyways). We also have checks in place to verify that the extracted data truly exists on the website.

tamiral 4y ago

love this! signing up

sergiotapia 4y ago

I'm in the market for this. How do you get around scrape blockers? For example Target and Walmart are tricky to get the markup for even with services that specialize in this like scrapfly. Do you guys have indie friendly pricing? I don't have $500/month to spend but could do $20/month.

davidjones332 4y ago
Lol $20 won't buy you a turkey sandwich these days
- quickthrower2 4y ago
  2023 In an airport 2024 In a cafe 2025 In a posh supermarket 2026 In any supermarket
- sergiotapia 4y ago
  https://scrapfly.io/pricing https://scrapfly.io/pricing https://scrapestack.com/product https://scrapestack.com/product https://www.scrapingbee.com/#pricing https://www.scrapingbee.com/#pricing There's quite a few products in this segment around that price range.
rustdeveloper 4y ago
For $20 you can get over 10k requests from scraping fish: https://scrapingfish.com/buy https://scrapingfish.com/buy

kolinko 4y ago

Watch out / warn users about possible mistakes. With long numbers I've seen it modify them slightly.