3 ms·
depending on what exactly "scraper tech" (lol) is, i suspect you may need a different, less opinionated tool to do the work you need to do. that said, i bet if
by 0gs 6mo ago
depending on what exactly "scraper tech" (lol) is, i suspect you may need a different, less opinionated tool to do the work you need to do. that said, i bet if you paid for enterprise, these problems would magically disappear? ;)
- decide1000 6mo agoWith scraper tech I mean a rust binary that is able to download and process thousands concurrent urls (millions per hour). Not to the same domain obviously. Paying more is not the issue here, its more the idea that an AI decides on what part of the spectrum I operate. Why is it opinionated? I am not doing anything wrong, why does it make me feel like I have to defend myself.
- mtndew4brkfst 6mo agoWhat is the specific concrete purpose of downloading millions of URLs per hour across different domains if it's "not doing anything wrong"?
- big-and-small 6mo agoMight be it for scrapping content for training an LLM? Oh no only big tech allowed to do it...
- mtndew4brkfst 6mo ago"The gangsters do it and get away with it so any random person should get to as well"? Not a particularly defensible position if that's an accurate paraphrase.
- decide1000 6mo agoMostly ecommerce and pricing data. I work for marketplaces, brands, retail stores and even our own saas competitors. We match the EAN (gtin) to the correct SKU within seconds (Google Shopping, Amazon, etc). Part of it is our own trained ML models.