mateozaratefw

3 ms·

1. 1.▲
   by mateozaratefw 2y ago

   Tik Tok crawler fucked us up by taking a product-name (e-commerce) and inserting it into the search bar recursively with the results page. Respect the game, but not respecting the robots.txt crawl delay is awful.
2. 2.▲
   by mateozaratefw 2y ago

   :o
3. 3.▲
   by mateozaratefw 2y ago

   We do that with MercadoLibre (way smaller Amazon for latam) and the limit they pagination results to 40 pages/2000 products. Never thought of the faceted search approach. Thanks for the idea!
4. 4.▲
   by mateozaratefw 2y ago

   Thanks! sounds hard asf. I believe we can handle the inconsistence problem because we want <all> the products (so it'll be really general), however i don't think Amazon can list all their products on the webpage because of t
5. 5.▲
   How to crawl big websites with no sitemap?

   9 points by mateozaratefw 2y ago|9 comments