6 ms·

What was the process for scraping 25M products ? I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work

by screye 3y ago

What was the process for scraping 25M products ? I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale. Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ? ______________ A recommendation for how to improve search. Your base captions will be pretty bad. You can use spot instances on a smaller GPU machine to run a dense captioning model (https://portal.vision.cognitive.azure.com/demo/dense-captioning https://portal.vision.cognitive.azure.com/demo/dense-caption...) and generate captions for all your images. Then for search, a simple vector store index would be a great retrieval solution here. It is better to do search using those as well. Both are pretty cheap and can be done reliably within 20-30 lines of code each in python. 3rd party tools for these are pretty stable.

DeathArrow 3y ago
>I have always used standard python tools like selenium, bs4 and the like There's nothing to scrap. You just download a JSON, the site owners kindly put on your disposal. Scraping is a more complex process, where you have to work around rate limiting and captchas. For the tool I built I wrote tens of thousands of lines of code and I still find daily issues I have to deal with if I want to scrap a particular web page, issues I don't always have the time to solve.
Ninjinka 3y ago
As someone who has scraped millions of items myself, I had success using Geziyor (https://github.com/geziyor/geziyor https://github.com/geziyor/geziyor) built in Go. Shopify sites are especially easy to scrape because they tend to share the same product data formatting and don't hide it behind JS rendering.
pencildiver 3y ago
Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json https://www.wildfox.com/products.json. Built a crawler in simple Javascript to run through a list that I bought on a site called "Built With", access their JSON file with the product listing data, and scrape the exact data we want for Agora. Then storing it in Mongo and, currently, using Mongo Atlas Search (i.e. saw they released Vector Search but haven't looked at it). It has been a process of trial and error to pick the right data fields that are required for the front-end experience but not wanting to increase the size of the data set drastically. And after initially using React, switched to NextJS to make it easier to structure URLs of each product listing page. Mongo will run me about $1,500 / month at the current CPU level. AWS all in will be about $700. I'm currently not storing the image files, so that reduces the cost as well. A few improvement that has helped so far: - Having 2 separate Search Indexes, one for the 'brand' and on for the 'product'. There's a second public JSON file that is available on all Shopify stores with relevant store data at [Base URL]/meta.json For example: https://wildfox.com/meta.json https://wildfox.com/meta.json - Removing the "tags" that are provided by store owners on Shopify. I believe these are placed for SEO reasons. These were 1 - 50 words / product so removing these reduced the data size we're dealing with. The tradeoff is that they can't be used to improve the search experience now. Hope this helps. Still wrapping my head around all of this.
- DeathArrow 3y ago
  >Mongo will run me about $1,500 / month at the current CPU level. AWS all in will be about $700. I'm currently not storing the image files, so that reduces the cost as well. It will probably cost you just $100 to rent a server from Hetzner and do the same thing. I would also use Redis or another kind of cache to hit the DB less.
- berkes 3y ago
  > site called "Built With", Do you have Alink. And are they any good?
  - stef25 3y ago
    Google ? https://builtwith.com/ https://builtwith.com/
    - Karl-Heinz 3y ago
      Berkes wanted to do good by sharing a provision with the OP, in case he/she buys something at builtwith. We all know how to Google. :)
    - berkes 3y ago
      I specifically asked the author if he could add some extra info on Builtwith. I can Google. But then I don't know if its truly the site the author was talking about. And I certainly don't know his or her insights on that site.
- dangoodmanUT 3y ago
  Yo fuck mongo just use RDS or some digitalocean DB. Or really just use opensearch/elasticsearch, or even typesense (don't bother with raft it's so broken) or meilisearch
  - jabo 3y ago
    We’ve interacted before on Twitter and GitHub, and I want to address your point about Raft in Typesense since you mention it explicitly: I can confidently say that Raft in Typesense is NOT broken. We run thousands of clusters on Typesense Cloud serving close to 2 Billion searches per month, reliably. We have airlines using us, a few national retailers with 100s of physical stores in their POS systems, logistic companies for scheduling, food delivery apps, large entertainment sites, etc - collectively these are use cases where a downtime of even an hour could cause millions of dollars in loss. And we power these reliably on Typesense Cloud, using Raft. For an n-node cluster, the Raft protocol only guarantees auto-recovery for a failure of up to (n-1)/2 nodes. Beyond that, manual intervention is needed. This is by design to prevent a split brain situation. This not a Typesense thing, but a Raft protocol thing.
helsinki 3y ago
25 million products is really not much at all to scrape.

by screye 3y ago

DeathArrow 3y ago

>I have always used standard python tools like selenium, bs4 and the like There's nothing to scrap. You just download a JSON, the site owners kindly put on your disposal. Scraping is a more complex process, where you have to work around rate limiting and captchas. For the tool I built I wrote tens of thousands of lines of code and I still find daily issues I have to deal with if I want to scrap a particular web page, issues I don't always have the time to solve.

Ninjinka 3y ago

As someone who has scraped millions of items myself, I had success using Geziyor (https://github.com/geziyor/geziyor https://github.com/geziyor/geziyor) built in Go. Shopify sites are especially easy to scrape because they tend to share the same product data formatting and don't hide it behind JS rendering.

pencildiver 3y ago

Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json https://www.wildfox.com/products.json. Built a crawler in simple Javascript to run through a list that I bought on a site called "Built With", access their JSON file with the product listing data, and scrape the exact data we want for Agora. Then storing it in Mongo and, currently, using Mongo Atlas Search (i.e. saw they released Vector Search but haven't looked at it). It has been a process of trial and error to pick the right data fields that are required for the front-end experience but not wanting to increase the size of the data set drastically. And after initially using React, switched to NextJS to make it easier to structure URLs of each product listing page. Mongo will run me about $1,500 / month at the current CPU level. AWS all in will be about $700. I'm currently not storing the image files, so that reduces the cost as well. A few improvement that has helped so far: - Having 2 separate Search Indexes, one for the 'brand' and on for the 'product'. There's a second public JSON file that is available on all Shopify stores with relevant store data at [Base URL]/meta.json For example: https://wildfox.com/meta.json https://wildfox.com/meta.json - Removing the "tags" that are provided by store owners on Shopify. I believe these are placed for SEO reasons. These were 1 - 50 words / product so removing these reduced the data size we're dealing with. The tradeoff is that they can't be used to improve the search experience now. Hope this helps. Still wrapping my head around all of this.

DeathArrow 3y ago
>Mongo will run me about $1,500 / month at the current CPU level. AWS all in will be about $700. I'm currently not storing the image files, so that reduces the cost as well. It will probably cost you just $100 to rent a server from Hetzner and do the same thing. I would also use Redis or another kind of cache to hit the DB less.
berkes 3y ago
> site called "Built With", Do you have Alink. And are they any good?
- stef25 3y ago
  Google ? https://builtwith.com/ https://builtwith.com/
  - Karl-Heinz 3y ago
    Berkes wanted to do good by sharing a provision with the OP, in case he/she buys something at builtwith. We all know how to Google. :)
  - berkes 3y ago
    I specifically asked the author if he could add some extra info on Builtwith. I can Google. But then I don't know if its truly the site the author was talking about. And I certainly don't know his or her insights on that site.
dangoodmanUT 3y ago
Yo fuck mongo just use RDS or some digitalocean DB. Or really just use opensearch/elasticsearch, or even typesense (don't bother with raft it's so broken) or meilisearch
- jabo 3y ago
  We’ve interacted before on Twitter and GitHub, and I want to address your point about Raft in Typesense since you mention it explicitly: I can confidently say that Raft in Typesense is NOT broken. We run thousands of clusters on Typesense Cloud serving close to 2 Billion searches per month, reliably. We have airlines using us, a few national retailers with 100s of physical stores in their POS systems, logistic companies for scheduling, food delivery apps, large entertainment sites, etc - collectively these are use cases where a downtime of even an hour could cause millions of dollars in loss. And we power these reliably on Typesense Cloud, using Raft. For an n-node cluster, the Raft protocol only guarantees auto-recovery for a failure of up to (n-1)/2 nodes. Beyond that, manual intervention is needed. This is by design to prevent a split brain situation. This not a Typesense thing, but a Raft protocol thing.

helsinki 3y ago

25 million products is really not much at all to scrape.