grafraf

4 ms·

1. 1.▲
   by grafraf 2y ago

   We are storing the result of the parsed scrape as parquet. I would advice to store the raw data as well in a different s3. The database should only have the data it needs and not act as a storage.
2. 2.▲
   by grafraf 2y ago

   We are selling price/distribution data about the products we scrape. We do run some ads and have an affiliate deals. The insight i can share is that the main (tech) goal is to make the product more user friendly and more aligned with t
3. 3.▲
   by grafraf 2y ago

   We received some harsh words in the start but everything we are doing is legally and by the book. We try to establish good relationship with the stores as the customers don't always focus on the price, but sometimes they want a specifi
4. 4.▲
   by grafraf 2y ago

   We have been blocked a couple of times during they years, usually using proxy has been enough. We try to reach out to the stores and try to establish a friendly relationship. The feelings have been mixed depending on what store we are talki
5. 5.▲
   by grafraf 2y ago

   Unfortunately no, but i can share some insights that i hope can be of value: - Tech: Everything is hosted in AWS. We are using Golang in docker containers that does the scraping. They run on ECS Fargate spots when needed using cronjob. The
6. 6.▲
   by grafraf 2y ago

   We have been doing it for the Swedish market in more than 8 years. We have a website https://www.matspar.se/ , where the customer can browse all the products of all major online stores, compare the prices and add the produc