12 ms·
Show HN: I scraped 25M Shopify products to build a search engine
Hi HN! I built Agora as a side-project leading up to the holiday season. I wanted to find an easier way to find Christmas gifts, without needing to go store-by-store.
My wife asked me for a a pair of red shoes for Christmas. I quickly typed it into Google and found a combination of ads from large retailers and links to a 1948 movie called 'Red Shoes'. I decided to build Agora to solve my own problem (and stay happily married). The product is a search engine that automatically crawls thousands of Shopify stores and makes them easily accessible with a search interface. There's a few additional features to enhance the buying experience including saving products, filters, reviews, and popular products.
I've started with exclusively Shopify stores and plan to expand the crawler to other e-commerce platforms like BigCommerce, WooCommerce, Wix, etc. The technical challenge I've found is keeping the search speed and performance strong as the data set becomes larger. There's about 25 million products on Agora right now. I'll ramp this up carefully to make sure we don't compromise the search speed and user experience.
I'd love any feedback!
- virtuosarmo 3y agoI believe Shopify built their own app / website where you can search for products exclusively from Shopify merchants. https://shop.app/ https://shop.app/
- connectingu 3y agoIncredible. Where can I connect with you? Want to pick your brain & swap some thoughts :)
- Canada 3y agoWorked well for me, great job. I searched for something I've been looking for and found some interesting options I haven't seen before.
- RagnarD 3y agoIs this really within the TOS of Shopify?
- jesterson 3y agoDoes it matter? Or you can't do anything not explicitly allowed by law? Shopify is the company spotted in so many shenanigans, so anything that undermines it's business I would personally welcome very well.
- bomewish 3y agoWhy not manticore as backend? Much better perf than ES, less memory intense, sql syntax. Just fantastic all round!!
- TekMol 3y agoThe Terms page goes to "Jaggi Enterprises", "A Modern Investment Fund. We buy, build, and invest in software companies with recurring revenue.". So maybe this is not really something a guy built for his wife, but some anonymous startup that googled "Which terms rank best on Hacker News" and then wrote the "I did ... my wife .." story?
- ltbarcly3 3y agoJaggi is a fake it until you make it fake portfolio. Most of the companies it runs are just lorem ipsum fake sites. I think it is likely true that this is a solo dev.
- SwedishExpat 3y agoYeah you're right: https://www.linkedin.com/company/jaggi/ https://www.linkedin.com/company/jaggi/
- TekMol 3y agoYou mean the site is not owned by Jaggi? Then why would the terms and privacy links go to Jaggi?
- pencildiver 3y agoOP here. Yup, I am in the process of starting a holding company LLC for my software products and small investments. Just went ahead and deleted 2 from the Investments page that are not launched yet but still in-development (just had landing pages up for those). Wasn't planning on releasing the Jaggi site yet, as I'm still wrapping my head around the holding company structure / it's new to me. Agora has been a side project of mine. TBH in retrospect, I wish I would have given this post more thought as the servers / search performance wasn't prepared for any significant traffic. So definitely didn't game HN.
- deleted 3y ago[deleted]
- codetrotter 3y agoOn the page where you show details about the product, I would like to have it include the same product from other Shopify stores by doing an image similarly comparison. And then highlight how the price compares. For example, here are some pretty crazy red shoes. But they are too expensive for me. Would be interesting to see if this is the only store selling these shoes, or if someone else has the same shoes much cheaper. https://www.searchagora.com/products/vasco-4-47fb0f87-5b89-470c-b775-6da4da5d75e5 https://www.searchagora.com/products/vasco-4-47fb0f87-5b89-4...
- jillesvangurp 3y agoThere are a few conferences dedicated to ecommerce search. Mices is pretty good. I did not go there this year but I know some of the people behind it. Good community and lots of stuff happening. Two points here. - 25 million is really not a lot for most search engines. Something like Elasticsearch can easily deal with that if you deal with it properly. And there are plenty of equally capable solutions. I have worked with logging clusters that processes log entries by those numbers on a daily basis. A modestly sized cluster goes a long way for that. Bare metal is cheaper than cloud for this. But a couple of simple servers with decent CPUs and memory and SSDs should go a long way here. Start worrying once you hit a few hundred GB of storage used. Anything below that is easy to deal with. - The key challenge with this volume is not performance but search quality. Building a competitive search engine is hard. You might have thousands of potential matches out of millions for any given query and your job is to pick the best 3, 5, 10 (whatever fits on your screen) ones. This is hard. So, what makes for a good answer is the key question to answer. All the naive solutions for this problem put you at the bottom of the market in terms of competitiveness. If you can't do better, you are just another low quality search engine not quite solving the problem. The bar is high these days for a good search engine and most of the better ecommerce companies have highly skilled search teams working on this.
- ganesha727 3y agoGg
- ganesha727 3y agoIdea! Shopify has a ton of resellers that sell junk from China. If you figure out how to avoid them, your life would be 10x easier.
- berkes 3y agoI was wondering on a tech solution for this lately. What I now do when I shop, is that I compare the images (and descriptions) of listings on Amazon, bol.com or Shopify with listings on Ali and Temu. If they're exactly the same, run. If they differ slightly, look closer (and most likely run). I guess automating this could make for a solution to at least detect cheap resellers.
- nox100 3y agoI have no clue how to implement a search but maybe some words are more important than others. I searched for "mens dress shirts button long sleeve" and after about 6 results it was all women's clothing.
- sanketgoyal11 3y agoHow did you find the list of shopify stores and names?
- pencildiver 3y agoFound a list on a site called "Built With". Mentioned this in another comment but I think it's meant for building sales outreach email lists. Once you have the store URL, you can get general store information at [Base URL]/meta.json Here's an example: https://wildfox.com/meta.json https://wildfox.com/meta.json
- dangoodmanUT 3y agoSuper cool!!!
- pencildiver 3y agoThanks! Laughed audibly at your comment in the other thread about Mongo haha Looking into all options right now. A tradeoff between current stability, price, and performance.
- jross225 3y agoheh, I used to work on the data team at Shopify. I built something similar to search internal dbs for secret santa gifts based on some weird criteria. Scraping might have a large margin of error because a lot of products tend to be ephemeral. Neat project though!
- pencildiver 3y agoAgreed on the large margin of error. Working on a bot to store and convert the images to webp to improve performance. Having the bot do a check for any images that don't exist and removing those listing. Will likely also need to triangulate this with a 404 check. Recently added an option for users to mark a product as "sold out" on the search results which will help as well. Unrelated, but what was the "weird criteria" for the secret santa exchange? Half joking but also helps with figuring out filters :)
- alvarome 3y agoI'm a Shopify store owner myself. I saw there is a $99 per month to get your product verified, how would this compete in terms of CPC with a traditional channel such as google ads or meta ads?
- minastirith 3y agoNot the OP but counting on an average of $3 dollars per click in the US across those channels, I'd say this pricing is way more effective with the amount of searches the site is getting.
- minastirith 3y agoLove it! Some improvements are needed on search but is an amazing MVP, I'll use this for my late christmas shopping
- connectingu 3y agoIncredible. Would love to connect with you. Where can I find you LOL
- kacyjames 3y agoAny Unicode input (Japanese or Greek text for example) currently causes a 500 error.
- qdequelen 3y agoHey, I'm the CEO of Meilisearch. If your issue is performance, I would love to give you a try with Meilisearch. You'll be able to create an "as you type" experience with our engine that responds in less than 50ms!
- ctocoder 3y agohow did you get a list of the 25 million stores to crawl?
- Wajid2502 3y agoGreat idea
- TessMorghan 3y ago[dead]
- difradev 3y agoAmazing job! I've one question: how did you find the price of every products? I mean, every product page has a different id or class that identify a price. Do you use a regex?
- pencildiver 3y agoThanks! Actually a lot easier than you'd expect. Not touching anything on the front-end of the Shopify store. Every Shopify store has a public JSON file at [Base URL]/products.json with 'price' as a field. Example here: https://wildfox.com/products.json https://wildfox.com/products.json One thing I messed up on originally was not pulling the 'currency' field which is actually in a different public JSON file called 'meta.json'. Example here: https://wildfox.com/meta.json https://wildfox.com/meta.json Separately, this was primary reason to only start with US stores: to make sure the currency shows up correctly and to purposely limit the initial audience to keep loading times reasonable. Working on adding all Shopify stores in the world now (a list of about 5 million active stores from what I have found).
- difradev 3y agoClear! Thanks!! :)
- pencildiver 3y agoHN— Not sure if anyone will see this but I wanted to thank you all for the support. Although I haven't slept much since going live, it has been amazing getting early feedback from the community. Agora is still in MVP stage but getting better by the day. Just pushed a big update: fixing an image shifting bug, a blur effect on loading, Redis for caching, brand pages, architecture fixes, and several other things. Currently working on improving the relevancy algorithm, adding all ~5 million Shopify stores, and then adding WooCommerce stores over the next few days. If you have any suggestions or ideas, reach out to me at support @ searchagora .com :)
- IcyHordr 3y agoSo cool, good luck in the marriage, you made a very cool thing!!
- pencildiver 3y agoThank you! Tied the knot back in May, so both marriage and Agora are new to me. Open to advice on either.
- joshdance 3y agoAmazing. Why doesn't Shopify built this natively?
- misterbwong 3y agoWhat technology did you use to build the scraper and how did you get around the usual challenges (anti bot, ip banning, etc) with scraping large amounts of data?
- cldellow 3y ago(not the OP, but I have some experience with Shopify) Shopify stores publish their product catalog at /products.json. From personal experience, you can hammer it pretty hard without being rate limited. A challenge is that the pricing info in that endpoint is based on the stock Shopify catalog fields, and can be misleading depending on the specific theme customizations that the merchant uses.
- pencildiver 3y agoScraper is built in Javascript and a Mongo database. Probably not the most scalable way to do it, but I found that all Shopify stores have a public JSON file available at [Base URL]/products.json. So found a list of stores, built a crawler to go store-by-store, and standardized the data on my end. Here's an example: https://www.wildfox.com/products.json https://www.wildfox.com/products.json
- bomewish 3y agoWhat’s the trade off using js for this? Would it have been much faster to use go or something?
- qdequelen 3y agoDid you only get the schema.json?
- thomasfromcdnjs 3y agoooo that is a hot tip!
- deleted 3y ago[deleted]
- 3y ago
- glohbalrob 3y agowow! Nice work. I've been trying to build an index of shopify stores. Did you search for all domains pointing to shopify's name servers?
- selcuka 3y agoI don't think that would work as many people also use Cloudflare etc. You may try using BuiltWith which is a paid service: https://trends.builtwith.com/websitelist/Shopify https://trends.builtwith.com/websitelist/Shopify
- hipadev23 3y agohttps://www.searchagora.com/search?query=red%20shoes https://www.searchagora.com/search?query=red%20shoes It doesn’t work very well.
- owlninja 3y agoI see some Red Wing shoe products first, then some bearings, some shoes, then someone in lingerie with red heels, and then more red shoes. I know Google is the horse to beat right now but they showed me nothing but red shoes to buy plus a few ancillary results for such a generic search - and if I was a good husband I would add a few words to narrow my search.
- cmcconomy 3y agoThat's funny, I made a domain-specific version of this for canadian coffee deals. https://beangrid.mcconomy.org/ https://beangrid.mcconomy.org/
- pencildiver 3y agoSuper cool project (especially as a coffee lover myself)!
- boringg 3y agoWhich coffee seems to hit the best in Canada (your take). I find the espresso in Canada hasn't been as good as the coffee brands in the US but I'm open to possibilities. Also like the project!
- wcarss 3y agopersonal somewhat-pedestrian list: Pilot, Detour, Reunion, Propeller, Phil and Sebastian
- cmcconomy 3y agoI enjoy Café St-Henri's Godshot when I can get it at a discount. Anything from Stereo is great, and I've enjoyed Monogram and De Mello. If I was buying at regular price, I would often get Social Coffee. Of course I put this together because Black Friday is when I load up on (relatively) cheap coffee and chuck it in the freezer, so this time of year I always branch out and try new places and new offerings from familiar places. I built this list mostly from a reddit compilation I found, and I've been slowly updating the source url list as I learn of new canadian roasters that happen to be Shopify customers.
- konschubert 3y agoHey, I have a Shopify store that sells e-paper calendars / smart screens. I tried to search for it but I could not find it. What should I do so your crawler can find me? https://shop.invisible-computers.com https://shop.invisible-computers.com
- pencildiver 3y agoSuper cool product! I'm currently using a list of Shopify stores, so it's still limited (i.e. wanted to start with a relatively small list to focus on the search experience). I'll submit your URL to the crawler now. If you want to reach out to support @ searchagora.com , I'd love to get your feedback as a Shopify store owner.
- shubham_sinha 3y agoHi, you could drop an email to onboard@peppyhop.com and we will be happy to onboard you. Please add target geography like you would like to target Indian market or US market
- crakhamster01 3y agoalways be closing lol
- pencildiver 3y agoYou’re live on Agora: https://www.searchagora.com/products/invisible-calendar-62669ace-a96c-477b-8136-bc925438b783 https://www.searchagora.com/products/invisible-calendar-6266... Thinking that we should have a page where store owners can submit their URL to be crawled.
- konschubert 3y agoCool, thanks!
- muratsu 3y agoAgora also doesn't return red shoes for the search query "red shoes". Seems like you haven't fully solved the problem yet :) From a technical perspective, crawling 25M products is impressive but the search itself doesn't provide much value to me. I already use large e-commerce sites (amazon, wallmart, ...) and targeted ones (Nordstrom, SSENSE, ...). Sure I may not be searching through all the shopify, wix stores but I need to know why that's valuable to me to begin with. Perhaps understanding the value prop of SMBs and educating me about it would be a better positioning for Agora than simply being a search engine.
- pencildiver 3y agoDefinitely have not solved the problem yet! The search algorithm prioritizes the brand called "Red Wing Shoe" so still figuring out ways to show real 'red shoes'. Have been thinking about passing the images through a detection tool and tag them to enhance the search experience. Re: Value Proposition. Absolutely, I think focusing on the SMB-angle and 'local shopping' will help direct users better. I'll definitely take this into account.
- charliejuggler 3y agoYou could try the method we used for our vector search demo for e-commerce (all open source, natch) - use CLIP to get vector embeddings for product pictures and then use these for boosting or matching. https://opensourceconnections.com/blog/2023/03/22/building-vector-search-in-chorus-a-technical-deep-dive/ https://opensourceconnections.com/blog/2023/03/22/building-v... Our demo works pretty well for searches like 'blue network cable' when the colour isn't always explicitly mentioned in the product data.
- paulddraper 3y agoBest of luck on your marriage
- deleted 3y ago[deleted]
- b2bsaas00 3y agoBasically it’s Amazon
- quickthrower2 3y agoAntizon
- taimurayaz 3y agoCool project but Shopify already has this. https://shop.app https://shop.app https://shop.app/search/results?query=red%20shoes https://shop.app/search/results?query=red%20shoes
- dinkleberg 3y agoWas about to share the same link, it seems like competing against Shopify would prove quite the challenge. The real way to differentiate IMO is with a targeted UX for different niches rather than the one search engine to satisfy all queries.
- gnabgib 3y agoBut seems to have filters (lots of liquor stores use Shopify) - shop.app shows only candy and swag[0], while searchagora shows ~130k results for the actual product [1] [0]: https://shop.app/search/results?query=Baileys+Irish+Cream+Liqueur https://shop.app/search/results?query=Baileys+Irish+Cream+Li... [1]: https://www.searchagora.com/search?query=Baileys%20Irish%20Cream%20Liqueur https://www.searchagora.com/search?query=Baileys%20Irish%20C...
- pencildiver 3y agoYeah absolutely. I hadn't heard of Shop until today but the value proposition is definitely similar. In the next week, I'll add other e-commerce platforms like BigCommerce, WooCommerce, support for custom built sites, etc. to really differentiate the user experience.
- gardenhedge 3y agoOh cool. That works a lot better than OPs
- treesciencebot 3y agoThis is amazing for finding cute collectibles from my favorite TV show that I would otherwise not noticed among random t-shirt and other "slap the picture and call it co-branded" products! I'm not super sure how long it is going to be around, but I think I'm gonna keep playing with it for a while.
- pencildiver 3y agoReally happy to hear this. I'll do my best to keep this around :)
- Asparagirl 3y agoCool! But how did you get the initial dataset of 643,000+ Shopify stores (data as per your “About” page) in the first place, to then scrape the products from their /products.json feeds? Or did you just try a huge list of domain names at random?
- xnx 3y agohttps://www.shopify.com/robots.txt https://www.shopify.com/robots.txt lists a lot of sitemap files, which tend to be a good starting point.
- KomoD 3y agoLooks like it's just Shopify's own pages and not anything related to actual stores.
- prayze 3y agoDid this suddenly get changed? Nothing but "# ,: # ,' | # / : # --' / # \/ />/ # /" is shown now.
- xnx 3y agoWeird. I think it did change. Google cache shows a 2229 line file: https://webcache.googleusercontent.com/search?q=cache%3Ahttps%3A%2F%2Fwww.shopify.com%2Frobots.txt https://webcache.googleusercontent.com/search?q=cache%3Ahttp...
- capableweb 3y agoSeems it might be looking at the referrer. Loading https://www.shopify.com/robots.txt https://www.shopify.com/robots.txt from clicking the link shows the weird line while opening it in a private browser window shows the right one.
- deleted 3y ago[deleted]
- Uptrenda 3y ago[flagged]
- rsweeney21 3y agoGoogle is a data scraper. OpenAI is a data scraper. Pretty much every major company uses data obtained from a data scraper.
- jokethrowaway 3y agoWhy? They're cataloguing public data. Do you think the same of search engines - which are equally scraping the web? If it's not useful to you, nobody is forcing you to use this product.
- Uptrenda 3y agoYou really didn't think about this. Search engine crawlers are specifically designed to ignore sites that have the right setups. In other words: they try to be good citizens of the web. Data scrapers on the other hand are known to steal others IP and use it to jump start their own services. When you do that you're using all the bandwidth of the site while devaluing everything they've built. They will have invested significant time and resources building that IP. But data scrapers think that just because its on a public website it means they can leech it and do what they like. No, there is such a thing as copy right and respecting authors. Fuck anyone who says otherwise. If you want to be a little kiddiot and steal content you're free to do so. But you're just undermining the work of everyone you steal from. Making it less and less viable for them to make more of it in the future. I stand by what I said. Data miners are fucking parasites and the web would be better off without them.
- twelve40 3y agohm in this case, he is driving business to these websites. Quite the opposite from OpenAI and scrapers you are talking about, no?
- 3y ago
- xnx 3y agoGreat project. If you continue to crawl the data, be sure to save it so you can detect price changes a la camelcamelcamel.
- pencildiver 3y agoGreat call! I am doing back-ups on Mongo and this is a good use-case for tracking changes. Also trying to figure out how to detect is a product is sold out or not being sold anymore.
- Minor49er 3y agoI worked on a competitor to CamelCamelCamel years ago. We had this exact issue since people would often click through to a page where the price or availability were different from what we were showing Ultimately, we ended up adding an interstitial page between the product listing on our site and the page on the seller's site This interstitial checked to see if we checked the price in the last couple of minutes, and if not, it would run a quick scrape of the page to ensure that we had the most up to date information I can't remember exactly what the messaging or behavior was when there was a difference. I think there was a message that was displayed if the prices were different. Or if the product was actually out of stock, it would pull the user back into our site with a toast explaining that the product was no longer available Anything less aggressive than this resulted in more customers experiencing price/availability errors or simply leaving the site, and anything more aggressive resulted in angry site owners who were losing bandwidth to our bots > Also trying to figure out how to detect is a product is sold out or not being sold anymore In these cases, either the page will say as much (eg: "Product Unavailable"), have some kind of stock or status code hidden beneath the UI to show that it's not available, or the target page will simply vanish from the web. However, none of these are guarantees. A site could say that a product has been discontinued, but the item could come back later, or under a different SKU, or whatever else
- secabeen 3y agoFor all of Amazon's faults, the fact that they tolerate CCC does drive a lot of my online purchases there. CCC used to track other sites, and was eventually blocked on all of them. If more sites want my business, showing their pricing history (either from internal data, or by letting someone build the DB) would go a long way.
- joshuamcginnis 3y agoI love your approach; you found a problem and developed a solution for it. And then you got the courage to share with the larger technical community. Good on you. There's obviously some rough edges (multiple duplicate products, issues with product links linking to empty pages, and no results for broad terms), but don't let that stop you. I'm certain they can all be fixed. Keep going! At the least, you'll come out of this with an excellent project in your portfolio.
- pencildiver 3y agoThank you, that means a lot. It has definitely been a whirlwind of emotions since posting on HN but glad I did. It's definitely an MVP so going to work fast to improve it.
- thih9 3y agoWhen I search for “op-1”, partial match like “Frontier Co-op Turkey Rub, Organic 1 lb. -- Frontier Co-op” gets ranked higher than “teenage engineering op-1”. I would expect the opposite.
- mandeepj 3y agoDo you plan to add filters: price etc? I was about to 'reviews' as well in the above list but decided not to as they are not always trustworthy. Now AI is so advanced, that it can be used to detect fake reviews and ignore them from sampling.
- pencildiver 3y agoYes. There's a very basic price range filter right now. Working on adding a ships to, location, and a few others. Open to any ideas that would help in the shopping experience. There are 'reviews' now and made the decision to only let authenticated users leave users so they are more trustworthy (i.e. thinking is that adding more friction will lead to higher quality reviews).
- quickthrower2 3y agoI like it. I need to be able to filter search to if it will deliver to my country. It desperately needs some indication that your action is being processed, like a spinner, when you search.
- pencildiver 3y agoAbsolutely. Working on a "ships to" filter and enhancing the 'price' filter. Also fixing the loading experience as we speak. Wasn't expecting this level of traffic so didn't account for slow server speed with the front-end experience.
- stef25 3y agoOr even "ships from" / "located in". I'm in Europe and don't want to deal with custom hassles or delays from shipping. Etsy and Reverb both have this option which I never fail to use.
- quickthrower2 3y agoThanks! It is reasonably fast but slow enough that a cue is needed so I know the input triggered.
- mvdtnz 3y ago[flagged]
- finneganscat 3y ago[dead]
- 8372049 3y agoOP said in another comment: > ... narrowed down the focus to stores to US only and between $100k - $1m in revenue to keep the initial data set manageable (and the CPU / Storage costs reasonable).
- system2 3y agohttps://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html I wouldn't be happy to hear someone calling something I worked on "dogshit" to be honest. I learned from his work today and appreciate his approach. It doesn't hurt to be kind.
- jonnycoder 3y agoAwesome! It would be good to listen to the enter key when typing in a search query. Your privacy and terms links point to what appears to be the saas code framework you used (just a guess). I was looking for your contact/email so I can ask you some questions.
- pencildiver 3y agoEnter key should work, but the loading speed is very slow right now. Fixing that right now :) You can reach out at support @ searchagora.com. Would love to talk!
- bsbechtel 3y agoI searched for 'pão francês' and my store was the #1 result. I think you're doing it right! :)
- dns_snek 3y agoI'm sorry, but I have to question where this heartfelt story about looking out for your wife is in any way real? The website certainly doesn't look like a side project, it has a fully fledged system for merchants to advertise on Agora for a fee, an affiliate system offering $50 commissions to onboard merchants and the ToS and Privacy policy link to a website with the following mission statement: > We buy, build, and invest in software companies with recurring revenue and product-led growth.
- pencildiver 3y agoOP here. Yup, haven't launched the holding company yet but idea is to have an LLC for all my software projects. Still a work-in-progress in both thinking and execution. Agora specifically came from a personal need and is obviously still an MVP project. Spun up a Merchant Page and Affiliate Program page in a few hours on Webflow using a template. There is a merchant dashboard built but the 'affiliate program' is a test.
- ruune 3y agoHow is "Jaggi Enterprises LCC" involved, where for example the terms lead to?
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- screye 3y agoWhat was the process for scraping 25M products ? I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale. Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ? ______________ A recommendation for how to improve search. Your base captions will be pretty bad. You can use spot instances on a smaller GPU machine to run a dense captioning model (https://portal.vision.cognitive.azure.com/demo/dense-captioning https://portal.vision.cognitive.azure.com/demo/dense-caption...) and generate captions for all your images. Then for search, a simple vector store index would be a great retrieval solution here. It is better to do search using those as well. Both are pretty cheap and can be done reliably within 20-30 lines of code each in python. 3rd party tools for these are pretty stable.
- DeathArrow 3y ago>I have always used standard python tools like selenium, bs4 and the like There's nothing to scrap. You just download a JSON, the site owners kindly put on your disposal. Scraping is a more complex process, where you have to work around rate limiting and captchas. For the tool I built I wrote tens of thousands of lines of code and I still find daily issues I have to deal with if I want to scrap a particular web page, issues I don't always have the time to solve.
- Ninjinka 3y agoAs someone who has scraped millions of items myself, I had success using Geziyor (https://github.com/geziyor/geziyor https://github.com/geziyor/geziyor) built in Go. Shopify sites are especially easy to scrape because they tend to share the same product data formatting and don't hide it behind JS rendering.
- pencildiver 3y agoGreat suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json https://www.wildfox.com/products.json. Built a crawler in simple Javascript to run through a list that I bought on a site called "Built With", access their JSON file with the product listing data, and scrape the exact data we want for Agora. Then storing it in Mongo and, currently, using Mongo Atlas Search (i.e. saw they released Vector Search but haven't looked at it). It has been a process of trial and error to pick the right data fields that are required for the front-end experience but not wanting to increase the size of the data set drastically. And after initially using React, switched to NextJS to make it easier to structure URLs of each product listing page. Mongo will run me about $1,500 / month at the current CPU level. AWS all in will be about $700. I'm currently not storing the image files, so that reduces the cost as well. A few improvement that has helped so far: - Having 2 separate Search Indexes, one for the 'brand' and on for the 'product'. There's a second public JSON file that is available on all Shopify stores with relevant store data at [Base URL]/meta.json For example: https://wildfox.com/meta.json https://wildfox.com/meta.json - Removing the "tags" that are provided by store owners on Shopify. I believe these are placed for SEO reasons. These were 1 - 50 words / product so removing these reduced the data size we're dealing with. The tradeoff is that they can't be used to improve the search experience now. Hope this helps. Still wrapping my head around all of this.
- Redster 3y agoHave Swedish family. Searched dala because family wants traditional Christmas ornaments. Sure enough, there were several results that were 10x cheaper than what I could find on the first page of Big Search Company. Great job!
- pencildiver 3y agoAmazing, glad you were able to find it. I also just learned about what a "Dala" is :)
- quaxar 3y agoGreat site. Having built a search engine that needed to handle product data on a similar scale, it's not an easy thing to manage. Some observations: - Don't use infinite scrolling, it's an outdated UI practice that leads to bad user experience. It also makes the footer entirely unviewable. - Clicking on a product card image does not reliably open up the product. I have to randomly click on it a few times (Chrome, Brave) - Clicking on product card image and title leads to different actions, this is a bit unexpected, should show some hint of the difference. - The product page pop up will reset the search list when closed, this messes up my search navigation, breaks the flow of browsing.
- moneywoes 3y agohow did you avoid ip based blocking? rotating proxies?
- moneywoes 3y agowhere did you find a list of shopify stores to scrape
- 8372049 3y ago> Bought an initial list of 2m stores for a few hundred dollars from a website called "Built With". Think they are used for building sales outreach lists. Then narrowed down the focus to stores to US only and between $100k - $1m in revenue to keep the initial data set manageable (and the CPU / Storage costs reasonable).
- ttt3ts 3y agoBuilt the same thing a while back while collecting a lead list for sales. Not bothered to keep data updated but was a fun thing to build in a couple days. (disclaimer mobile experience is meh cause it was a fun project) https://zensear.ch https://zensear.ch How did you find list of all Shopify stores? I ended up just checking every .com, .net, etc as I didn't find an easy way to figure it out directly from shopify.
- 8372049 3y agoIn another comment, OP writes: > Bought an initial list of 2m stores for a few hundred dollars from a website called "Built With". Think they are used for building sales outreach lists. Then narrowed down the focus to stores to US only and between $100k - $1m in revenue to keep the initial data set manageable (and the CPU / Storage costs reasonable).
- deleted 3y ago[deleted]
- slimebot80 3y agoNice. But can I ask what motivated you? I don't see any affiliate details in the links - do you monetise at all?
- ttt3ts 3y agoWanted to play with typesense. The tech behind building something like this isn't hard. Marketing and traffic is. No point in monetizing with no users.
- alvarome 3y agoIt looks like from the "become a merchant" you can directly boost your products which I assume will prioritize them and hence drive more traffic to your Shopify store
- noduerme 3y agoThis is great - just a couple UI things bugging me. 1. When clicking "Open" on a product, the user should be able to open that in a separate tab. Currently that's not possible; I'm sure because it's being delivered in a single page (can't check now because you're getting hugged to death by HN). 2. When the server's slow, as it just was, there should be some kind of waiter / loader to immediately show the user that the "Open" click was sent on a product. Otherwise people will keep clicking it (or worse, clicking other products) and there's no indication that it's loading. 3. Once a product is open, it's not clear how to get back from it. I see the "X" in the corner, but doing that seems to take me back to a blank search page, not to my search results. The back button also doesn't take me back to the search results...
- pencildiver 3y agoThanks for sharing this! Definitely wasn't expecting this level of traffic so didn't account for some front-end loading experiences. Implementing these now. For 3, thinking to let the back button work the same as the "x", that way a user can return to where they are in a search result regardless of what they click on.
- asdadsdad 3y agocool project. You might have notice, but there's a non-trivial amount of fraud on shopify (fake shops, info stealers, etc). Might be interesting to look at that dataset and explore a bit =) it's quite fascinating
- smcin 3y agoWhat are the telltales, for spotting those? And how aggressively does Shopify verify/police them?
- pencildiver 3y agoI've definitely noticed that already. Any advice on how to spot that? Another challenge is that there are products sold on the original site and third-party marketplaces, both of which could sell on Shopify. So need to find a way to automatically detect the type of store.
- asdadsdad 3y agoyou might check this for inspiration (https://seguranca-informatica.pt/shopping-trap-the-online-stores-scam-that-hits-users-worldwide/#.Y0ywfezMJmr https://seguranca-informatica.pt/shopping-trap-the-online-st...). I used to have a huge IOC collection, but now stopped tracking them. things like HTML markup, pricing patterns, IPs might inform on specific clusters of fraudsters. I don't think Shopify cares tbh
- pencildiver 3y agoSuper helpful, thank you.
- twothamendment 3y agoSearching is slow (kinda expected that right now), but after clicking a product and then hitting back, I have to wait for the search again. Not at computer so I didn't check the headers, but maybe allow the client to cache the response for a short time so it doesn't need to load search results again.
- pencildiver 3y agoJust upgraded the storage and put in a few fixes so it's working a bit faster now. Working on caching some responses locally as we speak. Great idea.
- system2 3y agoHow are you planning to monetize this? You mentioned you are spending around $2K just to run it. Is there a commission strategy or ads? Or populate with your products at one point so you sell your own thing?
- alvarome 3y agoIt looks like from the "become a merchant" you can directly boost your products which I assume will prioritize them and hence drive more traffic to your Shopify store
- shadowbanned4 3y agoThis isn't worth the cost or effort. Shopify already has an internal tool with this functionality that they are planning to publicize.
- 6510 3y agoThere is no need to limit it to that, most shops have some kind of product feed.
- krauses 3y agoWhat's your revenue model? I see you expanded on the details of your $1.5K monyhly cost, but failing to see how you make money? Affiliates fees?
- pencildiver 3y agoRight now, charging Shopify store owners $99 / product / month to give them a 'verified' tag and boost their product in search results. Currently not making money on affiliate fees. I wanted to first prove that people would actually use this / find value in it. Fortunately a few merchants have reached out already via email to talk through the business model so this will likely evolve as we learn more.
- jaipilot747 3y agoWhat are you verifying?
- bluepnume 3y agoAmazing! Does it have an api?
- pencildiver 3y agoNot yet! Definitely something to consider as I upgrade the architecture. Would it be helpful to have API access to all products on Agora for your own app?
- EvanAnderson 3y agoAside: The ending of the 1948 "The Red Shoes" was funny to me, but I think I was a little loopy after slogging thru it. I don't know if I recommend it or not.
- pencildiver 3y agoI definitely need to watch it now.
- pitched 3y agoShopify has tried a few times to build a tool like this but hasn’t ever managed to get any traction. I think that missing any curation at all could be what eventually kills it. Their current attempt is https://shop.app https://shop.app and a query for red shoes is mostly red shoes.
- hackideiomat 3y ago> a query for red shoes is mostly red shoes well I get mostly black shoes lol Edit: ah no, they just use half a page for shoe shops first with black shoes as logo??
- dangoodmanUT 3y agoads baby
- senecaso 3y agoYa, curation is sadly required in the Shopify ecosystem. There are millions of shops, there is a tonne of garbage. Its also difficult (but not impossible) to properly classify items so that you can better target results for a given query. One of the first problems that anyone attempting this will run into is the amount of mature content available on Shopify shops. Innocent queries turn up many NSFW images that may offend some users, so you have to be able to get on top of that one pretty quick. I remember in once case, I found what appeared to be an escort service listing "models" on Shopify. It was super creepy. I needed to get in front of that one pretty quick as well, as it was turning up in results.
- rocauc 3y agoReally neat. I tried your search for red shoes, and I found some, er, unexpected imagery on page 1. One thing you could do is add semantic search so when a user searches "red shoes," the index returns images that look like red shoes even if the metadata doesn't say anything about color or item types. To do this, I'd use a model like CLIP. Here's an example of using CLIP and Supabase to do semantic image search: https://blog.roboflow.com/how-to-use-semantic-search-supabase-openai-clip/ https://blog.roboflow.com/how-to-use-semantic-search-supabas...
- pencildiver 3y agoAwesome, thanks for the suggestion / link! Actually left another comment about potentially doing semantic image search to improve results so wrapping my head around it now.
- senecaso 3y agoI hope you have better luck than I did! A few years ago, my partner and I built vendazzo.com (now defunct). It was an e-commerce search engine on products listed on Shopify shops (sound familiar? :)). At the time, we had > 100m products listed, and I don't remember how many shops we were indexing.. over 100k I think, but we had access to over a million. Overall, I think your approach is very similar to ours, but we managed to keep our costs lower. At the time, we were spending ~$550/mo, and our search times were under 300ms. We had established partnerships with a number of shops, and we had a few users, but not nearly enough. That's where the wheels came off. The site operated for over a year, but the monthly costs wore us down until we finally decided to pull the plug. I still maintain that this is a good idea, and constantly have to fight off the urge to "try again", however, to do it properly, I think funding would be necessary, or finding some way to organically gain a lot of users. Looking back, there are things I could have done to reduce my opex further, but in the end, it still wouldn't have mattered if I couldn't figure out how to acquire users.
- pencildiver 3y agoThanks for sharing this! If you're up for it, I'd love to talk more about your experience, especially the technical tooling. Working as fast as I can to understand the right way to approach the tech, as there are tradeoffs with performance and price. I'm at support @ searchagora .com
- grumpyviscacha 3y agoWow, it's cool to see this idea trending on HN! Full disclosure, I'm one of the co-founders at https://www.marmalade.co https://www.marmalade.co. Speaking from personal experience, it’s been a long road getting from the universe of all Shopify products to a curated inventory that’s easy for people to shop on. While ChatGPT isn't going to replace human curation anytime soon, the AI tailwind has made it much easier to build search and recommendation systems. On our end, we've definitely caught the semantic search bug. Watch out for it - you’ll wake up one day with a cross-modal hybrid search index on pinecone and any number of models on huggingface :). However, as you rightly point out, user growth is still the key. We're working toward launching a community aspect of the platform in the coming months as a solution.
- yoru-sulfur 3y agoFor those unaware, Shopify already has platform wide search. You can use https://shop.app/ https://shop.app/ (or the app), and it also has some chatbot thing that can offer suggestions
- senecaso 3y agoYes, this has been available for a few years now. Initially, they only indexed a very small number of shops, so it was less useful. Based on a few queries, it seems like the are still using some form of text-based search with rank boosting. Seems like they still aren't searching their entire base of shops, but they have increased the number of shops for sure, and they seem to be continuing to invest in the product, which is nice. It seems more useful now than it did the last time I checked!
- jasonlbaptiste 3y agoYou should def give Algolia and Typesense a try. You can get 10k in free Algolia credits for the first year too via Secret (startup deals site).
- pencildiver 3y agoWill do. Thanks for the heads up!
- 1vuio0pswjnm7 3y ago"There's about 25 million products on Agora right now." How many stores are represented in index.
- ashvardanian 3y agoCool project! As you scale, you may benefit from these two projects I maintain, and the Big Tech uses :) https://github.com/unum-cloud/usearch https://github.com/unum-cloud/usearch - for faster search https://github.com/unum-cloud/uform https://github.com/unum-cloud/uform - for cheaper multi-lingual multi-modal embeddings Feel free to reach out with feedback and feature requests!
- dangoodmanUT 3y agoI've been following your work for a while, was really excited to play with UDisk but I guess that got dropped in favor of AI solutions?
- ashvardanian 3y agoDue to constrained resources we had to prioritize the smaller and simpler projects - USearch, UForm, UCall, and on the personal side - StringZilla, and SimSIMD. That’s a lot for a team without revenue and VC funding. I am still actively thinking about open-sourcing UDisk. Thanks for keeping tabs on us :)
- pencildiver 3y agoCan't believe I missed this. Taking a look at both repos now. The further I get into this space, the less I feel like I know. Appreciate you sharing, will reach out!
- ashvardanian 3y agoDon’t worry, our solutions aren’t well known. They are either used by enthusiasts, or the Big Tech, and the latter don’t like mentioning that :)
- gajus 3y agoThis is cool
- freefruit 3y agoCould you make it so, that I can easily open a product in a new tab. I like to compare lots of products at the same time.
- pencildiver 3y agoAbsolutely. Currently on the search results page, you can click on the title / price area to open the product URL in a new tab. Open to any suggestions as well. So the product URL is accessible from the search results page or if you click on the product image to open the product listing and click on 'visit product'. Let me know if that makes sense and if you have any suggestions to make it better!
- gokulkrishh09 3y ago[dead]
- shubham_sinha 3y ago[dead]
- usrme 3y agoMaybe I'm clearly ignorant, but how does this differ from Klarna (https://www.klarna.com https://www.klarna.com)?
- tmikaeld 3y agoDo you mean pricerunner? There's similar price comparison sites, but they don't index every store available. You basically have to submit your listing.
- spiderfarmer 3y agoIs there a similarity at all? One is a search engine, the other a leeching "buy now pay later" scheme as a service.
- sails 3y agoClicking an item could show you similar items before it takes you to the item (or have capability for similar)
- callmeed 3y agoI built this a couple years ago (now defunct) for the same reason :) The public JSON endpoints on shopify stores make it pretty easy to get the data. You mentioned using Mongo but it sounds expensive. I honestly think you could do this with just elastic or even postgres full text search and save money. Here's a pro tip + feature you should implement: Shopify has a semi-hidden hack where you can link directly to checkout of a product if you know the variant ID. You could add a BUY NOW button to your site without forcing the user to navigate the original site or checkout flow. Example: https://hapaboardshop.com/cart/42165521907955 https://hapaboardshop.com/cart/42165521907955 (it also supports quantities and coupon codes) A word of caution: more products isn't necessarily better. I definitely found there to be a long tail of really bad shopify stores and products. IMO it's better to curate or audit the stores you index–otherwise you risk your site being littered with kitchy t-shirts or drop-shipping garbage.
- senecaso 3y agoI didnt know about the link to checkout. That's a slightly nicer user experience for sure. Still, its confusing for users who want to do more shopping at the same time. I had users who clicked on a number of items, clicked "add to cart" in each one (all different shops), and then couldn't figure out how to checkout on the main site afterwards! Obviously people were looking for a more complete one-stop-shopping experience than I was providing at the time.
- callmeed 3y agoI mean a single checkout from multiple shopify stores isn't really possible (at least by 3rd parties) My hypothesis is that, if you could drive traffic to your site and offer a fast checkout experience, there's probably multiple ways to monetize that. Driving the traffic is the hard part.
- pencildiver 3y agoThanks for the heads up! I spent some time trying to get the cart route to work. Doesn't seem to be supported anymore (link you sent leads to a 404 page). Tried it with every combination of Product ID, Variant ID, etc. Let me know if you have any ideas on how to get this to work. It would be a great feature to add to Agora. And I agree on quality over quantity. Writing a script to remove all stores that are shutdown, products that are sold out, and a few other characteristics. Heavily focusing on the search algorithm and data quality now.