10 ms·

Good to know. Thanks for the info. I'm thinking about building my own price comparison site in another niche. How are you scraping the data right now?

• generalizations 3y ago
  An unholy combination of bash, python and TCL scripts running in async loops on an old computer under my desk :D TCL is actually a great scripting language - it handles text strings as well as bash (and this is all just text), it does file I/O really easily, and it has nested arrays that bash doesn't do. For my purposes, it's a lot cleaner to work with than python. For the scraping itself - I tried wget with useragent strings, and quickly hit 503 blocked errors. After that I gave it the whole nine yards, and threw every trick I could come up with at the problem (considering my last project involved netbooting xen, this was completely outside my field). I honestly would rather not say how I did it though. If anyone from Amazon's bot detection is here, I don't want to give them hints. Edit: I'm collecting similar projects in the sidebar of my site, it'd be fun to add yours when you publish it.