6 ms·

I wrote a thing to pull the prices of all the hard drives on Amazon every 12 hours and sort them by price per GB. https://diskprices.com/ https://diskprices.com

by synack 9y ago

I wrote a thing to pull the prices of all the hard drives on Amazon every 12 hours and sort them by price per GB. https://diskprices.com/ https://diskprices.com/

mark-r 9y ago
I hope that's not the sort of thing that gets you banned from Amazon forever. That would be kind of harsh.
- synack 9y ago
  As my understanding of the terms of service goes, it's allowed as long as I don't compare the prices to other sites.
- SquareWheel 9y ago
  It's not an abuse of the API because prices are not being compared against products on other stores. However he is abusing their affiliate policy by not including an associates program disclaimer on the page.
  - synack 9y ago
    There's a disclaimer at the bottom.
    - SquareWheel 9y ago
      It's not disclosing that you make use of affiliates. Read section 5 of the associates program agreement. https://affiliate-program.amazon.com/help/operating/agreement https://affiliate-program.amazon.com/help/operating/agreemen...
jdc0589 9y ago
looks like the one thats the cheapest on your site @ $0.022 (Seagate Expansion 8TB Desktop External Hard Drive USB 3.0 (STEB8000100) ) is actually on sale and works out to $0.0186 right now. I may jump on that quickly....
- synack 9y ago
  Yeah, the Prime Day promotion isn't exposed through their API so I can't account for that easily.
  - jdc0589 9y ago
    figures. that's a damn good deal though; too bad its limited to 1 per customer.
Theodores 9y ago
I presume you get affiliate revenue too. Does that cover your hosting fees? Your site is hard disk drives redux, could you apply the same recipe for a comparison site to anything else with a numeric bang per buck quantifier, to then profit? I suspect there are few such categories but you could put together an interesting store that way if the affiliate marketing paid its way.
- synack 9y ago
  I turn a small profit on the affiliate revenue, yes.
- parametrek 9y ago
  > could you apply the same recipe for a comparison site to anything else with a numeric bang per buck quantifier Absolutely. I've done this with flashlights, for example.
antisthenes 9y ago
How did you select those disks? Did you just hard code a list of ASINs to query through their API or is this done by scraping Amazon front-end?
- synack 9y ago
  Amazon's product categories are identified as BrowseNodes, each has a unique ID. I found the IDs of all the relevant BrowseNodes, then use the API to query what products are available in each BrowseNode. I then use a ridiculous list of regexes to parse the product names to figure out the capacity and drive type, as Amazon's tags for those fields are really unreliable.
  - antisthenes 9y ago
    I only ask because the HDD list in particular seemed to be quite short. The last time I did this, I scraped the internal hard drive category (from the front-end) and filtered everything out that was smaller than 4TB and still ended up with at least 200 drives. For tags, are you using the "Size" or the <ns2:HardDiskSize Units="TB">4.0</ns2:HardDiskSize> tag? I did a few test queries and found those specific tags to be pretty reliable. If you're open to it, we can discuss it outside of HN to see if there's a better way. disclaimer: I do not have any vested interest in Amazon affiliate programs, I work with their API for my day job and always looking to learn. *edit: are you using the advertising api or the mws product api?
roter 9y ago
Just a heads up, when I filtered on just SSDs, the top of the list was a BarraCuda HDD (ST3000DM008). It seems to have been incorrectly placed under SSDs on Amazon. Bookmarked your site though, thanks.
RobLach 9y ago
I'm still wondering why external drives are cheaper than internal ones. It seems silly that if I wanted to buy an 8TB drive for my desktop I should buy an external and toss the enclosure. My only guess is market pressure because of higher demand. Possibly externals drives are pulled from older stock? Does anyone have another explanation?
- PascLeRasc 9y ago
  People who shuck on /r/homelab often find totally different models when buying a few from the same place. It lets drive manufacturers stick any drive they want in the enclosure as long as it's the advertised capacity and RPM.
- verbatim 9y ago
  Just a guess, but, one factor could be a shorter warranty?
  - losteric 9y ago
    Plus some external drive customers immediately shuck the device and receive zero warranty.
- mrb 9y ago
  "My only guess is market pressure because of higher demand." That's exactly right. In the mass consumer market (ie. people buying hard drives at retailers), a lot more external drives are sold than internal drives. Larger market → more competition → lower prices. On the other hand, for companies buying directly from manufacturers or from distributors, I would bet an internal drive sells for (a bit) less than the comparable external drive. Though I have never seen a price sheet. Would love to hear confirmation from someone in the industry.
- Glyptodon 9y ago
  Externals also often only have 1 yr warranties vs 2, 3, or 5 for bare drives in my experience.
- fencepost 9y ago
dabockster 9y ago
Nice! Is this open source? I'd love to contribute to this.
parametrek 9y ago
This is good work! I might reference your project in an article I've been writing about responsible affiliate sites.
xmichael99 9y ago
Great site, I can see I'll be using it alot! What would be really nice is if you could add in a m2 ssd drive type.
RJIb8RBYxzAMX9u 9y ago
What's the risk of getting counterfeits, or getting used drives sold as new? Now that NewEgg's also into the 3rd party re-seller shenanigans, I've been getting my hardware from BB or Fry's more often. Zemo's right: "And the Avengers [Amazon / NewEgg]? They went home. I knew I couldn't kill them. More powerful men [Walmart] than me have tried. But if I could get them to kill each other [3rd party re-sellers]..."
arthurfm 9y ago
Just in case you weren't aware, when I view your website on my Windows 10 PC or Android smartphone all of the prices have an 'Â' preceding the £. Screenshot: https://vgy.me/f9rd4b.png https://vgy.me/f9rd4b.png
- kalleboo 9y ago
  The page doesn't seem to be declaring an encoding - on my computer set to Japanese the browser is assuming Shift-JIS and making a mess of everything
sschueller 9y ago
Odd, some appear to be missing. The cheapest 10 TB on that list is $417. I can get a Seagate IronWolf for $341 on Amazon[1] [1] https://www.amazon.com/Seagate-IronWolf-3-5-Inch-Internal-ST10000VN0004/dp/B01IA9GU0Q/ref=sr_1_5?ie=UTF8&qid=1499858143&sr=8-5&keywords=Seagate+IronWolf https://www.amazon.com/Seagate-IronWolf-3-5-Inch-Internal-ST...
Freaky 9y ago
I made one for scan.co.uk: https://hur.st/scandisk/ https://hur.st/scandisk/
glenneroo 9y ago
Nice work! The German version unfortunately isn't able to display Euro signs properly "â‚¬0.250" is what I see with Chrome.
doomjunky 9y ago
You can fix the encoding problem on your site with: <meta charset="UTF-8">
mikewhy 9y ago
This is super handy! To anyone using it, always click through and see what the product is. Right now it says the cheapest per GB internal drive is this one[0]. Turns out it's actually just the external inclosure. It doesn't actually come with storage. [0]: https://www.amazon.ca/StarTech-com-10Gbps-Enclosure-SATA-Drives/dp/B00XLAZEFC https://www.amazon.ca/StarTech-com-10Gbps-Enclosure-SATA-Dri...

by synack 9y ago

I wrote a thing to pull the prices of all the hard drives on Amazon every 12 hours and sort them by price per GB. https://diskprices.com/ https://diskprices.com/

mark-r 9y ago

I hope that's not the sort of thing that gets you banned from Amazon forever. That would be kind of harsh.

synack 9y ago
As my understanding of the terms of service goes, it's allowed as long as I don't compare the prices to other sites.
SquareWheel 9y ago
It's not an abuse of the API because prices are not being compared against products on other stores. However he is abusing their affiliate policy by not including an associates program disclaimer on the page.
- synack 9y ago
  There's a disclaimer at the bottom.
  - SquareWheel 9y ago
    It's not disclosing that you make use of affiliates. Read section 5 of the associates program agreement. https://affiliate-program.amazon.com/help/operating/agreement https://affiliate-program.amazon.com/help/operating/agreemen...

jdc0589 9y ago

looks like the one thats the cheapest on your site @ $0.022 (Seagate Expansion 8TB Desktop External Hard Drive USB 3.0 (STEB8000100) ) is actually on sale and works out to $0.0186 right now. I may jump on that quickly....

synack 9y ago
Yeah, the Prime Day promotion isn't exposed through their API so I can't account for that easily.
- jdc0589 9y ago
  figures. that's a damn good deal though; too bad its limited to 1 per customer.

Theodores 9y ago

I presume you get affiliate revenue too. Does that cover your hosting fees? Your site is hard disk drives redux, could you apply the same recipe for a comparison site to anything else with a numeric bang per buck quantifier, to then profit? I suspect there are few such categories but you could put together an interesting store that way if the affiliate marketing paid its way.

synack 9y ago
I turn a small profit on the affiliate revenue, yes.
parametrek 9y ago
> could you apply the same recipe for a comparison site to anything else with a numeric bang per buck quantifier Absolutely. I've done this with flashlights, for example.

antisthenes 9y ago

How did you select those disks? Did you just hard code a list of ASINs to query through their API or is this done by scraping Amazon front-end?

synack 9y ago
Amazon's product categories are identified as BrowseNodes, each has a unique ID. I found the IDs of all the relevant BrowseNodes, then use the API to query what products are available in each BrowseNode. I then use a ridiculous list of regexes to parse the product names to figure out the capacity and drive type, as Amazon's tags for those fields are really unreliable.
- antisthenes 9y ago
  I only ask because the HDD list in particular seemed to be quite short. The last time I did this, I scraped the internal hard drive category (from the front-end) and filtered everything out that was smaller than 4TB and still ended up with at least 200 drives. For tags, are you using the "Size" or the <ns2:HardDiskSize Units="TB">4.0</ns2:HardDiskSize> tag? I did a few test queries and found those specific tags to be pretty reliable. If you're open to it, we can discuss it outside of HN to see if there's a better way. disclaimer: I do not have any vested interest in Amazon affiliate programs, I work with their API for my day job and always looking to learn. *edit: are you using the advertising api or the mws product api?

roter 9y ago

Just a heads up, when I filtered on just SSDs, the top of the list was a BarraCuda HDD (ST3000DM008). It seems to have been incorrectly placed under SSDs on Amazon. Bookmarked your site though, thanks.

RobLach 9y ago

I'm still wondering why external drives are cheaper than internal ones. It seems silly that if I wanted to buy an 8TB drive for my desktop I should buy an external and toss the enclosure. My only guess is market pressure because of higher demand. Possibly externals drives are pulled from older stock? Does anyone have another explanation?

PascLeRasc 9y ago
People who shuck on /r/homelab often find totally different models when buying a few from the same place. It lets drive manufacturers stick any drive they want in the enclosure as long as it's the advertised capacity and RPM.
verbatim 9y ago
Just a guess, but, one factor could be a shorter warranty?
- losteric 9y ago
  Plus some external drive customers immediately shuck the device and receive zero warranty.
mrb 9y ago
"My only guess is market pressure because of higher demand." That's exactly right. In the mass consumer market (ie. people buying hard drives at retailers), a lot more external drives are sold than internal drives. Larger market → more competition → lower prices. On the other hand, for companies buying directly from manufacturers or from distributors, I would bet an internal drive sells for (a bit) less than the comparable external drive. Though I have never seen a price sheet. Would love to hear confirmation from someone in the industry.
Glyptodon 9y ago
Externals also often only have 1 yr warranties vs 2, 3, or 5 for bare drives in my experience.
fencepost 9y ago

dabockster 9y ago

Nice! Is this open source? I'd love to contribute to this.

parametrek 9y ago

This is good work! I might reference your project in an article I've been writing about responsible affiliate sites.

xmichael99 9y ago

Great site, I can see I'll be using it alot! What would be really nice is if you could add in a m2 ssd drive type.

RJIb8RBYxzAMX9u 9y ago

What's the risk of getting counterfeits, or getting used drives sold as new? Now that NewEgg's also into the 3rd party re-seller shenanigans, I've been getting my hardware from BB or Fry's more often. Zemo's right: "And the Avengers [Amazon / NewEgg]? They went home. I knew I couldn't kill them. More powerful men [Walmart] than me have tried. But if I could get them to kill each other [3rd party re-sellers]..."

arthurfm 9y ago

Just in case you weren't aware, when I view your website on my Windows 10 PC or Android smartphone all of the prices have an 'Â' preceding the £. Screenshot: https://vgy.me/f9rd4b.png https://vgy.me/f9rd4b.png

kalleboo 9y ago
The page doesn't seem to be declaring an encoding - on my computer set to Japanese the browser is assuming Shift-JIS and making a mess of everything

sschueller 9y ago

Odd, some appear to be missing. The cheapest 10 TB on that list is $417. I can get a Seagate IronWolf for $341 on Amazon[1] [1] https://www.amazon.com/Seagate-IronWolf-3-5-Inch-Internal-ST10000VN0004/dp/B01IA9GU0Q/ref=sr_1_5?ie=UTF8&qid=1499858143&sr=8-5&keywords=Seagate+IronWolf https://www.amazon.com/Seagate-IronWolf-3-5-Inch-Internal-ST...

Freaky 9y ago

I made one for scan.co.uk: https://hur.st/scandisk/ https://hur.st/scandisk/

glenneroo 9y ago

Nice work! The German version unfortunately isn't able to display Euro signs properly "â‚¬0.250" is what I see with Chrome.

doomjunky 9y ago

You can fix the encoding problem on your site with: <meta charset="UTF-8">

mikewhy 9y ago

This is super handy! To anyone using it, always click through and see what the product is. Right now it says the cheapest per GB internal drive is this one[0]. Turns out it's actually just the external inclosure. It doesn't actually come with storage. [0]: https://www.amazon.ca/StarTech-com-10Gbps-Enclosure-SATA-Drives/dp/B00XLAZEFC https://www.amazon.ca/StarTech-com-10Gbps-Enclosure-SATA-Dri...