7 ms·

Show HN: I scrape Steam data every month and it's yours to download for free

161 points by csmets 2y ago

Yeah, there's AI, but I added it because I found it easier to find answers I'm looking for. For the data scientists, you can download the CSV and go crazy. Would love to know what discoveries or learnings can be found from it. To download the raw scraped data you need to become a paid member but you don't really need it unless you're wanting to finesse a table of data for a particular need. The cost is mostly just an incentive to help me pay the bills for running the website. The bunch of available CSV files contain large amounts of data which has everything from tags, genres, pricing, wishlists, estimated revenue, etc. It's what the AI is reading from. Hope you find it useful :-)

endre 2y ago
nice
ddxv 2y ago
Hi, I'm interested in scraping steam too. Do you have the scraper code available open source or one you recommend?
- lolinder 2y ago
  Have you looked over the data that OP is providing here and determined that it doesn't meet your needs? Generally it's polite to avoid scraping if you can help it, so I'd start by considering whether OP is already providing what you are looking for.
  - DrammBA 2y ago
    On the other hand you need to be a paid member to download the raw scraped data, so it isn't unreasonable to want to learn how to scrape it instead.
    - schnebbau 2y ago
      Good idea, let's save $5 and sink dozens of hours into building our own instead
      - DrammBA 2y ago
        Isn't that the hacker spirit, wanting to put things together yourself? Let's revisit the comment that started this thread: > Hi, I'm interested in scraping steam too.
        schnebbau 2y ago
        I was responding directly to your objection with having to pay for it as the reason to do this, not hacking for the sake of hacking.
        diggan 2y ago
        Hacking for the sake of hacking VS hacking for the sake of saving money, why does it matter?
        schnebbau 2y ago
        Jesus Christ, it's 5 bucks. 5 bucks is not a reason to roll your own version. Roll your own version because you want to roll your own version, not to save 5 bucks.
        DrammBA 2y ago
        you seem to be really hung up on the 5 bucks while at the same time being angry that people are hung up on the 5 bucks, it's just 5 bucks man, if people want to pay it or not doesn't matter, let people hack it away if they want
        deleted 2y ago
        [deleted]
        nickthegreek 2y ago
- netruk44 2y ago
  I wrote a simple scraper for a 'steam game semantic search' app I built a while ago. It definitely won't fetch all the data that this person does though. It only fetches the current list of games on Steam, their store page information and some reviews for the game. The code quality probably isn't amazing, but it might give you an idea of how to get started with your own scraper. https://github.com/Netruk44/steam-embedding-search/blob/main/01_gamedataset/steamapi.py https://github.com/Netruk44/steam-embedding-search/blob/main...
  - ddxv 2y ago
    Thanks! That's perfect, just want somewhere to get started.
- DrammBA 2y ago
  https://steamdb.info/faq/#how-are-we-getting-this-information https://steamdb.info/faq/#how-are-we-getting-this-informatio... I found this explanation from steamdb that points to the various projects and libraries they use to gather all the data they have. It's not a how-to, but it has very useful info.
aranw 2y ago
Nice! It would be nice however to see more detail about the data you collect and what exactly you provide on top of it using AI or through aggregation etc
kmfrk 2y ago
I got some answers that weren't specifically about my questions in some instances. As someone who's just trying out the free demo, it's not a big deal, but maybe you can provide a way to flag answers for to redeem their credits? It would probably increase retention and help people chase down bugs.
Apreche 2y ago
Do you have data that https://steamdb.info/ https://steamdb.info/ doesn’t have?
- noirscape 2y ago
  Steamdb lacks an API for one, and the devs officially have a policy that they'll never make one, saying you should just scrape Steam directly instead of bugging them about it[0]. It means that steamdb, while extraordinarily useful for casual prodding at what's stored on Valve's servers, isn't very good if you want to run data analysis or something like that on the metadata of Steam games at scale. Not sure if it's legal to charge for the raw scrape when OP doesn't seem to be affiliated with Valve, but that's not up to me to figure out. [0]: https://steamdb.info/faq/ https://steamdb.info/faq/
  - joseda-hg 2y ago
    That seems pretty reasonable, it's their data, they just make useful visualizations
  - seanw444 2y ago
    This whole time I was under the impression that SteamDB was owned by Valve. Huh.
    - mistergoodwin 2y ago
      All run by one person who doesn't even take donations for the service they provide. They joined here on an Ask HN about "Those making $0/month or less on side projects – Show and tell" https://news.ycombinator.com/item?id=34548908 https://news.ycombinator.com/item?id=34548908
ghfhghg 2y ago
I guess the main differentiator over steamdb is getting the data in CSV? Might be good to clarify in the FAQ because the people I know who would pay for this are not the most techy types.
bdd8f1df777b 2y ago
It seems to be missing reviews? I have always thought about building my own recommendation engine from steam data, given how steam's own recommendation never works for me.
somenameforme 2y ago
Out of curiosity, what formula did you end up using for reviews:sales? I've looked into this a bunch and it's a very tough problem!
giancarlostoro 2y ago
> Yeah, there's AI, but I added it because I found it easier to find answers I'm looking for. For the data scientists, you can download the CSV and go crazy. This is kind of the only way I use AI really, to summarize things, and extract details, then review from the raw sources to make sure the LLM isn't misleading me. I find myself using this approach instead of Googling for things since Google crippled their search the last few years, it feels like every year its harder to find things with Google. I miss 2007 Google...
- dewey 2y ago
  Give Kagi a try, it's basically Google before it went to shit.
z3c0 2y ago
I had to refresh before posting, because I wanted to see if someone else beat me to being that HN commenter but... From the Terms of Service (emphasis mine): 6. Restrictions on Use You agree not to: Use the Service for any unlawful purpose. Attempt to reverse-engineer, modify, or *create derivative works of the Service.* Share, resell, or distribute downloadable data provided by the Service without explicit written permission. Do you intend to delineate the data provided by the service from "the Service" itself? It seems most fair that data received via Fair Use remains in that arena, pun fully intended. That aside, it's an intriguing dataset nonetheless, but I'd prefer to see a sample of the data before signing up.
- JadoJodo 2y ago
  At a glance, it appears the product is the “chat with the data" feature; The CSV is free.
  - DrammBA 2y ago
    What I don't understand is the difference between 'Download all CSV data' in the free tier and 'Download CSV data' / 'Download raw data' in the paid member tier. It seems that the free CSV data is likely an extract or digest of the raw data offered as a sample.
  - z3c0 2y ago
    I might be inclined to seek the raw data, should it be more cost effective than scraping Steam myself. Being a user, free, paid, or anonymous, can still be under the thumb of their ToS, especially so if they force a dialog in front of you to agree to the ToS while signing up. I'm merely pointing out hurdles to the OP that may obstruct some of the people they are trying to reach.
- akudha 2y ago
  Steamdb.info displays graphs etc. Is that considered a “derivative work”? I am not sure what is considered derivative work and what isn’t
  - 2y ago
Ksudijaan 2y ago
It is not that difficult to scrape Steam data using SteamKit, right? I build a website around Steam data a couple of years ago, with a small scraper app which was hourly scraping new updates (using SteamKit) and putting it into my database. The biggest advantage that SteamDB has, is that it has a ton of historical data. That is not retrievable from the Steam Network, so the only way to have gotten historical data is to start early. My website is now defunct for a year, but I've kept the scraper running. I now have 7 years of historical data in my database.
bloomingkales 2y ago
Question for OP, or anyone that considered it: Do you think Steam reviews are coordinated?
- bluefirebrand 2y ago
  I think for basically any possible online discussion, from Facebook to Hacker News to Steam Reviews, you should always keep in mind that some portion of it is probably astroturfed, to some scale Anything from a small indie game to a huge AAA title, you can bet that the creators got their friends and family to post some nice reviews early, just to give it that positive bump
  - bloomingkales 2y ago
    I was specifically alarmed by what looked like review bombing of a indie game. I just can't imagine it. I need to write a small llm plugin that collapses coordinated/astroturfed reviews.
    - bluefirebrand 2y ago
      The smaller the scale the easier to astroturf, honestly If there are only 20 reviews it's pretty easy for one person to review bomb on their own if they want to It gets much harder when there are 2 million reviews
  - hackburg 2y ago
    [dead]
- shagie 2y ago
  > Do you think Steam reviews are coordinated? Yes. It's not even a question. Steam flags outliers too. https://store.steampowered.com/app/281990/Stellaris/ https://store.steampowered.com/app/281990/Stellaris/ It got review bombed starting on Feb 14th because a different game that the company makes (HOI4) released DLC that upset the sensibilities of part of that player base. ( https://old.reddit.com/r/Stellaris/comments/1iqzih8/why_is_stellaris_getting_review_bombed/ https://old.reddit.com/r/Stellaris/comments/1iqzih8/why_is_s... ) --- There are Steam review bots for discord ( https://www.codecks.io/steam-bot/ https://www.codecks.io/steam-bot/ ) and that also encourages people who are members of a game's discord to leave a (positive) review. --- It's a certainty that reviews are coordinated through a number of different means.
m00dy 2y ago
If you need to be a paid member to download csv file, then it is not free :) lol
- xerox13ster 2y ago
  If you need to make an account and give this guy personal information (a digital commodity like oil) to see the data it's not free lmao
  - stronglikedan 2y ago
    > If you need to make an account and give this guy personal information In this case, you don't. That's just to weed out people who can't figure out temporary emails. I just used one to create an account without turning over any PI.
    - xerox13ster 2y ago
      [flagged]
      - antasvara 2y ago
        Is your issue actually with "creating an account," or is it with "giving up your personal information?" Because as the commenter indicated, you can create an account on this site without giving up any personal information. So while you're "correct" in the sense that you do need an account, it seems that the meat of your point (giving up personal data) has been addressed.
    - deleted 2y ago
      [deleted]
- nickthegreek 2y ago
  free tier allows you to download the csv.
bitbasher 2y ago
You use the chat but the credit used isn't updated immediately in the lower left.
stared 2y ago
Regarding Steam data, I am curious about how games are being played (hours spent) and, even more, about their co-occurrence (i.e., player X spent both time on game A and game B). I would love to make a visualization like https://p.migdal.pl/tagoverflow/?site=gaming&size=32 https://p.migdal.pl/tagoverflow/?site=gaming&size=32, but for Steam data. Also, for deeper insight than sales volumes (e.g., game design, general trends, demographics, types of players), such things would be crucial. and
ryanisnan 2y ago
I thought this would a stavros post. Thanks for your efforts!
eamsen 2y ago
Can you please provide examples for the raw data? As a user, I would like to know what I'm buying before paying.
appleaday1 2y ago
looking into this thank you.
thot_experiment 2y ago
> To download the raw scraped data you need to become a paid member If I have to pay to download the data how is it mine to download for free?
rapfaria 2y ago
Do the HN crowd read "for free" on the title, click it, scan the page in a milisecond, see "Pricing" on the top, and come back to complain in the comments? Geez
- deleted 2y ago
  [deleted]
- satiric 2y ago
  Yes, it's disingenuous. Free means $0.00.
- gloosx 2y ago
  There is a hope I would see just a single bold FREE when I click pricing still, so I check it first before complaining. And yes, "for free" means free as in the air you breathe for free. Anything else falls into a "mousetrap free" category.
happyopossum 2y ago
> I scrape Steam data every month and it's yours to download for free Does not line up with > To download the raw scraped data you need to become a paid member Sooo, clickbait or just plain dishonest?
- voodooEntity 2y ago
  ye clickbait first i saw was pricing. this should be deleted (thread)
- antasvara 2y ago
  A CSV of this data is free to download per the pricing page, whereas the raw data (not sure what that looks like versus the CSV) requires a paid account. So I guess it depends if you consider the CSV as fundamentally different from the raw data in a way that makes this clickbait.

161 points by csmets 2y ago

endre 2y ago

nice

ddxv 2y ago

Hi, I'm interested in scraping steam too. Do you have the scraper code available open source or one you recommend?

lolinder 2y ago
Have you looked over the data that OP is providing here and determined that it doesn't meet your needs? Generally it's polite to avoid scraping if you can help it, so I'd start by considering whether OP is already providing what you are looking for.
- DrammBA 2y ago
  On the other hand you need to be a paid member to download the raw scraped data, so it isn't unreasonable to want to learn how to scrape it instead.
  - schnebbau 2y ago
    Good idea, let's save $5 and sink dozens of hours into building our own instead
    - DrammBA 2y ago
      Isn't that the hacker spirit, wanting to put things together yourself? Let's revisit the comment that started this thread: > Hi, I'm interested in scraping steam too.
      - schnebbau 2y ago
        I was responding directly to your objection with having to pay for it as the reason to do this, not hacking for the sake of hacking.
        diggan 2y ago
        Hacking for the sake of hacking VS hacking for the sake of saving money, why does it matter?
        schnebbau 2y ago
        Jesus Christ, it's 5 bucks. 5 bucks is not a reason to roll your own version. Roll your own version because you want to roll your own version, not to save 5 bucks.
        DrammBA 2y ago
        you seem to be really hung up on the 5 bucks while at the same time being angry that people are hung up on the 5 bucks, it's just 5 bucks man, if people want to pay it or not doesn't matter, let people hack it away if they want
        deleted 2y ago
        [deleted]
        nickthegreek 2y ago
netruk44 2y ago
I wrote a simple scraper for a 'steam game semantic search' app I built a while ago. It definitely won't fetch all the data that this person does though. It only fetches the current list of games on Steam, their store page information and some reviews for the game. The code quality probably isn't amazing, but it might give you an idea of how to get started with your own scraper. https://github.com/Netruk44/steam-embedding-search/blob/main/01_gamedataset/steamapi.py https://github.com/Netruk44/steam-embedding-search/blob/main...
- ddxv 2y ago
  Thanks! That's perfect, just want somewhere to get started.
DrammBA 2y ago
https://steamdb.info/faq/#how-are-we-getting-this-information https://steamdb.info/faq/#how-are-we-getting-this-informatio... I found this explanation from steamdb that points to the various projects and libraries they use to gather all the data they have. It's not a how-to, but it has very useful info.

aranw 2y ago

Nice! It would be nice however to see more detail about the data you collect and what exactly you provide on top of it using AI or through aggregation etc

kmfrk 2y ago

I got some answers that weren't specifically about my questions in some instances. As someone who's just trying out the free demo, it's not a big deal, but maybe you can provide a way to flag answers for to redeem their credits? It would probably increase retention and help people chase down bugs.

Apreche 2y ago

Do you have data that https://steamdb.info/ https://steamdb.info/ doesn’t have?

noirscape 2y ago
Steamdb lacks an API for one, and the devs officially have a policy that they'll never make one, saying you should just scrape Steam directly instead of bugging them about it[0]. It means that steamdb, while extraordinarily useful for casual prodding at what's stored on Valve's servers, isn't very good if you want to run data analysis or something like that on the metadata of Steam games at scale. Not sure if it's legal to charge for the raw scrape when OP doesn't seem to be affiliated with Valve, but that's not up to me to figure out. [0]: https://steamdb.info/faq/ https://steamdb.info/faq/
- joseda-hg 2y ago
  That seems pretty reasonable, it's their data, they just make useful visualizations
- seanw444 2y ago
  This whole time I was under the impression that SteamDB was owned by Valve. Huh.
  - mistergoodwin 2y ago
    All run by one person who doesn't even take donations for the service they provide. They joined here on an Ask HN about "Those making $0/month or less on side projects – Show and tell" https://news.ycombinator.com/item?id=34548908 https://news.ycombinator.com/item?id=34548908

ghfhghg 2y ago

I guess the main differentiator over steamdb is getting the data in CSV? Might be good to clarify in the FAQ because the people I know who would pay for this are not the most techy types.

bdd8f1df777b 2y ago

It seems to be missing reviews? I have always thought about building my own recommendation engine from steam data, given how steam's own recommendation never works for me.

somenameforme 2y ago

Out of curiosity, what formula did you end up using for reviews:sales? I've looked into this a bunch and it's a very tough problem!

giancarlostoro 2y ago

> Yeah, there's AI, but I added it because I found it easier to find answers I'm looking for. For the data scientists, you can download the CSV and go crazy. This is kind of the only way I use AI really, to summarize things, and extract details, then review from the raw sources to make sure the LLM isn't misleading me. I find myself using this approach instead of Googling for things since Google crippled their search the last few years, it feels like every year its harder to find things with Google. I miss 2007 Google...

dewey 2y ago
Give Kagi a try, it's basically Google before it went to shit.

z3c0 2y ago

I had to refresh before posting, because I wanted to see if someone else beat me to being that HN commenter but... From the Terms of Service (emphasis mine): 6. Restrictions on Use You agree not to: Use the Service for any unlawful purpose. Attempt to reverse-engineer, modify, or *create derivative works of the Service.* Share, resell, or distribute downloadable data provided by the Service without explicit written permission. Do you intend to delineate the data provided by the service from "the Service" itself? It seems most fair that data received via Fair Use remains in that arena, pun fully intended. That aside, it's an intriguing dataset nonetheless, but I'd prefer to see a sample of the data before signing up.

JadoJodo 2y ago
At a glance, it appears the product is the “chat with the data" feature; The CSV is free.
- DrammBA 2y ago
  What I don't understand is the difference between 'Download all CSV data' in the free tier and 'Download CSV data' / 'Download raw data' in the paid member tier. It seems that the free CSV data is likely an extract or digest of the raw data offered as a sample.
- z3c0 2y ago
  I might be inclined to seek the raw data, should it be more cost effective than scraping Steam myself. Being a user, free, paid, or anonymous, can still be under the thumb of their ToS, especially so if they force a dialog in front of you to agree to the ToS while signing up. I'm merely pointing out hurdles to the OP that may obstruct some of the people they are trying to reach.
akudha 2y ago
Steamdb.info displays graphs etc. Is that considered a “derivative work”? I am not sure what is considered derivative work and what isn’t
- 2y ago

Ksudijaan 2y ago

It is not that difficult to scrape Steam data using SteamKit, right? I build a website around Steam data a couple of years ago, with a small scraper app which was hourly scraping new updates (using SteamKit) and putting it into my database. The biggest advantage that SteamDB has, is that it has a ton of historical data. That is not retrievable from the Steam Network, so the only way to have gotten historical data is to start early. My website is now defunct for a year, but I've kept the scraper running. I now have 7 years of historical data in my database.

bloomingkales 2y ago

Question for OP, or anyone that considered it: Do you think Steam reviews are coordinated?

bluefirebrand 2y ago
I think for basically any possible online discussion, from Facebook to Hacker News to Steam Reviews, you should always keep in mind that some portion of it is probably astroturfed, to some scale Anything from a small indie game to a huge AAA title, you can bet that the creators got their friends and family to post some nice reviews early, just to give it that positive bump
- bloomingkales 2y ago
  I was specifically alarmed by what looked like review bombing of a indie game. I just can't imagine it. I need to write a small llm plugin that collapses coordinated/astroturfed reviews.
  - bluefirebrand 2y ago
    The smaller the scale the easier to astroturf, honestly If there are only 20 reviews it's pretty easy for one person to review bomb on their own if they want to It gets much harder when there are 2 million reviews
- hackburg 2y ago
  [dead]
shagie 2y ago
> Do you think Steam reviews are coordinated? Yes. It's not even a question. Steam flags outliers too. https://store.steampowered.com/app/281990/Stellaris/ https://store.steampowered.com/app/281990/Stellaris/ It got review bombed starting on Feb 14th because a different game that the company makes (HOI4) released DLC that upset the sensibilities of part of that player base. ( https://old.reddit.com/r/Stellaris/comments/1iqzih8/why_is_stellaris_getting_review_bombed/ https://old.reddit.com/r/Stellaris/comments/1iqzih8/why_is_s... ) --- There are Steam review bots for discord ( https://www.codecks.io/steam-bot/ https://www.codecks.io/steam-bot/ ) and that also encourages people who are members of a game's discord to leave a (positive) review. --- It's a certainty that reviews are coordinated through a number of different means.

m00dy 2y ago

If you need to be a paid member to download csv file, then it is not free :) lol

xerox13ster 2y ago
If you need to make an account and give this guy personal information (a digital commodity like oil) to see the data it's not free lmao
- stronglikedan 2y ago
  > If you need to make an account and give this guy personal information In this case, you don't. That's just to weed out people who can't figure out temporary emails. I just used one to create an account without turning over any PI.
  - xerox13ster 2y ago
    [flagged]
    - antasvara 2y ago
      Is your issue actually with "creating an account," or is it with "giving up your personal information?" Because as the commenter indicated, you can create an account on this site without giving up any personal information. So while you're "correct" in the sense that you do need an account, it seems that the meat of your point (giving up personal data) has been addressed.
  - deleted 2y ago
    [deleted]
nickthegreek 2y ago
free tier allows you to download the csv.

bitbasher 2y ago

You use the chat but the credit used isn't updated immediately in the lower left.

stared 2y ago

Regarding Steam data, I am curious about how games are being played (hours spent) and, even more, about their co-occurrence (i.e., player X spent both time on game A and game B). I would love to make a visualization like https://p.migdal.pl/tagoverflow/?site=gaming&size=32 https://p.migdal.pl/tagoverflow/?site=gaming&size=32, but for Steam data. Also, for deeper insight than sales volumes (e.g., game design, general trends, demographics, types of players), such things would be crucial. and

ryanisnan 2y ago

I thought this would a stavros post. Thanks for your efforts!

eamsen 2y ago

Can you please provide examples for the raw data? As a user, I would like to know what I'm buying before paying.

appleaday1 2y ago

looking into this thank you.

thot_experiment 2y ago

> To download the raw scraped data you need to become a paid member If I have to pay to download the data how is it mine to download for free?

rapfaria 2y ago

Do the HN crowd read "for free" on the title, click it, scan the page in a milisecond, see "Pricing" on the top, and come back to complain in the comments? Geez

deleted 2y ago
[deleted]
satiric 2y ago
Yes, it's disingenuous. Free means $0.00.
gloosx 2y ago
There is a hope I would see just a single bold FREE when I click pricing still, so I check it first before complaining. And yes, "for free" means free as in the air you breathe for free. Anything else falls into a "mousetrap free" category.

happyopossum 2y ago

> I scrape Steam data every month and it's yours to download for free Does not line up with > To download the raw scraped data you need to become a paid member Sooo, clickbait or just plain dishonest?

voodooEntity 2y ago
ye clickbait first i saw was pricing. this should be deleted (thread)
antasvara 2y ago
A CSV of this data is free to download per the pricing page, whereas the raw data (not sure what that looks like versus the CSV) requires a paid account. So I guess it depends if you consider the CSV as fundamentally different from the raw data in a way that makes this clickbait.