4 ms·

Hi, I'm interested in scraping steam too. Do you have the scraper code available open source or one you recommend?

by ddxv 2y ago

lolinder 2y ago
Have you looked over the data that OP is providing here and determined that it doesn't meet your needs? Generally it's polite to avoid scraping if you can help it, so I'd start by considering whether OP is already providing what you are looking for.
- DrammBA 2y ago
  On the other hand you need to be a paid member to download the raw scraped data, so it isn't unreasonable to want to learn how to scrape it instead.
  - schnebbau 2y ago
    Good idea, let's save $5 and sink dozens of hours into building our own instead
    - DrammBA 2y ago
      Isn't that the hacker spirit, wanting to put things together yourself? Let's revisit the comment that started this thread: > Hi, I'm interested in scraping steam too.
      - schnebbau 2y ago
        I was responding directly to your objection with having to pay for it as the reason to do this, not hacking for the sake of hacking.
        diggan 2y ago
        Hacking for the sake of hacking VS hacking for the sake of saving money, why does it matter?
        schnebbau 2y ago
        Jesus Christ, it's 5 bucks. 5 bucks is not a reason to roll your own version. Roll your own version because you want to roll your own version, not to save 5 bucks.
        DrammBA 2y ago
        you seem to be really hung up on the 5 bucks while at the same time being angry that people are hung up on the 5 bucks, it's just 5 bucks man, if people want to pay it or not doesn't matter, let people hack it away if they want
        deleted 2y ago
        [deleted]
        nickthegreek 2y ago
        Its not $5. It is free or $5/month. The reason OP wants to scrape doesnt matter. His question was reasonable to ask here and could lead to finding out about some open source projects. You have not added anything to the conversation besides being wrong about the price.
        deleted 2y ago
        [deleted]
    - matly 2y ago
      Let's expand our own skillset by investing time instead of money (paying someone else). Sounds like a reasonable proposition to me.
netruk44 2y ago
I wrote a simple scraper for a 'steam game semantic search' app I built a while ago. It definitely won't fetch all the data that this person does though. It only fetches the current list of games on Steam, their store page information and some reviews for the game. The code quality probably isn't amazing, but it might give you an idea of how to get started with your own scraper. https://github.com/Netruk44/steam-embedding-search/blob/main/01_gamedataset/steamapi.py https://github.com/Netruk44/steam-embedding-search/blob/main...
- ddxv 2y ago
  Thanks! That's perfect, just want somewhere to get started.
DrammBA 2y ago
https://steamdb.info/faq/#how-are-we-getting-this-information https://steamdb.info/faq/#how-are-we-getting-this-informatio... I found this explanation from steamdb that points to the various projects and libraries they use to gather all the data they have. It's not a how-to, but it has very useful info.

Hi, I'm interested in scraping steam too. Do you have the scraper code available open source or one you recommend?

by ddxv 2y ago

Hi, I'm interested in scraping steam too. Do you have the scraper code available open source or one you recommend?

Have you looked over the data that OP is providing here and determined that it doesn't meet your needs? Generally it's polite to avoid scraping if you can help it, so I'd start by considering whether OP is already providing what you are looking for.

DrammBA 2y ago
On the other hand you need to be a paid member to download the raw scraped data, so it isn't unreasonable to want to learn how to scrape it instead.
- schnebbau 2y ago
  Good idea, let's save $5 and sink dozens of hours into building our own instead
  - DrammBA 2y ago
    Isn't that the hacker spirit, wanting to put things together yourself? Let's revisit the comment that started this thread: > Hi, I'm interested in scraping steam too.
    - schnebbau 2y ago
      I was responding directly to your objection with having to pay for it as the reason to do this, not hacking for the sake of hacking.
      - diggan 2y ago
        Hacking for the sake of hacking VS hacking for the sake of saving money, why does it matter?
        schnebbau 2y ago
        Jesus Christ, it's 5 bucks. 5 bucks is not a reason to roll your own version. Roll your own version because you want to roll your own version, not to save 5 bucks.
        DrammBA 2y ago
        you seem to be really hung up on the 5 bucks while at the same time being angry that people are hung up on the 5 bucks, it's just 5 bucks man, if people want to pay it or not doesn't matter, let people hack it away if they want
        deleted 2y ago
        [deleted]
        nickthegreek 2y ago
        Its not $5. It is free or $5/month. The reason OP wants to scrape doesnt matter. His question was reasonable to ask here and could lead to finding out about some open source projects. You have not added anything to the conversation besides being wrong about the price.
      - deleted 2y ago
        [deleted]
  - matly 2y ago
    Let's expand our own skillset by investing time instead of money (paying someone else). Sounds like a reasonable proposition to me.

netruk44 2y ago

I wrote a simple scraper for a 'steam game semantic search' app I built a while ago. It definitely won't fetch all the data that this person does though. It only fetches the current list of games on Steam, their store page information and some reviews for the game. The code quality probably isn't amazing, but it might give you an idea of how to get started with your own scraper. https://github.com/Netruk44/steam-embedding-search/blob/main/01_gamedataset/steamapi.py https://github.com/Netruk44/steam-embedding-search/blob/main...

ddxv 2y ago
Thanks! That's perfect, just want somewhere to get started.

DrammBA 2y ago

https://steamdb.info/faq/#how-are-we-getting-this-information https://steamdb.info/faq/#how-are-we-getting-this-informatio... I found this explanation from steamdb that points to the various projects and libraries they use to gather all the data they have. It's not a how-to, but it has very useful info.