3 ms·

Google Play Store in Numbers. Open Source Crawler for Mobile Apps Data

39 points by marcellolins 12y ago

lern_too_spel 12y ago
The robots.txt forbids any user agent from crawling the pages this crawler crawls. It's unwise to do this in the open.
- chatmasta 12y ago
  He's not exposing his personal server IP addresses. He's doing it in private, and opensourcing the code for others to do the same. robots.txt are laughably unenforced policies, and are disrespected every day.
  - smilliken 12y ago
    In practice robots.txt is just a suggestion; even Google doesn't fully respect it.
    - lern_too_spel 12y ago
      Google specifically recommended using robots.txt in Copiepresse v. Google. I haven't noticed them disobeying robots.txt on my own sites.
      - smilliken 12y ago
        I've seen several blog posts on exceptional cases before, but the only case I've personally witnessed is how the crawl-delay directive is ignored. Granted you can set it if you're a webmaster tools user, but even then the setting disables itself after a few months.
    - _lce0 12y ago
      robots.txt is a nice way to make spiders ignore to index content, but doesn't imply not to access those
      - lern_too_spel 12y ago
        That is not how robots.txt has been interpreted by any search engine. Robots.txt tells them not to access certain pages and says nothing about whether they should index those pages. (It's possible to index a page based solely on what links to it.) If you want your content crawled but not indexed, you should use the robots meta tag with a value of NOINDEX.
smilliken 12y ago
Shameless self-promotion: MixRank's API has all of this and much more available, including Appstore. Email in profile if you're interested.
- marcellolins 12y ago
  Its a shame to think people seeing this as a "self promotion", when its not. What i am promoting is the data that this crawler obtained, and all the insights people can get for free, just by using it. As opposed to an API, that will limit either the access for the data or the way you can access it, acessing the raw data on a database directly has no limits. Anyway, thanks for the reference, i will make sure to add it to the project's wiki as a related link.
  - notatoad 12y ago
    i don't think he's accusing you of self promotion, he's calling out his own recommendation for mixpanel as being self-promotion. Smilliken is "co-founder of mixrank" according to his profile.
    - marcellolins 12y ago
      Woah, now i feel bad. Sorry about that Smiliken : (
  - smilliken 12y ago
    I don't have time to review the code right now, but feel free to reach out with any questions (email in profile). Thanks for open-sourcing this! Link to our api docs: https://mixrank.com/api/documentation#/playstore/apps/{pname} https://mixrank.com/api/documentation#/playstore/apps/{pname...
- d0ugie 12y ago
  Okay... So, having been down this road and beyond already, what do you think of this guy's approach which is nice and visible for inspection? Might help not only him but some of us learn further, us tinkerers, those who might have a faint curiosity in c# etc if you shared. Then you'd be able to plug away shame-free.
inthewoods 12y ago
Applause Analytics (www.applause.com/mobile-analytics) provides some of this data plus data on the iTunes app store.
- marcellolins 12y ago
  TY for the reference. I know that apps annie also provide some really cool analytics. Regarding the itunes data, i also wrote a crawler for that, which is on github aswell

39 points by marcellolins 12y ago

lern_too_spel 12y ago

The robots.txt forbids any user agent from crawling the pages this crawler crawls. It's unwise to do this in the open.

chatmasta 12y ago
He's not exposing his personal server IP addresses. He's doing it in private, and opensourcing the code for others to do the same. robots.txt are laughably unenforced policies, and are disrespected every day.
- smilliken 12y ago
  In practice robots.txt is just a suggestion; even Google doesn't fully respect it.
  - lern_too_spel 12y ago
    Google specifically recommended using robots.txt in Copiepresse v. Google. I haven't noticed them disobeying robots.txt on my own sites.
    - smilliken 12y ago
      I've seen several blog posts on exceptional cases before, but the only case I've personally witnessed is how the crawl-delay directive is ignored. Granted you can set it if you're a webmaster tools user, but even then the setting disables itself after a few months.
  - _lce0 12y ago
    robots.txt is a nice way to make spiders ignore to index content, but doesn't imply not to access those
    - lern_too_spel 12y ago
      That is not how robots.txt has been interpreted by any search engine. Robots.txt tells them not to access certain pages and says nothing about whether they should index those pages. (It's possible to index a page based solely on what links to it.) If you want your content crawled but not indexed, you should use the robots meta tag with a value of NOINDEX.

smilliken 12y ago

Shameless self-promotion: MixRank's API has all of this and much more available, including Appstore. Email in profile if you're interested.

marcellolins 12y ago
Its a shame to think people seeing this as a "self promotion", when its not. What i am promoting is the data that this crawler obtained, and all the insights people can get for free, just by using it. As opposed to an API, that will limit either the access for the data or the way you can access it, acessing the raw data on a database directly has no limits. Anyway, thanks for the reference, i will make sure to add it to the project's wiki as a related link.
- notatoad 12y ago
  i don't think he's accusing you of self promotion, he's calling out his own recommendation for mixpanel as being self-promotion. Smilliken is "co-founder of mixrank" according to his profile.
  - marcellolins 12y ago
    Woah, now i feel bad. Sorry about that Smiliken : (
- smilliken 12y ago
  I don't have time to review the code right now, but feel free to reach out with any questions (email in profile). Thanks for open-sourcing this! Link to our api docs: https://mixrank.com/api/documentation#/playstore/apps/{pname} https://mixrank.com/api/documentation#/playstore/apps/{pname...
d0ugie 12y ago
Okay... So, having been down this road and beyond already, what do you think of this guy's approach which is nice and visible for inspection? Might help not only him but some of us learn further, us tinkerers, those who might have a faint curiosity in c# etc if you shared. Then you'd be able to plug away shame-free.

inthewoods 12y ago

Applause Analytics (www.applause.com/mobile-analytics) provides some of this data plus data on the iTunes app store.

marcellolins 12y ago
TY for the reference. I know that apps annie also provide some really cool analytics. Regarding the itunes data, i also wrote a crawler for that, which is on github aswell