8 ms·

Google Play doesn't have an API. So I built one.

52 points by jonursenbach 14y ago

charliesome 14y ago
https://github.com/pastfuture/MarketBot/blob/master/App/Android/AmazonAppstoreApp.php https://github.com/pastfuture/MarketBot/blob/master/App/Andr... \PastFuture\MarketBot\App\Android\AmazonAppstoreApp Welcome to Modern PHP.
TazeTSchnitzel 14y ago
If you want to add iTunes, here's an article on how to get the iTunes Store HTML from PHP: http://hactheplanet.com/blog/5 http://hactheplanet.com/blog/5
- jonursenbach 14y ago
  Yeah we're currently using their JSON API, it just hasn't been migrated over to this library yet.
waitwhatwhoa 14y ago
Do you run into any query limits with aggressive scraping? A colleague wrote an android market scraper a year or two back and she was repeatedly stymied by limits on the number of queries she could make to the store before getting cut off.
- jonursenbach 14y ago
  We don't do any aggressive scraping at the moment, so I can't really answer to that. The way our backend stuff works is that when we want to add an app to the site (http://gdgt.com/best/apps http://gdgt.com/best/apps), we query for the search results, and then grab the app we're looking for.
kanzure 14y ago
I see, it's based off of: http://github.com/chadrem/market_bot http://github.com/chadrem/market_bot Edit: I would like to point out that Google Play actually does have an API. It uses protocol buffers, if you check the Google Play apk. http://github.com/kanzure/android-market-api-py http://github.com/kanzure/android-market-api-py
- jonursenbach 14y ago
  It's not official though, and the results you get back from it are incredibly varied.
  - kanzure 14y ago
    > and the results you get back from it are incredibly varied. Can you show me? I haven't noticed.
    - jonursenbach 14y ago
      I don't have any examples anymore, because I deleted all the code I had written using the protocol buffer API when I decided to start writing MarketBot. Searching for something as simple as "foursquare" would never return the Foursquare app back and the results I was getting were never the same between requests.
est 14y ago
Oh god parsing HTML pages. Why don't Google Play provide JSON API like iTunes do? http://itunes.apple.com/lookup?id=343200656 http://itunes.apple.com/lookup?id=343200656 and alternatively http://itunes.apple.com/lookup?bundleId=com.clickgamer.AngryBirds http://itunes.apple.com/lookup?bundleId=com.clickgamer.Angry... The second link can be used in client app to self-checking for new version updates.
- Evbn 14y ago
  There are tons of Android Market website clones. How do they get their data?
  - est 14y ago
    apps are likely to be published on G Play first, then to 3rd party markets. If an app is exclusively published on 3rd pary market, then it's likely distributed directly through apk , there's no way to identify them using a centralized way.
    - kanzure 14y ago
      > then it's likely distributed directly through apk , there's no way to identify them using a centralized way. You can identify them via md5 hash of the apk, the "versionName" attribute and the "versionCode" attribute.
jeffehobbs 14y ago
Bon fortuna!
gailees 14y ago
Like a BOSS. How about the Apple Store?
- SwaroopH 14y ago
  Look above [1], they already have a JSON output for apps. [1] http://news.ycombinator.com/item?id=4634634 http://news.ycombinator.com/item?id=4634634
krob 14y ago
I did something like this, but using xpath queries, only problem is when they decide to change these, you will have "fun" times retesting all the dom locations to fix your scraper when the time does come when it breaks.
- kanzure 14y ago
  > I did something like this, but using xpath queries, only problem is when they decide to change these That's a strong reason for going with a library that is used by multiple users (more people to distribute the workload over). But more realistically, you can just use the existing API endpoints because there are client apps in the wild that rely on those APIs working the same way-- at least until Google sunsets Google Play.
  - krob 14y ago
    there was an API which i found a while back, but it didn't work at all, all the unit tests it had failed, so my guess was that google changed their api once again and the project was worthless. The solution i came up with was the only reliable one i could build. It was only like 6 lines of code.
aphidbyteinc 14y ago
Nice.

52 points by jonursenbach 14y ago

https://github.com/pastfuture/MarketBot/blob/master/App/Android/AmazonAppstoreApp.php https://github.com/pastfuture/MarketBot/blob/master/App/Andr... \PastFuture\MarketBot\App\Android\AmazonAppstoreApp Welcome to Modern PHP.

TazeTSchnitzel 14y ago

If you want to add iTunes, here's an article on how to get the iTunes Store HTML from PHP: http://hactheplanet.com/blog/5 http://hactheplanet.com/blog/5

jonursenbach 14y ago
Yeah we're currently using their JSON API, it just hasn't been migrated over to this library yet.

waitwhatwhoa 14y ago

Do you run into any query limits with aggressive scraping? A colleague wrote an android market scraper a year or two back and she was repeatedly stymied by limits on the number of queries she could make to the store before getting cut off.

jonursenbach 14y ago
We don't do any aggressive scraping at the moment, so I can't really answer to that. The way our backend stuff works is that when we want to add an app to the site (http://gdgt.com/best/apps http://gdgt.com/best/apps), we query for the search results, and then grab the app we're looking for.

kanzure 14y ago

I see, it's based off of: http://github.com/chadrem/market_bot http://github.com/chadrem/market_bot Edit: I would like to point out that Google Play actually does have an API. It uses protocol buffers, if you check the Google Play apk. http://github.com/kanzure/android-market-api-py http://github.com/kanzure/android-market-api-py

jonursenbach 14y ago
It's not official though, and the results you get back from it are incredibly varied.
- kanzure 14y ago
  > and the results you get back from it are incredibly varied. Can you show me? I haven't noticed.
  - jonursenbach 14y ago
    I don't have any examples anymore, because I deleted all the code I had written using the protocol buffer API when I decided to start writing MarketBot. Searching for something as simple as "foursquare" would never return the Foursquare app back and the results I was getting were never the same between requests.

est 14y ago

Oh god parsing HTML pages. Why don't Google Play provide JSON API like iTunes do? http://itunes.apple.com/lookup?id=343200656 http://itunes.apple.com/lookup?id=343200656 and alternatively http://itunes.apple.com/lookup?bundleId=com.clickgamer.AngryBirds http://itunes.apple.com/lookup?bundleId=com.clickgamer.Angry... The second link can be used in client app to self-checking for new version updates.

Evbn 14y ago
There are tons of Android Market website clones. How do they get their data?
- est 14y ago
  apps are likely to be published on G Play first, then to 3rd party markets. If an app is exclusively published on 3rd pary market, then it's likely distributed directly through apk , there's no way to identify them using a centralized way.
  - kanzure 14y ago
    > then it's likely distributed directly through apk , there's no way to identify them using a centralized way. You can identify them via md5 hash of the apk, the "versionName" attribute and the "versionCode" attribute.

jeffehobbs 14y ago

Bon fortuna!

gailees 14y ago

Like a BOSS. How about the Apple Store?

SwaroopH 14y ago
Look above [1], they already have a JSON output for apps. [1] http://news.ycombinator.com/item?id=4634634 http://news.ycombinator.com/item?id=4634634

krob 14y ago

I did something like this, but using xpath queries, only problem is when they decide to change these, you will have "fun" times retesting all the dom locations to fix your scraper when the time does come when it breaks.

kanzure 14y ago
> I did something like this, but using xpath queries, only problem is when they decide to change these That's a strong reason for going with a library that is used by multiple users (more people to distribute the workload over). But more realistically, you can just use the existing API endpoints because there are client apps in the wild that rely on those APIs working the same way-- at least until Google sunsets Google Play.
- krob 14y ago
  there was an API which i found a while back, but it didn't work at all, all the unit tests it had failed, so my guess was that google changed their api once again and the project was worthless. The solution i came up with was the only reliable one i could build. It was only like 6 lines of code.

aphidbyteinc 14y ago

Nice.