4 ms·

My own Rap Genius API

28 points by timrogers 13y ago

georgespencer 13y ago
Nice work!
- timrogers 13y ago
  Thanks George - I appreciate it.
dewitt 13y ago
I appreciate that you want access to their content, and I may personally agree they ought to build an API. But isn't it up to them to decide whether or not to do so? Otherwise isn't it just stealing? (Fully expecting this to be a controversial comment, but I'm curious what the prevailing sentiments on scraping without permission are these days. Good, because all content should be free, especially crowd-sourced content? Or bad, because the contributors didn't agree to such third-party usage, and it could limit Rap Genius' ability to build a business?)
- timrogers 13y ago
  OP here. It's a fair point. I'd love to hear the thoughts of other HNers. I tend to think that this kind of thing is okay for personal use, whereas it wouldn't be en masse - for instance, if you were using it to power and advantage a competing music site. Others' opinions may differ. For instance, I've previously built scrapers for banking services (American Express [https://github.com/timrogers/amex https://github.com/timrogers/amex] and Lloyds TSB [https://github.com/timrogers/lloydstsb] https://github.com/timrogers/lloydstsb]) and the UK's university application system, UCAS (https://github.com/timrogers/ucas https://github.com/timrogers/ucas) and none of these organisations seem to have had any problem with it, but the library has been really gratefully received by users.
  - xtrumanx 13y ago
    Any advice on how to build the API if a banking service uses 2-factor auth? Perhaps SMSing the secret number they send you back to your system (assuming you can't give your system direct access to phone number your the secret number will arrive at)?
    - timrogers 13y ago
      Interesting question - I haven't had to try it yet. My scrapers are all read-only at the moment, and (afaik) none of the UK banks require two-factor in any form for that kind of operation. Potentially using Twilio (http://www.twilio.com http://www.twilio.com) you might be able to have an application which receives the two-factor request itself and handles it. It'd make the process async though, which wouldn't be ideal.
  - dewitt 13y ago
    Thoughtful reply, I really appreciate that. Thank you. And don't get me wrong—I love APIs, I build them for a living. And I know how much users love them. Unfortunately, not all companies love providing them, or have the economics of doing so fully worked out.
    - timrogers 13y ago
      No problem - great to engage with people's opinions. HN has such a wealth of knowledge. You might find my response below (https://news.ycombinator.com/item?id=6229617 https://news.ycombinator.com/item?id=6229617) interesting, on that front. My guess is that for Rap Genius, the only API that will make real sense is one offered on a commercial basis.
  - xSwag 13y ago
    As someone applying to university this year, I can't thank you enough for the ucas scraper!
    - timrogers 13y ago
      Hah - glad you like it. Did you find it before, or is this the first time you've spotted it? I built a cool frontend too that you might be able to cannibalise if you'd like to share your status online - see https://github.com/timrogers/ucas-frontend https://github.com/timrogers/ucas-frontend. If you wanna chat at all about any of this stuff, drop me a line at me@timrogers.co.uk.
- deleted 13y ago
  [deleted]
- simon_weber 13y ago
  I see non-sanctioned access like this as a huge boon to both the service and its users: it enables easy marketing, community-supported apps, easy research on what features users want, etc. I also think the fear of abuse is mostly unwarranted. If somebody wanted to steal your content or abuse your platform, they'd figure out a way to do so -- with or without an unofficial api (especially if all that's needed is http scraping). (disclaimer: I maintain https://github.com/simon-weber/Unofficial-Google-Music-API https://github.com/simon-weber/Unofficial-Google-Music-API)
  - timrogers 13y ago
    OP here. I agree - for me, the only exception is where data is being abused for overtly commercial purposes. This obviously isn't one of those cases, and I'm doubtful as to whether unofficial APIs (i.e. screen scraping) can ever work properly for that kind of scale.
    - simon_weber 13y ago
      Right, of course. It can get really hard to tell where the line is, though. For example, multiple for-pay third-party clients for Google Music now exist [0]. Personally, I think this is still a value-add for Google, but I could understand why some would think it's exploitative. Another thought experiment: say the unofficial use of the platform originally required a fair amount of reverse engineering. Can the person who figured it out be held responsible for other's abusive behavior? [0] https://itunes.apple.com/us/app/gomusic/id457883228 https://itunes.apple.com/us/app/gomusic/id457883228
      - timrogers 13y ago
        Very interesting point - I'm not sure how best to judge what is exploitative and what isn't. I guess it becomes exploitative if it serious subverts in some way the business model of the original data/service provider. But then I'm not sure still...maybe that is okay if consumers massively benefit from the reverse engineering, and thus overall utility is increased...! What do you think?
why-el 13y ago
I would assume they are at least thinking about an API, as it's fairly easy to construct/what they are doing is easily portable elsewhere (Think Medium like comments all over the place). With this said, nice work.
- timrogers 13y ago
  Thanks. I'd imagine an API is on the cards, but I could be wrong. As far as I can see, beyond the website, their data has two markets: - Embedding in personal blogs - this is probably best achieved by a simple embed code (which I'm planning to use my gem to build), rather than a full API - Commercial use - for instance, Spotify (or competitor) might like to display rich lyrics in-app. The situations which would demand a full-on API, I suspect, would be commercial use. Thus, I'd imagine that any API they build will be private and commercial, providing a (first?) revenue stream.
kenshiro_o 13y ago
Great stuff! I honestly find it shameful that they don't have an api. A couple of months ago I built one in Node.js (https://github.com/kenshiro-o/RapGenius-JS https://github.com/kenshiro-o/RapGenius-JS). If anyone is interested in improving/using the project let me know!
- timrogers 13y ago
  I actually saw your library, it's super cool - it's currently pretty much the top result for "Rap Genius API", apart from the tweet I refer to (https://twitter.com/RapGenius/statuses/245057326321655808 https://twitter.com/RapGenius/statuses/245057326321655808). As I've said in other comments, I'd love RG to have a public API, but I can see why they don't. It'd be cool to compare notes on parsing Rap Genius. Drop me an email if you have a moment, me@timrogers.co.uk.
  - kenshiro_o 13y ago
    Sure! I'll send you a mail tomorrow when I return to London. Sharing ideas on how to improve parsing is a great idea!
    - timrogers 13y ago
      Awesome! Speak to you soon hopefully!

28 points by timrogers 13y ago

georgespencer 13y ago

Nice work!

timrogers 13y ago
Thanks George - I appreciate it.

dewitt 13y ago

I appreciate that you want access to their content, and I may personally agree they ought to build an API. But isn't it up to them to decide whether or not to do so? Otherwise isn't it just stealing? (Fully expecting this to be a controversial comment, but I'm curious what the prevailing sentiments on scraping without permission are these days. Good, because all content should be free, especially crowd-sourced content? Or bad, because the contributors didn't agree to such third-party usage, and it could limit Rap Genius' ability to build a business?)

timrogers 13y ago
OP here. It's a fair point. I'd love to hear the thoughts of other HNers. I tend to think that this kind of thing is okay for personal use, whereas it wouldn't be en masse - for instance, if you were using it to power and advantage a competing music site. Others' opinions may differ. For instance, I've previously built scrapers for banking services (American Express [https://github.com/timrogers/amex https://github.com/timrogers/amex] and Lloyds TSB [https://github.com/timrogers/lloydstsb] https://github.com/timrogers/lloydstsb]) and the UK's university application system, UCAS (https://github.com/timrogers/ucas https://github.com/timrogers/ucas) and none of these organisations seem to have had any problem with it, but the library has been really gratefully received by users.
- xtrumanx 13y ago
  Any advice on how to build the API if a banking service uses 2-factor auth? Perhaps SMSing the secret number they send you back to your system (assuming you can't give your system direct access to phone number your the secret number will arrive at)?
  - timrogers 13y ago
    Interesting question - I haven't had to try it yet. My scrapers are all read-only at the moment, and (afaik) none of the UK banks require two-factor in any form for that kind of operation. Potentially using Twilio (http://www.twilio.com http://www.twilio.com) you might be able to have an application which receives the two-factor request itself and handles it. It'd make the process async though, which wouldn't be ideal.
- dewitt 13y ago
  Thoughtful reply, I really appreciate that. Thank you. And don't get me wrong—I love APIs, I build them for a living. And I know how much users love them. Unfortunately, not all companies love providing them, or have the economics of doing so fully worked out.
  - timrogers 13y ago
    No problem - great to engage with people's opinions. HN has such a wealth of knowledge. You might find my response below (https://news.ycombinator.com/item?id=6229617 https://news.ycombinator.com/item?id=6229617) interesting, on that front. My guess is that for Rap Genius, the only API that will make real sense is one offered on a commercial basis.
- xSwag 13y ago
  As someone applying to university this year, I can't thank you enough for the ucas scraper!
  - timrogers 13y ago
    Hah - glad you like it. Did you find it before, or is this the first time you've spotted it? I built a cool frontend too that you might be able to cannibalise if you'd like to share your status online - see https://github.com/timrogers/ucas-frontend https://github.com/timrogers/ucas-frontend. If you wanna chat at all about any of this stuff, drop me a line at me@timrogers.co.uk.
deleted 13y ago
[deleted]
simon_weber 13y ago
I see non-sanctioned access like this as a huge boon to both the service and its users: it enables easy marketing, community-supported apps, easy research on what features users want, etc. I also think the fear of abuse is mostly unwarranted. If somebody wanted to steal your content or abuse your platform, they'd figure out a way to do so -- with or without an unofficial api (especially if all that's needed is http scraping). (disclaimer: I maintain https://github.com/simon-weber/Unofficial-Google-Music-API https://github.com/simon-weber/Unofficial-Google-Music-API)
- timrogers 13y ago
  OP here. I agree - for me, the only exception is where data is being abused for overtly commercial purposes. This obviously isn't one of those cases, and I'm doubtful as to whether unofficial APIs (i.e. screen scraping) can ever work properly for that kind of scale.
  - simon_weber 13y ago
    Right, of course. It can get really hard to tell where the line is, though. For example, multiple for-pay third-party clients for Google Music now exist [0]. Personally, I think this is still a value-add for Google, but I could understand why some would think it's exploitative. Another thought experiment: say the unofficial use of the platform originally required a fair amount of reverse engineering. Can the person who figured it out be held responsible for other's abusive behavior? [0] https://itunes.apple.com/us/app/gomusic/id457883228 https://itunes.apple.com/us/app/gomusic/id457883228
    - timrogers 13y ago
      Very interesting point - I'm not sure how best to judge what is exploitative and what isn't. I guess it becomes exploitative if it serious subverts in some way the business model of the original data/service provider. But then I'm not sure still...maybe that is okay if consumers massively benefit from the reverse engineering, and thus overall utility is increased...! What do you think?

why-el 13y ago

I would assume they are at least thinking about an API, as it's fairly easy to construct/what they are doing is easily portable elsewhere (Think Medium like comments all over the place). With this said, nice work.

timrogers 13y ago
Thanks. I'd imagine an API is on the cards, but I could be wrong. As far as I can see, beyond the website, their data has two markets: - Embedding in personal blogs - this is probably best achieved by a simple embed code (which I'm planning to use my gem to build), rather than a full API - Commercial use - for instance, Spotify (or competitor) might like to display rich lyrics in-app. The situations which would demand a full-on API, I suspect, would be commercial use. Thus, I'd imagine that any API they build will be private and commercial, providing a (first?) revenue stream.

kenshiro_o 13y ago

Great stuff! I honestly find it shameful that they don't have an api. A couple of months ago I built one in Node.js (https://github.com/kenshiro-o/RapGenius-JS https://github.com/kenshiro-o/RapGenius-JS). If anyone is interested in improving/using the project let me know!

timrogers 13y ago
I actually saw your library, it's super cool - it's currently pretty much the top result for "Rap Genius API", apart from the tweet I refer to (https://twitter.com/RapGenius/statuses/245057326321655808 https://twitter.com/RapGenius/statuses/245057326321655808). As I've said in other comments, I'd love RG to have a public API, but I can see why they don't. It'd be cool to compare notes on parsing Rap Genius. Drop me an email if you have a moment, me@timrogers.co.uk.
- kenshiro_o 13y ago
  Sure! I'll send you a mail tomorrow when I return to London. Sharing ideas on how to improve parsing is a great idea!
  - timrogers 13y ago
    Awesome! Speak to you soon hopefully!