19 ms·

Archiveteam are backing up SoundCloud

338 points by hunglee2 9y ago

naturalgradient 9y ago
Archiveteam seems like a really cool project, what I was wondering (and couldn't find in the FAQ) is who is paying for all the storage? Is it donated by big tech companies?
- sandralopz56 9y ago
  I just got paid $6784 working off my laptop this month. And if you think that's cool, my divorced friend has twin toddlers and made over $9k her first month. It feels so good making so much money when other people have to work for so much less. This is what I do,•••>>http://bit.do/dnRs5 http://bit.do/dnRs5
- pas 9y ago
  Volunteers, no corps as far as I know. https://en.wikipedia.org/wiki/Archive_Team https://en.wikipedia.org/wiki/Archive_Team
  - rsync 9y ago
    rsync.net donated online storage for the original geocities backup efforts in 2009.
- danohu 9y ago
  A decent amount eventually ends up at archive.org [not all, because archive.ort needs to be a bit more careful, but they have a decent symbiotic relationship]
  - cJ0th 9y ago
    I love archive.org but they have to start becoming a) very careful indeed and b.) better at cleaning up the content that has been uploaded. By this time there is an insane amount of illegal content on it like ripped CDs and copies of books which still can be bought. I don't want them to go down because of this kind of stuff.
- nextlevelwizard 9y ago
  It was founded by Jason Scott (of textfiles.com) who works for the Internet Archive (which is nonprofit). So while I think the Archive Team is fully community driven project what they backup ends up on Internet Archive.
_pmf_ 9y ago
I don't know whether SoundCloud does not have a huge, Wikipediaesque donation banner on every page illustrating the severity of their financial situation; it's embarassing for them, but think of it like this: the artists have a right to know that the platform that manages their life's work needs their support.
- imartin2k 9y ago
  "I don't know whether SoundCloud does not have a huge, Wikipediaesque donation banner on every page illustrating the severity of their financial situation" Maybe because they are profit-oriented company that raised hundreds of million USD in venture capital. Asking for donations would be kinda unethical, and the founders would know that. However, if SoundCloud somehow would be transformed into a non-profit organization...
  - sk0g 9y ago
    So you're saying Richard Hendricks' New Internet should be used to keep Soundcloud afloat?
StavrosK 9y ago
Hmm, I've been playing with IPFS lately, and just had an idea: Since IPFS is perfect for archival, Archiveteam could put their files on IPFS, and users could help out by pinning stuff on their local nodes. For example, I could ask their website to give me a 10 GB list of files to pin (if I wanted to "donate" 10 GB to them), and I'd keep them available. The only problem is that I don't know whether IPFS has any way to gauge availability, so I'm not sure if the team could tell which files were only hosted by a few people.
- rakoo 9y ago
  Archiveteam has discussed about backing up the Internet Archive a long time ago in their project called INTERNETARCHIVE.BAK (http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK). They decided to go with git-annex because its creator was a cofounder of the ArchiveTeam and was willing to work with that project, who is still waiting for IPFS's proposal on how to do this (http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/ipfs_implementation http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/i...). You can participate in the effort of course. Have a few hundreds of GB and a good connection ? Head over to http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/git-annex_implementation http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/g... and follow the steps !
- cyphar 9y ago
  ArchiveTeam is putting the files in the Internet Archive archives. While IPFS is great, I'm not sure I agree it's good for archival because it depends on the availability of the IPFS network. The Internet Archive does work to make sure that there are sufficent backups to mean they can recreate their archive.
  - Tamara99 9y ago
    l juSт gот pålD $6784 шояκlng оff my låpтоp тнlS mоnтн. ånD lf yоu тнlnκ тнåт'S cооl, my DlvояcêD fяlênD нåS тшln тоDDlêяS ånD måDê оvêя $9κ нêя flяSт mоnтн. lт fêêlS Sо gооD måκlng Sо mucн mоnêy шнên отнêя pêоplê нåvê то шояκ fоя Sо mucн lêSS. тнlS lS шнåт l Dо,•••>>http://bit.do/dnRs5 http://bit.do/dnRs5
skeletonjelly 9y ago
> Resource Limit Is Reached > The website is temporarily unable to service your request as it exceeded resource limit. Please try again later. I suppose I prefer an archive over the blog being unavailable
- c8g 9y ago
  >I suppose I prefer an archive over the blog being unavailable http://web.archive.org/web/20170717083540/http://archiveteam.org/index.php?title=SoundCloud&utm_content=bufferb94ff&utm_medium=social&utm_source=linkedin.com&utm_campaign=buffer http://web.archive.org/web/20170717083540/http://archiveteam... it is interesting that their blog is not good enough for archive. link above and below are different (first one is updated but second one is not, because link contains some extra info). they will be able to save a lot of space by finding duplicate. http://web.archive.org/web/20170606104512/http://archiveteam.org/index.php?title=SoundCloud http://web.archive.org/web/20170606104512/http://archiveteam...
chinathrow 9y ago
What's the current context here? Is SoundCloud going away anytime soon?
- imartin2k 9y ago
  That's at least the fear of many musicians, DJs and of the existing (but reportedly shrinking) user base after a recent round of layoffs and reports that SoundCloud's money will run out in Q4.
- vasanthv 9y ago
  Soundcloud CEO says no its not going away anytime soon. https://www.digitalmusicnews.com/2017/07/14/soundcloud-ceo-responds-shutdown/ https://www.digitalmusicnews.com/2017/07/14/soundcloud-ceo-r...
  - iagooar 9y ago
    Would they say it if they knew it, though?
imartin2k 9y ago
I hadn't heard of Archiveteam before, but the fact that HN brings their site down doesn't make me too confident in their backing up skills :)
- Xylakant 9y ago
  HN has been bringing down a few sites that were unprepared. True, it's possible to harden your page for unexpected traffic spikes, but that's always work and maybe you just decide it's not worth the effort and invest the time elsewhere. (whether that's a good decision is up for debate)
- jacquesm 9y ago
  HN can direct more traffic than you think, and it can do so very quickly. AT's backing up skills are fine, and if you've managed to be on HN for nearly 5 years without hearing about them then that does not make me too confident about your reading skills :)
  - imartin2k 9y ago
    fair point :) for me it's more this "first impression thing".
- chipperyman573 9y ago
  Backing up data then storing it to be infrequently read and handing a major spike of traffic are two very different things.
- dmacedo 9y ago
  You're thinking scalability. What they do is reliability in archives, and "team work" (it's in the name) for retrieval. Cheers
  - imartin2k 9y ago
    Mostly, I saw the humourous in it, which I tried to emphasize with the smilie. Didn't land.
ipsum2 9y ago
I'm interested in people's opinions on the legality of this. They mention "Archive Team considers the SoundCloud service in danger and, as it hosts a lot of original content, finds it important to prepare to save it selectively (a full grab would be too big and would raise concerns of mass copyright infringement).", but how is downloading any portion of artist's music not copyright infringement? I've written my own Soundcloud offline audio player, but didn't distribute it because it was against their TOS.
- sk0g 9y ago
  Maybe they're only pulling either indie stuff with no copyright, or the rarer stuff that they're less likely to get in trouble over.
  - icebraining 9y ago
    indie stuff with no copyright There's no such thing, copyright is automatic, it applies as soon as the author puts the work in some tangible medium (like an hard drive).
    - Bromskloss 9y ago
      I've never thought about the medium thing. If you write a song (and perhaps perform it in public and everything), but never commit it into writing or recording, does copyright not apply?
      - DanBC 9y ago
        What would the copyrights apply to if there's no recording?
        icebraining 9y ago
        Live concerts played from memory? E.g. Eubie Blake supposedly only wrote Charleston Rag down 16 years after composing it.
omarforgotpwd 9y ago
That's probably not going to help with their huge bandwidth costs
- anilakar 9y ago
  Ironically their web server is already over limit. >508 Resource Limit Is Reached
probably_wrong 9y ago
The more I think about this, the more convinced I am that Archiveteam are actually detrimental (in the long run) to the well-being of the Internet. Don't get me wrong, I appreciate the work they do, and without them lots of content would simply disappear. But solving this problem should be at the core of the protocol itself (Xanadu, anyone?[1]), not depending on the resources and goodwill of a single team. Just like IPv6, I don't think the problem will be solved as long as there's a patch that somehow works. [1] https://en.wikipedia.org/wiki/Project_Xanadu https://en.wikipedia.org/wiki/Project_Xanadu
- Buetol 9y ago
  I downvoted this. Maybe we will get to this point one day, but in the meantime I think we should all appreciate the amount of conservation work they do for the future generations. Exactly how NGOs help feed people hoping that one day the system will be fixed.
  - lucb1e 9y ago
    I upvoted this. Regardless of whether I agree, I think it is a reasonable question to ask / discussion to have, and the downvote button should be used for comments that don't help the conversation in any way.
- jacquesm 9y ago
  Apt username. Yes, you're wrong. See, we don't have Xanadu and we don't have an in-protocol solution so somebody needs to do the dirty work here and AT stepped up and does an absolutely incredible job. So as long as there's a patch that somehow works we have at least a solution. If that patch wouldn't be there we'd have nothing.
- robert_foss 9y ago
  The perfect solution is the enemy of the good. There is no perfect solution in sight and this a good one until a better one comes along.
- manigandham 9y ago
simonhfrost 9y ago
Anyone interested in backing up their own personal (public) SoundCloud files will find this tool useful: https://github.com/mafintosh/soundcloud-to-dat https://github.com/mafintosh/soundcloud-to-dat
- rapnie 9y ago
  Cool!! Based on the Dat project (http://datproject.org http://datproject.org). These guys might become a good alternative to IPFS, especially if they reposition themselves as a message-based system, instead of just file sharing.
rnhmjoj 9y ago
Someone has already backed it up: https://www.reddit.com/r/DataHoarder/comments/6n1pap/soundcloud_may_only_have_50_days_left_save_your/dk6pou7/ https://www.reddit.com/r/DataHoarder/comments/6n1pap/soundcl...
- ipsum2 9y ago
  This seems a bit ridiculous. 900TB costs $22000 in hard drives (assuming $100/4TB HDD), without any redundancy. I wonder what their storage solution is like.
  - evook 9y ago
    Almost certainly Google Drive unlimited.
    - spookyuser 9y ago
      I guess soundcloud is technically using Google Cloud now...
    - syshum 9y ago
      And this is how Google Drive Unlimited becomes Google Drive Limited... Like every other Unlimited service, because people like this abuse the shit out of it then when it inevitably dies they run around exclaiming "dont blame me brah they said it was unlimited...., shouldn't have called it unlimited"
      - nextlevelwizard 9y ago
        While this is clearly abusing intent behind the "unlimited" sales pitch, I still do not think it's in violation of it. There's also most likely more than just this guy owning copy of the files from SoundCloud, so once Google has copy of them all they can just cache it for the rest of the people. I.e. you don't need unique files for every user who has some file, instead you can just give out the one to anyone who requests it.
majortennis 9y ago
Aww I like soundcloud
philfrasty 9y ago
Just from a legal standpoint: isn't „...considers the SoundCloud service in danger...“ slander? Especially since the Internet Archive isn't a nobody (with maybe some inside information?). Just remembered cases like Deutsche Bank vs Leo Kirch which are legal nightmares.
- naasking 9y ago
  > Just from a legal standpoint: isn't „...considers the SoundCloud service in danger...“ slander? I think you mean libel, and I think it's a reasonable conclusion that everyone is thinking given that SoundCloud just laid off so many people.
- nextlevelwizard 9y ago
  Good luck suing not for profit volunteer group when you yourself are in financial trouble to begin with. I'm not saying it's impossible, but there's not much to gain.
  - deleted 9y ago
    [deleted]
- beardicus 9y ago
  Archive Team is not the Internet Archive, and the above quote is clearly opinion.
- db48x 9y ago
  Maybe in Germany it could be, but not in the US.
CharlesDodgson 9y ago
Does anyone feel that this is all a bit pointless. Like is there a greater social need to preserve SC? I found a lot of the content to ephemeral, things like podcasts or DJ mixes. I dunno, it just seems a bit silly to put resources on it.
- 69mlgsniperdad 9y ago
  Obviously, with any free hosting service, there is junk on it... However, if you check wikipedia or wherever, and find out what soundcloud is, you will understand. Soundcloud is a place for dj's, musicians, bands to upload their music. Many smaller bands, soundcloud is the only place many of their songs are stored. In fact many mainstream artists started there, whose 'garage band' recordings, and sets recorded before they went big. There is an abundance of original music, that might be otherwise lost.
- nextlevelwizard 9y ago
  Someone might have poured all of their creativity to content on that platform and to loosely quote Jason Scott "this might be the largest audience this specific line of genes have/will ever reach". Is that not worth preserving? Even if it just takes a thumb drive worth of space? Even if the content sucks, it still part of our history as a species. Think of some of the old works/diaries/letters that are now thought to hold value. Like I've been lately reading "letter from Seneca". Why are they any more worthy of preservation than something on SoundCloud?
  - CharlesDodgson 9y ago
    i guess it's subjective, some people would think a bank of Vapourwave music on SC is important. I'm pretty happy to have a 128kbp version of blink 182 albums on my backup, because they mean something to me. I'm just really questioning the need to archive for the sake of archiving.
- wambotron 9y ago
  I have my own personal recordings up there (although they're also on bandcamp and I still have them backed up). I know of a lot of people who use it to host their music. It's not just podcasts and remixes.
cetra3 9y ago
I've got a tool to grab all of the songs from your feed. I use this to offline sync mixes (not individual songs). https://github.com/cetra3/rustcloud https://github.com/cetra3/rustcloud It would be an absolute shame if Soundcloud disappears. There has been so much music I have discovered on this service.
chazzeromus 9y ago
Probably not an amazing and erudite source as others have posted but there's this tweet: https://twitter.com/chancetherapper/status/885921220759355397 https://twitter.com/chancetherapper/status/88592122075935539... Soundcloud is here to stay
- naturalgradient 9y ago
  I dont understand. Is this supposed to imply this rapper will somehow invest and personally try to ensure soundcloud will survive? Or did he call the CEO, the CEO told him 'Yes we will stay'? I mean what is he supposed to say? 'No, don't bother uploading new songs, we will run out of cash soon, thanks for the call'?
  - chazzeromus 9y ago
    Hm that's a good point. I didn't consider the cynical approach to a CEO's reassurance, as obvious as it seems.
Steeeve 9y ago
If these kind of entities actually want to preserve resources, they shouldn't be generating a petabyte of bandwidth charges. Contact soundcloud and come to an agreement that will be responsible.
- beardicus 9y ago
  Archiveteam exists mostly because your suggested course of action rarely works.
  - Steeeve 9y ago
    If they can't convince soundcloud it's a positive, they should not be doing it.
    - Whitestrake 9y ago
      I don't think SoundCloud is an entity with some optimistic face you could approach and convince through reason or merit. I expect they, like any business, would take a stance primarily based on liability, cost, benefit, etc... overriding how anyone working at the company actually feels about it. It costs them nothing to ignore/outright reject without consideration a crazy proposal like "lets give our entire database of content to a third party". Assessing the technical and legal ramifications of that proposal costs time/effort/money. Why bother? If SoundCloud gives their content to Archive Team, they're shouldering the possibility of some kind of liability, surely. If they say nothing, let Archive Team take it themselves, from their website, they let Archive Team (who understand and are willing to) take responsibility for that.
- avian 9y ago
  I imagine a company with its funds running out will not be particularly interested in dedicating resources to the archive team. Not to mention that doing that would be kind of signing their own death certificate. It's hard to argue that you are "on path to profitability" when you're busy handing off your assets to a museum. How much does a petabyte of bandwidth go for these days anyway? It might as well be cheaper than paying engineering, legal and management to arrange for some kind of off-line data transfer.
Sami_Lehtinen 9y ago
Did anyone backup GrooveShark? I had some unique pieces stored there which seem to be lost forever.
streamspeek 9y ago
I'd recommend transcoding to 96kb/s VBR Ogg Opus during crawling.
- mynewtb 9y ago
  The goal of archiving is making a perfect copy. Transcoding ruins that.
  - fiatjaf 9y ago
    That's not the goal of archiving.
  - colejohnson66 9y ago
    Not relevant in this situation, but to be fair, archiving analog media isn’t about making a perfect copy, just a good one. Also, when old DRM encumbered games (StarForce, etc.) are archived, the DRM is removed first. So not a perfect copy there either, but just an as good as you can get copy.
    - mynewtb 9y ago
      as perfect as possible then, no need to argue about the 'perfect' phrasing. It's clear what I meant in the context. Both your example require transformation, while archiving unprotected digital media is as easy as making a copy.
- voltagex_ 9y ago
  Why? You increase the CPU requirements of the crawl, lose quality from an already low bitrate file and then have to QA the new encode. All for a very small decrease in filesize.
random_calc 9y ago
How would you find all files to download without scraping all Soundcloud pages?
radarsat1 9y ago
Their website seems to be down: I'm just wondering, are they downloading everything accessible through the player, or just songs marked "Download"? Even given that they could restrict themselves to songs marked okay to download, how much of that will be DJ mixes containing copyrighted songs? I'm just wondering because Soundcloud actually has support to specify your copyright terms, which does not default to "everyone can download this", so it's an interesting case.. The website works now, it just says "selective content" I'm sure SC serves up a lot of content per day, but how do you think they will react by suddenly having someone download all of their 900 TB or whatever it is in one day? How much will Archiveteam be contributing to SC's downfall by suddenly causing them a huge unexpected bill? As someone who really wants the SC content backed up properly, I nonetheless see how this raises some interesting legal issues.
- Qub3d 9y ago
  I can only access it intermittently. Here is the google cache: http://webcache.googleusercontent.com/search?q=cache:eCl5VSBt62EJ:www.archiveteam.org/index.php%3Ftitle%3DSoundCloud+&cd=1&hl=en&ct=clnk&gl=us http://webcache.googleusercontent.com/search?q=cache:eCl5VSB... In response to your concerns; I think ArchiveTeam simply doesn't care. They are very firm in their convictions, and they don't exactly listen to requests to not archive things.[0] If you're curious, here[1] is the initial discussion that AT had. People bring up copyright concerns. [0]:http://webcache.googleusercontent.com/search?q=cache:fmhU2zS7QMwJ:www.archiveteam.org/index.php%3Ftitle%3DRobots.txt+&cd=1&hl=en&ct=clnk&gl=us http://webcache.googleusercontent.com/search?q=cache:fmhU2zS... [1]:http://archive.fart.website/bin/irclogger_log/archiveteam?date=2015-08-27,Thu&sel=180#l176 http://archive.fart.website/bin/irclogger_log/archiveteam?da...
- cyphar 9y ago
  I think in most cases ArchiveTeam's actions have been copyright-infringement on some level. They just don't care and find that keeping user content safe from unrepentant deletion to be more important. If you've ever seen a Jason Scott talk, he isn't the sort of guy who gives a shit if you DMCA him while he's sucking up all of your bandwidth archiving your content two days before your servers shut down.
eatbitseveryday 9y ago
How can tracks be downloaded when many of them are pay-to-access? or stream-only?
- voltagex_ 9y ago
  Generally, if you "stream" (send) me bits, they're mine to do what I want with (practically, maybe not legally). Although, if anyone could tell me how to do that with AES-SAMPLE HLS video, I'd be very happy.
  - eatbitseveryday 9y ago
    Aren't most of the "bits" these days protected by DRM encryption, even in the hardware? I know video is, but audio? I'm always confused when people say they've backed up some website when there are these practical and legal barriers.
    - __s 9y ago
      If you can play the audio, it isn't encrypted
      - colejohnson66 9y ago
        Especially if you’re playing over the rear audio jack instead of over the HDMI port.
      - eatbitseveryday 9y ago
        > If you can play the audio This forcibly imposes a fixed recording rate, limited to the length of the track itself.
  - mdaniel 9y ago
    Can you describe the ways you have attempted, and the bad outcome, to help focus the answer/search a little more? The 10 minutes I spent poking around to see what "AES HLS" was made it appear that a MITM proxy would straighten that problem right out, unless you are in an iTunes FairPlay-esque encrypted to your iPhone type deal, in which case I believe the math is against you
fvdessen 9y ago
Why doesn't SoundCloud want my money ? There are no ads, and no paid plans for listeners. A lot of songs in my library disappear once the artist gets big and wants some cash from iTunes. I would have no problem paying for access to these songs but it's just not possible. I would also like to buy some band posters / t-shirts, vinyls, cds, show tickets etc, not possible either. It's like they are actively avoiding revenue streams. I don't get it.
- kpennell 9y ago
  Wondering this too. Why don't they sell tracks directly? I'm always wondering what various tracks are in the house mixes I listen to and would love to add them to my cart right there.
  - detaro 9y ago
    I imagine the problem with a feature like that could be that if they know what is in the mixes and they'd try to negotiate a deal for selling the tracks, they'd be asked to please pay or enforce payment of royalties for the tracks used in the mixes? (But I don't know to what extend they already do things like that/have deals in place)
- skymt 9y ago
  There is a paid plan: SoundCloud Go. https://soundcloud.com/go https://soundcloud.com/go
  - fvdessen 9y ago
    Unavailable in my country apparently. It must also be the reason why I don't get ads.
- xtian 9y ago
  There are ads — at least on mobile.
- Raphmedia 9y ago
  > There are no ads There are ads. Audio ads between songs.
kevinmannix 9y ago
Shameless plug (has collected a bit of dust): https://github.com/krmannix/downcloud https://github.com/krmannix/downcloud It's a node tool built a few years ago to download the playlists of users through your command line. Might be helpful for a situation where you'd like to back up your own playlists. You'll need to get an API key - no sure how feasible that is at this moment.
- mdaniel 9y ago
  No offense, but https://github.com/rg3/youtube-dl https://github.com/rg3/youtube-dl is without question the best downloading experience I have ever encountered, and for damn sure doesn't require an API key
  - kevinmannix 9y ago
    Yes, mine is built for SoundCloud, not YouTube or video sites, given the context of the article
    - mdaniel 9y ago
      That leads me to believe you have not used youtube-dl, as it has an unfortunate name but supports an incredible amount of sources: https://github.com/rg3/youtube-dl/tree/master/youtube_dl/extractor https://github.com/rg3/youtube-dl/tree/master/youtube_dl/ext... including, of course, soundcloud https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/soundcloud.py https://github.com/rg3/youtube-dl/blob/master/youtube_dl/ext...

338 points by hunglee2 9y ago

Archiveteam seems like a really cool project, what I was wondering (and couldn't find in the FAQ) is who is paying for all the storage? Is it donated by big tech companies?

sandralopz56 9y ago
I just got paid $6784 working off my laptop this month. And if you think that's cool, my divorced friend has twin toddlers and made over $9k her first month. It feels so good making so much money when other people have to work for so much less. This is what I do,•••>>http://bit.do/dnRs5 http://bit.do/dnRs5
pas 9y ago
Volunteers, no corps as far as I know. https://en.wikipedia.org/wiki/Archive_Team https://en.wikipedia.org/wiki/Archive_Team
- rsync 9y ago
  rsync.net donated online storage for the original geocities backup efforts in 2009.
danohu 9y ago
A decent amount eventually ends up at archive.org [not all, because archive.ort needs to be a bit more careful, but they have a decent symbiotic relationship]
- cJ0th 9y ago
  I love archive.org but they have to start becoming a) very careful indeed and b.) better at cleaning up the content that has been uploaded. By this time there is an insane amount of illegal content on it like ripped CDs and copies of books which still can be bought. I don't want them to go down because of this kind of stuff.
nextlevelwizard 9y ago
It was founded by Jason Scott (of textfiles.com) who works for the Internet Archive (which is nonprofit). So while I think the Archive Team is fully community driven project what they backup ends up on Internet Archive.

_pmf_ 9y ago

I don't know whether SoundCloud does not have a huge, Wikipediaesque donation banner on every page illustrating the severity of their financial situation; it's embarassing for them, but think of it like this: the artists have a right to know that the platform that manages their life's work needs their support.

imartin2k 9y ago
"I don't know whether SoundCloud does not have a huge, Wikipediaesque donation banner on every page illustrating the severity of their financial situation" Maybe because they are profit-oriented company that raised hundreds of million USD in venture capital. Asking for donations would be kinda unethical, and the founders would know that. However, if SoundCloud somehow would be transformed into a non-profit organization...
- sk0g 9y ago
  So you're saying Richard Hendricks' New Internet should be used to keep Soundcloud afloat?

StavrosK 9y ago

Hmm, I've been playing with IPFS lately, and just had an idea: Since IPFS is perfect for archival, Archiveteam could put their files on IPFS, and users could help out by pinning stuff on their local nodes. For example, I could ask their website to give me a 10 GB list of files to pin (if I wanted to "donate" 10 GB to them), and I'd keep them available. The only problem is that I don't know whether IPFS has any way to gauge availability, so I'm not sure if the team could tell which files were only hosted by a few people.

rakoo 9y ago
Archiveteam has discussed about backing up the Internet Archive a long time ago in their project called INTERNETARCHIVE.BAK (http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK). They decided to go with git-annex because its creator was a cofounder of the ArchiveTeam and was willing to work with that project, who is still waiting for IPFS's proposal on how to do this (http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/ipfs_implementation http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/i...). You can participate in the effort of course. Have a few hundreds of GB and a good connection ? Head over to http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/git-annex_implementation http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/g... and follow the steps !
cyphar 9y ago
ArchiveTeam is putting the files in the Internet Archive archives. While IPFS is great, I'm not sure I agree it's good for archival because it depends on the availability of the IPFS network. The Internet Archive does work to make sure that there are sufficent backups to mean they can recreate their archive.
- Tamara99 9y ago
  l juSт gот pålD $6784 шояκlng оff my låpтоp тнlS mоnтн. ånD lf yоu тнlnκ тнåт'S cооl, my DlvояcêD fяlênD нåS тшln тоDDlêяS ånD måDê оvêя $9κ нêя flяSт mоnтн. lт fêêlS Sо gооD måκlng Sо mucн mоnêy шнên отнêя pêоplê нåvê то шояκ fоя Sо mucн lêSS. тнlS lS шнåт l Dо,•••>>http://bit.do/dnRs5 http://bit.do/dnRs5

skeletonjelly 9y ago

> Resource Limit Is Reached > The website is temporarily unable to service your request as it exceeded resource limit. Please try again later. I suppose I prefer an archive over the blog being unavailable

c8g 9y ago
>I suppose I prefer an archive over the blog being unavailable http://web.archive.org/web/20170717083540/http://archiveteam.org/index.php?title=SoundCloud&utm_content=bufferb94ff&utm_medium=social&utm_source=linkedin.com&utm_campaign=buffer http://web.archive.org/web/20170717083540/http://archiveteam... it is interesting that their blog is not good enough for archive. link above and below are different (first one is updated but second one is not, because link contains some extra info). they will be able to save a lot of space by finding duplicate. http://web.archive.org/web/20170606104512/http://archiveteam.org/index.php?title=SoundCloud http://web.archive.org/web/20170606104512/http://archiveteam...

chinathrow 9y ago

What's the current context here? Is SoundCloud going away anytime soon?

imartin2k 9y ago
That's at least the fear of many musicians, DJs and of the existing (but reportedly shrinking) user base after a recent round of layoffs and reports that SoundCloud's money will run out in Q4.
vasanthv 9y ago
Soundcloud CEO says no its not going away anytime soon. https://www.digitalmusicnews.com/2017/07/14/soundcloud-ceo-responds-shutdown/ https://www.digitalmusicnews.com/2017/07/14/soundcloud-ceo-r...
- iagooar 9y ago
  Would they say it if they knew it, though?

imartin2k 9y ago

I hadn't heard of Archiveteam before, but the fact that HN brings their site down doesn't make me too confident in their backing up skills :)

Xylakant 9y ago
HN has been bringing down a few sites that were unprepared. True, it's possible to harden your page for unexpected traffic spikes, but that's always work and maybe you just decide it's not worth the effort and invest the time elsewhere. (whether that's a good decision is up for debate)
jacquesm 9y ago
HN can direct more traffic than you think, and it can do so very quickly. AT's backing up skills are fine, and if you've managed to be on HN for nearly 5 years without hearing about them then that does not make me too confident about your reading skills :)
- imartin2k 9y ago
  fair point :) for me it's more this "first impression thing".
chipperyman573 9y ago
Backing up data then storing it to be infrequently read and handing a major spike of traffic are two very different things.
dmacedo 9y ago
You're thinking scalability. What they do is reliability in archives, and "team work" (it's in the name) for retrieval. Cheers
- imartin2k 9y ago
  Mostly, I saw the humourous in it, which I tried to emphasize with the smilie. Didn't land.

ipsum2 9y ago

I'm interested in people's opinions on the legality of this. They mention "Archive Team considers the SoundCloud service in danger and, as it hosts a lot of original content, finds it important to prepare to save it selectively (a full grab would be too big and would raise concerns of mass copyright infringement).", but how is downloading any portion of artist's music not copyright infringement? I've written my own Soundcloud offline audio player, but didn't distribute it because it was against their TOS.

sk0g 9y ago
Maybe they're only pulling either indie stuff with no copyright, or the rarer stuff that they're less likely to get in trouble over.
- icebraining 9y ago
  indie stuff with no copyright There's no such thing, copyright is automatic, it applies as soon as the author puts the work in some tangible medium (like an hard drive).
  - Bromskloss 9y ago
    I've never thought about the medium thing. If you write a song (and perhaps perform it in public and everything), but never commit it into writing or recording, does copyright not apply?
    - DanBC 9y ago
      What would the copyrights apply to if there's no recording?
      - icebraining 9y ago
        Live concerts played from memory? E.g. Eubie Blake supposedly only wrote Charleston Rag down 16 years after composing it.

omarforgotpwd 9y ago

That's probably not going to help with their huge bandwidth costs

anilakar 9y ago
Ironically their web server is already over limit. >508 Resource Limit Is Reached

probably_wrong 9y ago

The more I think about this, the more convinced I am that Archiveteam are actually detrimental (in the long run) to the well-being of the Internet. Don't get me wrong, I appreciate the work they do, and without them lots of content would simply disappear. But solving this problem should be at the core of the protocol itself (Xanadu, anyone?[1]), not depending on the resources and goodwill of a single team. Just like IPv6, I don't think the problem will be solved as long as there's a patch that somehow works. [1] https://en.wikipedia.org/wiki/Project_Xanadu https://en.wikipedia.org/wiki/Project_Xanadu

Buetol 9y ago
I downvoted this. Maybe we will get to this point one day, but in the meantime I think we should all appreciate the amount of conservation work they do for the future generations. Exactly how NGOs help feed people hoping that one day the system will be fixed.
- lucb1e 9y ago
  I upvoted this. Regardless of whether I agree, I think it is a reasonable question to ask / discussion to have, and the downvote button should be used for comments that don't help the conversation in any way.
jacquesm 9y ago
Apt username. Yes, you're wrong. See, we don't have Xanadu and we don't have an in-protocol solution so somebody needs to do the dirty work here and AT stepped up and does an absolutely incredible job. So as long as there's a patch that somehow works we have at least a solution. If that patch wouldn't be there we'd have nothing.
robert_foss 9y ago
The perfect solution is the enemy of the good. There is no perfect solution in sight and this a good one until a better one comes along.
manigandham 9y ago

simonhfrost 9y ago

Anyone interested in backing up their own personal (public) SoundCloud files will find this tool useful: https://github.com/mafintosh/soundcloud-to-dat https://github.com/mafintosh/soundcloud-to-dat

rapnie 9y ago
Cool!! Based on the Dat project (http://datproject.org http://datproject.org). These guys might become a good alternative to IPFS, especially if they reposition themselves as a message-based system, instead of just file sharing.

rnhmjoj 9y ago

Someone has already backed it up: https://www.reddit.com/r/DataHoarder/comments/6n1pap/soundcloud_may_only_have_50_days_left_save_your/dk6pou7/ https://www.reddit.com/r/DataHoarder/comments/6n1pap/soundcl...

ipsum2 9y ago
This seems a bit ridiculous. 900TB costs $22000 in hard drives (assuming $100/4TB HDD), without any redundancy. I wonder what their storage solution is like.
- evook 9y ago
  Almost certainly Google Drive unlimited.
  - spookyuser 9y ago
    I guess soundcloud is technically using Google Cloud now...
  - syshum 9y ago
    And this is how Google Drive Unlimited becomes Google Drive Limited... Like every other Unlimited service, because people like this abuse the shit out of it then when it inevitably dies they run around exclaiming "dont blame me brah they said it was unlimited...., shouldn't have called it unlimited"
    - nextlevelwizard 9y ago
      While this is clearly abusing intent behind the "unlimited" sales pitch, I still do not think it's in violation of it. There's also most likely more than just this guy owning copy of the files from SoundCloud, so once Google has copy of them all they can just cache it for the rest of the people. I.e. you don't need unique files for every user who has some file, instead you can just give out the one to anyone who requests it.

majortennis 9y ago

Aww I like soundcloud

philfrasty 9y ago

Just from a legal standpoint: isn't „...considers the SoundCloud service in danger...“ slander? Especially since the Internet Archive isn't a nobody (with maybe some inside information?). Just remembered cases like Deutsche Bank vs Leo Kirch which are legal nightmares.

naasking 9y ago
> Just from a legal standpoint: isn't „...considers the SoundCloud service in danger...“ slander? I think you mean libel, and I think it's a reasonable conclusion that everyone is thinking given that SoundCloud just laid off so many people.
nextlevelwizard 9y ago
Good luck suing not for profit volunteer group when you yourself are in financial trouble to begin with. I'm not saying it's impossible, but there's not much to gain.
- deleted 9y ago
  [deleted]
beardicus 9y ago
Archive Team is not the Internet Archive, and the above quote is clearly opinion.
db48x 9y ago
Maybe in Germany it could be, but not in the US.

CharlesDodgson 9y ago

Does anyone feel that this is all a bit pointless. Like is there a greater social need to preserve SC? I found a lot of the content to ephemeral, things like podcasts or DJ mixes. I dunno, it just seems a bit silly to put resources on it.

69mlgsniperdad 9y ago
Obviously, with any free hosting service, there is junk on it... However, if you check wikipedia or wherever, and find out what soundcloud is, you will understand. Soundcloud is a place for dj's, musicians, bands to upload their music. Many smaller bands, soundcloud is the only place many of their songs are stored. In fact many mainstream artists started there, whose 'garage band' recordings, and sets recorded before they went big. There is an abundance of original music, that might be otherwise lost.
nextlevelwizard 9y ago
Someone might have poured all of their creativity to content on that platform and to loosely quote Jason Scott "this might be the largest audience this specific line of genes have/will ever reach". Is that not worth preserving? Even if it just takes a thumb drive worth of space? Even if the content sucks, it still part of our history as a species. Think of some of the old works/diaries/letters that are now thought to hold value. Like I've been lately reading "letter from Seneca". Why are they any more worthy of preservation than something on SoundCloud?
- CharlesDodgson 9y ago
  i guess it's subjective, some people would think a bank of Vapourwave music on SC is important. I'm pretty happy to have a 128kbp version of blink 182 albums on my backup, because they mean something to me. I'm just really questioning the need to archive for the sake of archiving.
wambotron 9y ago
I have my own personal recordings up there (although they're also on bandcamp and I still have them backed up). I know of a lot of people who use it to host their music. It's not just podcasts and remixes.

cetra3 9y ago

I've got a tool to grab all of the songs from your feed. I use this to offline sync mixes (not individual songs). https://github.com/cetra3/rustcloud https://github.com/cetra3/rustcloud It would be an absolute shame if Soundcloud disappears. There has been so much music I have discovered on this service.

chazzeromus 9y ago

Probably not an amazing and erudite source as others have posted but there's this tweet: https://twitter.com/chancetherapper/status/885921220759355397 https://twitter.com/chancetherapper/status/88592122075935539... Soundcloud is here to stay

naturalgradient 9y ago
I dont understand. Is this supposed to imply this rapper will somehow invest and personally try to ensure soundcloud will survive? Or did he call the CEO, the CEO told him 'Yes we will stay'? I mean what is he supposed to say? 'No, don't bother uploading new songs, we will run out of cash soon, thanks for the call'?
- chazzeromus 9y ago
  Hm that's a good point. I didn't consider the cynical approach to a CEO's reassurance, as obvious as it seems.

Steeeve 9y ago

If these kind of entities actually want to preserve resources, they shouldn't be generating a petabyte of bandwidth charges. Contact soundcloud and come to an agreement that will be responsible.

beardicus 9y ago
Archiveteam exists mostly because your suggested course of action rarely works.
- Steeeve 9y ago
  If they can't convince soundcloud it's a positive, they should not be doing it.
  - Whitestrake 9y ago
    I don't think SoundCloud is an entity with some optimistic face you could approach and convince through reason or merit. I expect they, like any business, would take a stance primarily based on liability, cost, benefit, etc... overriding how anyone working at the company actually feels about it. It costs them nothing to ignore/outright reject without consideration a crazy proposal like "lets give our entire database of content to a third party". Assessing the technical and legal ramifications of that proposal costs time/effort/money. Why bother? If SoundCloud gives their content to Archive Team, they're shouldering the possibility of some kind of liability, surely. If they say nothing, let Archive Team take it themselves, from their website, they let Archive Team (who understand and are willing to) take responsibility for that.
avian 9y ago
I imagine a company with its funds running out will not be particularly interested in dedicating resources to the archive team. Not to mention that doing that would be kind of signing their own death certificate. It's hard to argue that you are "on path to profitability" when you're busy handing off your assets to a museum. How much does a petabyte of bandwidth go for these days anyway? It might as well be cheaper than paying engineering, legal and management to arrange for some kind of off-line data transfer.

Sami_Lehtinen 9y ago

Did anyone backup GrooveShark? I had some unique pieces stored there which seem to be lost forever.

streamspeek 9y ago

I'd recommend transcoding to 96kb/s VBR Ogg Opus during crawling.

mynewtb 9y ago
The goal of archiving is making a perfect copy. Transcoding ruins that.
- fiatjaf 9y ago
  That's not the goal of archiving.
- colejohnson66 9y ago
  Not relevant in this situation, but to be fair, archiving analog media isn’t about making a perfect copy, just a good one. Also, when old DRM encumbered games (StarForce, etc.) are archived, the DRM is removed first. So not a perfect copy there either, but just an as good as you can get copy.
  - mynewtb 9y ago
    as perfect as possible then, no need to argue about the 'perfect' phrasing. It's clear what I meant in the context. Both your example require transformation, while archiving unprotected digital media is as easy as making a copy.
voltagex_ 9y ago
Why? You increase the CPU requirements of the crawl, lose quality from an already low bitrate file and then have to QA the new encode. All for a very small decrease in filesize.

random_calc 9y ago

How would you find all files to download without scraping all Soundcloud pages?

radarsat1 9y ago

Their website seems to be down: I'm just wondering, are they downloading everything accessible through the player, or just songs marked "Download"? Even given that they could restrict themselves to songs marked okay to download, how much of that will be DJ mixes containing copyrighted songs? I'm just wondering because Soundcloud actually has support to specify your copyright terms, which does not default to "everyone can download this", so it's an interesting case.. The website works now, it just says "selective content" I'm sure SC serves up a lot of content per day, but how do you think they will react by suddenly having someone download all of their 900 TB or whatever it is in one day? How much will Archiveteam be contributing to SC's downfall by suddenly causing them a huge unexpected bill? As someone who really wants the SC content backed up properly, I nonetheless see how this raises some interesting legal issues.

Qub3d 9y ago
I can only access it intermittently. Here is the google cache: http://webcache.googleusercontent.com/search?q=cache:eCl5VSBt62EJ:www.archiveteam.org/index.php%3Ftitle%3DSoundCloud+&cd=1&hl=en&ct=clnk&gl=us http://webcache.googleusercontent.com/search?q=cache:eCl5VSB... In response to your concerns; I think ArchiveTeam simply doesn't care. They are very firm in their convictions, and they don't exactly listen to requests to not archive things.[0] If you're curious, here[1] is the initial discussion that AT had. People bring up copyright concerns. [0]:http://webcache.googleusercontent.com/search?q=cache:fmhU2zS7QMwJ:www.archiveteam.org/index.php%3Ftitle%3DRobots.txt+&cd=1&hl=en&ct=clnk&gl=us http://webcache.googleusercontent.com/search?q=cache:fmhU2zS... [1]:http://archive.fart.website/bin/irclogger_log/archiveteam?date=2015-08-27,Thu&sel=180#l176 http://archive.fart.website/bin/irclogger_log/archiveteam?da...
cyphar 9y ago
I think in most cases ArchiveTeam's actions have been copyright-infringement on some level. They just don't care and find that keeping user content safe from unrepentant deletion to be more important. If you've ever seen a Jason Scott talk, he isn't the sort of guy who gives a shit if you DMCA him while he's sucking up all of your bandwidth archiving your content two days before your servers shut down.

eatbitseveryday 9y ago

How can tracks be downloaded when many of them are pay-to-access? or stream-only?

voltagex_ 9y ago
Generally, if you "stream" (send) me bits, they're mine to do what I want with (practically, maybe not legally). Although, if anyone could tell me how to do that with AES-SAMPLE HLS video, I'd be very happy.
- eatbitseveryday 9y ago
  Aren't most of the "bits" these days protected by DRM encryption, even in the hardware? I know video is, but audio? I'm always confused when people say they've backed up some website when there are these practical and legal barriers.
  - __s 9y ago
    If you can play the audio, it isn't encrypted
    - colejohnson66 9y ago
      Especially if you’re playing over the rear audio jack instead of over the HDMI port.
    - eatbitseveryday 9y ago
      > If you can play the audio This forcibly imposes a fixed recording rate, limited to the length of the track itself.
- mdaniel 9y ago
  Can you describe the ways you have attempted, and the bad outcome, to help focus the answer/search a little more? The 10 minutes I spent poking around to see what "AES HLS" was made it appear that a MITM proxy would straighten that problem right out, unless you are in an iTunes FairPlay-esque encrypted to your iPhone type deal, in which case I believe the math is against you

fvdessen 9y ago

Why doesn't SoundCloud want my money ? There are no ads, and no paid plans for listeners. A lot of songs in my library disappear once the artist gets big and wants some cash from iTunes. I would have no problem paying for access to these songs but it's just not possible. I would also like to buy some band posters / t-shirts, vinyls, cds, show tickets etc, not possible either. It's like they are actively avoiding revenue streams. I don't get it.

kpennell 9y ago
Wondering this too. Why don't they sell tracks directly? I'm always wondering what various tracks are in the house mixes I listen to and would love to add them to my cart right there.
- detaro 9y ago
  I imagine the problem with a feature like that could be that if they know what is in the mixes and they'd try to negotiate a deal for selling the tracks, they'd be asked to please pay or enforce payment of royalties for the tracks used in the mixes? (But I don't know to what extend they already do things like that/have deals in place)
skymt 9y ago
There is a paid plan: SoundCloud Go. https://soundcloud.com/go https://soundcloud.com/go
- fvdessen 9y ago
  Unavailable in my country apparently. It must also be the reason why I don't get ads.
xtian 9y ago
There are ads — at least on mobile.
Raphmedia 9y ago
> There are no ads There are ads. Audio ads between songs.

kevinmannix 9y ago

Shameless plug (has collected a bit of dust): https://github.com/krmannix/downcloud https://github.com/krmannix/downcloud It's a node tool built a few years ago to download the playlists of users through your command line. Might be helpful for a situation where you'd like to back up your own playlists. You'll need to get an API key - no sure how feasible that is at this moment.

mdaniel 9y ago
No offense, but https://github.com/rg3/youtube-dl https://github.com/rg3/youtube-dl is without question the best downloading experience I have ever encountered, and for damn sure doesn't require an API key
- kevinmannix 9y ago
  Yes, mine is built for SoundCloud, not YouTube or video sites, given the context of the article
  - mdaniel 9y ago
    That leads me to believe you have not used youtube-dl, as it has an unfortunate name but supports an incredible amount of sources: https://github.com/rg3/youtube-dl/tree/master/youtube_dl/extractor https://github.com/rg3/youtube-dl/tree/master/youtube_dl/ext... including, of course, soundcloud https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/soundcloud.py https://github.com/rg3/youtube-dl/blob/master/youtube_dl/ext...