20 ms·
RawGit is now in a sunset phase and will soon shut down
- klenwell 8y agoI figured this day was coming. Excellent writeup. Sorry to see it go. Thanks for all the raw fish. Sounds like the cost of hosting this were not insignificant. Curious how was this financed. EDIT: To answer my own question, I found the FAQ page, which is also great: https://github.com/rgrove/rawgit/blob/master/FAQ.md https://github.com/rgrove/rawgit/blob/master/FAQ.md Sounds like donations helped (maybe). Oh and Stackpath. Kudos to them, too.
- codefined 8y ago> It's super nice of you to offer, but I don't need any donations at this time. RawGit's server costs are minimal, and the lovely people at StackPath provide RawGit's CDN service free of charge. Thank you though!
- rgrove 8y agoHi! I'm the guy behind RawGit. I've actually never accepted donations. I paid the meager cost of the $10/month DigitalOcean droplet for the origin server, and StackPath (formerly MaxCDN) sponsored the CDN, which handled the majority of RawGit's traffic. Once, I worked out how much the CDN bill would have been if StackPath hadn't sponsored me and my head nearly exploded. So yeah, I definitely couldn't have done it without them!
- tardyp 8y agoMaxcdn must be very happy with all the data they gather with your project... I stopped using all those cdn based js deliver service when I saw all the cookies they were adding to my user's browsers.
- rgrove 8y agoStackPath doesn't add any cookies to cdn.rawgit.com responses. Here are the complete headers from a sample response: HTTP/1.1 200 OK Date: Fri, 12 Oct 2018 17:56:30 GMT Content-Type: application/json;charset=utf-8 Transfer-Encoding: chunked Connection: close X-Content-Type-Options: nosniff X-Robots-Tag: none Access-Control-Allow-Origin: * ETag: "9c153866d0cad7024c0f31eb7b65be11582a6737" Cache-Control: max-age=86400 Vary: Accept-Encoding RawGit-Cache-Status: HIT Server: NetDNA-cache/2.2 Strict-Transport-Security: max-age=31536000; preload X-Cache: HIT
- snthd 8y agoPossibly - https://en.wikipedia.org/wiki/HTTP_ETag#Tracking_using_ETags https://en.wikipedia.org/wiki/HTTP_ETag#Tracking_using_ETags
- Maxious 8y agoThe ETags are identical to those from Github: $ curl -I https://cdn.rawgit.com/sindresorhus/awesome/master/readme.md https://cdn.rawgit.com/sindresorhus/awesome/master/readme.md etag: "1218d241f6c717fa83d0f1afa64809d6ce7451c8" $ curl -I https://raw.githubusercontent.com/sindresorhus/awesome/master/readme.md https://raw.githubusercontent.com/sindresorhus/awesome/maste... ETag: "1218d241f6c717fa83d0f1afa64809d6ce7451c8"
- wumms 8y agoHow much would the bill have been? Rough estimate: 176TB per month: $3.500 to $4.200.
- rgrove 8y agoIt was a while ago now so my memory is hazy, but that sounds about right.
- jdorfman 8y agoIt was a pleasure working with you on this Ryan. You did an amazing job.
- rgrove 8y agoThanks Justin! Likewise.
- tim333 8y agoDid you consider trying to sell the thing to GitHub or GitLab? That many users must be worth something I'd imagine.
- platz 8y ago> 176 terabytes of bandwidth How did he pay for this?
- wolfgang42 8y agoThe FAQ says "CDN hosting generously donated by StackPath".
- dguo 8y agoIt looks like he didn't, for the most part: https://github.com/rgrove/rawgit/blob/master/FAQ.md#can-i-donate-moneybitcoinpie-to-help-you-out https://github.com/rgrove/rawgit/blob/master/FAQ.md#can-i-do... "It's super nice of you to offer, but I don't need any donations at this time. RawGit's server costs are minimal, and the lovely people at StackPath provide RawGit's CDN service free of charge. Thank you though!"
- icebraining 8y agoCongratulations, it was a great gift for the web. Nothing lasts forever anyway.
- samat 8y agoThis was very useful service, sad to see it go.
- geerlingguy 8y agoThis will be interesting... I just did a grep of my local codebases and found 11 instances where CSS or JS resources were being downloaded from rawgit.com. I have a feeling when it's turned off, a lot of sites are going to break.
- jimaek 8y agoRyan is not going to allow any sites to break. Still, everyone should switch to an alternative
- Gaelan 8y agoUntil “at least October of 2019”. Who knows what’ll happen after that
- rgrove 8y agoI know! :D
- Gaelan 8y ago…and? :)
- onion2k 8y agoAnd, ultimately, it's not rawgit's job to maintain a website. If you use a free service as part of your deployment it's your job to keep an eye on it in case it shuts down. Rawgit has provided a fantastic service that's proved very useful and I'm glad it's been available as long as it has.
- manquer 8y agoIt is bit more complex than that. it is incredibly hard to keep an eye on whether a library you use has a dependency which uses such services, and even if they fix it in the next version it may not be fixed in the version you use.
- lima 8y agoMaybe Red Hat (who use Rawgit a lot) or, for that matter, StackPath want to take over?
- jimaek 8y agoIt was a really cool project and I think the first to offer this kind of features. Sad to see it go. But right now jsDelivr supports both GitHub and npm as source for CDN files. So here is an easy tool for migration https://www.jsdelivr.com/rawgit https://www.jsdelivr.com/rawgit
- Ajedi32 8y agoNot really a full replacement, unfortunately: > For security reasons, we serve HTML files with Content-Type: text/plain. We recommend using GitHub Pages if this is a problem.
- ngokevin 8y agoWish the URLs were a bit cleaner (short hashes, no subdomains, no GH path), but I get that JSDelivr is not catered just for GitHub.
- sdegutis 8y ago> "Unfortunately, RawGit has also become an attractive distribution mechanism for malware. RawGit was meant to improve people's lives, but jerks are increasingly using it to hurt people." This has been my experience too, and I'm now convinced that hosting any online services can only be done by corporations whose business model entirely relies on it, and that a small full-time staff is needed to combat malicious users effectively and keep the service usable for everyone else.
- exelius 8y agoThis is Service Management 101 and why “free” services don’t exist offline. There is always a catch.
- syn0byte 8y agoA Corporation who's entire business model relies on it is actively incentivized NOT to combat spam any more than absolutely required to minimize their own monetary losses as a direct result of the abuses. As long as they aren't held responsible for their users actions it's just eating away a revenue stream. Capitalism!
- pjc50 8y agoThis is something the decentralization fans need to take into account too: if successful, ipfs will be full of malware being distributed from compromised routers etc.
- JCharante 8y agoRawGit was an amazing service, it's truly the end of an era for when you want to render the HTML page for source code you're viewing by just changing the domain.
- enimodas 8y agoWhy not sell or give away the project to someone who has time to fight the abuse?
- tyingq 8y agoPretty sure you could recreate this with an nginx reverse proxy and a little configuration to set the Content-type header dynamically. I don't think any code would be required. As mentioned, the bandwidth costs and chasing abuse would be the higher effort part.
- rgrove 8y agoExactly. And plenty of people have done exactly this. There's nothing technically interesting or unique about RawGit except that it got popular.
- dotancohen 8y agoLike Twitter or Bitly clones. There's nothing technically interesting or unique to write, but just try getting a userbase.
- rgrove 8y agoI did consider this, and several people have offered to take over, but ultimately I feel that the thing that was most useful about RawGit — it made serving HTML at scale dead simple for anyone with a GitHub repo — is also the thing that made it most prone to abuse. It came down to a simple equation: fighting abuse on RawGit will _always_ take more time and effort than spreading abuse via RawGit. One persistent jerk working a few hours a day could do so much damage so quickly that mitigating it would require multiple people working full time. There's just no good way to scale that and retain the functionality that actually made RawGit useful. I talked about this a little more here: https://github.com/rgrove/rawgit/pull/191#issuecomment-428310526 https://github.com/rgrove/rawgit/pull/191#issuecomment-42831...
- shawn 8y ago
- superasn 8y agoI've never used rawgit for my own projects, but it did make life a lot easier to check out demos and test run examples in JS libraries hosted on github. Totally understand that the effort required to fight malware is tedious. But instead of shutting it down you may want to try a $5/mo plan. The money is not for making money but kind of like how Google chrome store required a $5 payment to publish your first chrome extension. Spammers hate paying it and I think as soon as money changes hand there is verification and a trail which makes them nervous too (many forum owners do this as well).
- madrox 8y agoI got the impression the author would rather not deal with the headache anymore, and charging money is trading one set of headaches for another. Building things is fun. Operating things is stressful. I don't blame them.
- superasn 8y agoThe stats he has quoted on the FAQ it wouldn't be surprising if it starts making $10K/mo (just a wild guess) and the malware reduces as a side-effect too. But I guess you're right. Not everyone is driven by money and it would certainly turn a hobby project into a full time job. Sounds fair to shut it down.
- flanbiscuit 8y agoIt was great while it lasted. I currently have a small extension that depends on it. Oh well, I knew the risks of using a free service so no complaints. Thanks for the free service!
- fenwick67 8y agoI used this for a lot of Codepen stuff - sad to see it go.
- rgrove 8y agoYou might find this useful: https://github.com/mallendeo/cdp-rawgit-fix https://github.com/mallendeo/cdp-rawgit-fix It'll scan all your pens and try to suggest jsDelivr URLs wherever you've used RawGit.
- guessmyname 8y agoHighcharts also uses RawGit to serve data for all the demos using jsfiddle.net [1][2][3]. [1] https://www.highcharts.com/demo/line-ajax https://www.highcharts.com/demo/line-ajax [2] https://jsfiddle.net/gh/get/library/pure/highcharts/highcharts/tree/master/samples/highcharts/demo/line-ajax/ https://jsfiddle.net/gh/get/library/pure/highcharts/highchar... [3] https://cdn.rawgit.com/highcharts/highcharts/057b672172ccc6c08fe7dbb27fc17ebca3f5b770/samples/data/analytics.csv https://cdn.rawgit.com/highcharts/highcharts/057b672172ccc6c...
- ravenstine 8y agoThanks for everything! Are you going to open source the Rawgit codebase? I'd be interested just for educational purposes.
- rgrove 8y agoRawGit has been open source from day one: https://github.com/rgrove/rawgit https://github.com/rgrove/rawgit
- piotrkaminski 8y agoIs there a service that can replace RawGit for Gists, specifically serving them with the correct MIME type? This was a convenient feature for users of https://reviewable.io https://reviewable.io to hack up a quick custom stylesheet or keyboard shortcuts override. It's not clear to me whether any of the suggested replacements support Gists.
- ic4l 8y agotry replacing "rawgit.com" with "rawgit2.com"
- deleted 8y ago[deleted]
- admay 8y ago"You either die a hero, or live long enough to see yourself become a villain"
- tokyodude 8y agooh well, a few hundred of the working answers I posted on stackoverflow will stop working. Not sure how I can fix them
- Ajedi32 8y agoThankfully, it's pretty easy to find the GitHub page for the linked source with a bit of URL surgery.
- jopsen 8y agoBy now you can configure GitHub to publish your master branch as github pages. So no need to use rawgit either. The only sad part, is all the links that are breaking.
- cmroanirgo 8y agoKudos: "Unfortunately, RawGit has also become an attractive distribution mechanism for malware. RawGit was meant to improve people's lives, but jerks are increasingly using it to hurt people. Since I have almost no time to devote to fighting malware and abuse on RawGit (and since it would be no fun even if I did have the time), I feel the responsible thing to do is to shut it down. I would rather kill RawGit than watch it be used to hurt people."
- ken 8y agoWhere does one draw the line? The internet is used for that, so should we shut down the internet? Is the content of the internet the responsibility of its creators, or operators? I completely sympathize with this position for a one-person service, but I think the real problem is that we still haven't figured out how to fix incentives so we can distribute the work of keeping the jerks in line. On the internet, the jerks are winning.
- pmcjones 8y ago> On the internet, the jerks are winning. Arguably, it's not just on the internet. But I try to be optimistic.
- ken 8y agoSure. I guess I'm old enough I still see the internet as a thing that we could hypothetically choose to shut down (without killing half the population) -- unlike, say, agriculture and civilization. Maybe that's not realistic, though.
- styfle 8y agoI am having a similar problem with npm. I guess people decided to upload warez to npm and it's affecting my tool that reads from npm. This is why we can't have nice things :( https://github.com/styfle/packagephobia/issues/141#issuecomment-428182653 https://github.com/styfle/packagephobia/issues/141#issuecomm...
- tlrobinson 8y agoThere are ~1M RawGit URLs embedded in open source projects on GitHub: https://github.com/search?q=rawgit.com&type=Code https://github.com/search?q=rawgit.com&type=Code I'm curious to see how many there are a year from now.
- alpb 8y agoPrecisely why I never use free CDNs or stuff like this to serve static content. Just host your own thing. It maybe costs a little bit more but at the end you'll have websites that will run for decades. Rawgit was useful, I wish it stayed up for a little longer, perhaps someone like Cloudflare could sponsor this?
- Ayesh 8y agoCDNJS is from CloudFlare, running with what I believe is better DNS and more POPs.
- scruffyherder 8y agoYet another trendy hip service folds with little to no notice. Meanwhile sourceforge keeps on going.
- fastball 8y ago... what?
- ic4l 8y agoFor the time being I threw this up. You should be able to replace rawgit.com, with rawgit2.com.
- freddie_mercury 8y agoAre you going to self fund the $50,000/year in hosting costs and devote time to fixing the malware problem?
- ic4l 8y agoIf the time comes, sure. Also it can be hosted for much less.
- rgrove 8y agoPlease don't use the name RawGit. This is likely to confuse people, and it's also likely to cause people to reach out to me when they need help with rawgit2.com.
- minhoryang 8y agoI loved your product and enjoyed it. Thanks always!