10 ms·
Using Backblaze B2 and Cloudflare Workers for free image hosting
- ww520 7y agoVery good info. Didn't know B2 is cheaper than S3.
- CherryJimbo 7y agoYou're very welcome - I'm glad it was helpful. B2 is significantly cheaper than S3, especially when paired with Cloudflare for free bandwidth. If you're interested, my company Nodecraft talked about a 23TB migration we did from S3 to B2 a little while ago: https://nodecraft.com/blog/development/migrating-23tb-from-s3-to-b2-in-just-7-hours https://nodecraft.com/blog/development/migrating-23tb-from-s...
- ww520 7y agoHow's the outgoing/ingress bandwidth comparison? Outgoing bandwidth is expensive on AWS.
- pavanagrawal123 7y agowasabi has it free and (afaik is cheaper than s3 and B2)
- elabajaba 7y agoAccording to their documentation, Wasabi will move you on to their $0.005/GB storage $0.01/GB download plan (same price as B2 without api charges) from their $0.0059/GB storage free egress plan if you download more than your total storage (eg. with 100TB stored don't download more than 100TB/month). https://wasabi.com/pricing/pricing-faqs/ https://wasabi.com/pricing/pricing-faqs/
- CherryJimbo 7y agoIt's entirely free if you only use the Cloudflare-routed URLs thanks to the Bandwidth Alliance: https://www.cloudflare.com/bandwidth-alliance https://www.cloudflare.com/bandwidth-alliance
- bengeren 7y agoHow does CloudFlare themselves afford to give bandwidth for free? I understand that I can pay $20/mo for pro account but they also have a $0/mo option with fewer bells and whistles. What gives them the advantage to charge nothing for bandwidth?
- mikehotel 7y agoTheir terms indicate it should be used for caching html primarily. So if they find costly abusers, they could use this clause to get them to upgrade to a paid tier. > 2.8 Limitation on Non-HTML Caching The Service is offered primarily as a platform to cache and serve web pages and websites. Unless explicitly included as a part of a Paid Service purchased by you, you agree to use the Service solely for the purpose of serving web pages as viewed through a web browser or other application and the Hypertext Markup Language (HTML) protocol or other equivalent technology. Use of the Service for the storage or caching of video (unless purchased separately as a Paid Service) or a disproportionate percentage of pictures, audio files, or other non-HTML content, is prohibited. https://www.cloudflare.com/terms/ https://www.cloudflare.com/terms/
- gcb0 7y agoprivate company with lots of funding rounds? /shrug they did have (claim to have) 90m revenue last year, after 10m every year before that.
- eastdakota 7y agoBecause we’re peered with Backblaxe (as well as AWS). There’s a fixed cost of setting up those peering arrangements, but, once in place, there’s no incremental cost. That’s why we have similar agreements to Backblaze in place with Google, Microsoft, IBM, Digital Ocean, etc. It’s pretty shameful, actually, that AWS has so far refused. When using Cloudflare, they don’t pay for the bandwidth, and we don’t pay for the Bandwidth, so why are customers paying for the bandwidth. Amazon pretends to be customer-focused. This is a clear example where they’re not.
- aidenn0 7y agoAny web service from amazon is usually on the pricey side of normal.
- viraptor 7y agoYou get more features and higher cost. Basically B2 gives you storage, but not detailed access management, endpoint internal to your vpc, and other extras.
- gcb0 7y agowhich honestly, most people just use for s3 and suffer the rest because the CTO reads too much HN. amazon today mostly sells based on "you can't get fired for choosing aws", ironically.
- BFLpL0QNek 7y agoIt is but I think they only have US datacenters. For backups, media and archival use-cases it looks really good for the price if you can live with it being in the US. If you are doing any large data-processing using S3 you get the advantage of data locality, with VPC endpoints you can also bypass NAT gateway data charges and get much higher bandwidth.
- trevyn 7y agoI think B2 is still only single-datacenter.
- votepaunchy 7y ago> backups, media and archival use-cases it looks really good for the price if you can live with it being in the US For these use cases S3 has lower pricing tiers (down to 0.1¢/GB-mo, matched by Azure and promised by GCP).
- haywirez 7y agoIt’s cheap but it’s proving unacceptably slow for me - sometimes I see 2.5s TTFB for accessing tiny audio files in my region (Berlin, EU). Server uploads are also quite unreliable, had to write a lot of custom retry logic to handle 503 errors (~30% probability when uploading in batch). Great for it’s intended use (backups), but I’ll be switching to an S3 compatible alternative soon - eyeing Digital Ocean Spaces or Wasabi...
- trevyn 7y agoHad a similar experience with B2, Wasabi was much faster in my testing.
- shakna 7y agoThose stats sound insane to me, and certainly don't reflect what I see. I see 50ms or less TTFB, for images in the sub 200Kb range, and for videos in the 500Mb+ range, from Australia where the internet is still terrible. I've only ever a single serve upload fail me - and it occurred when an upload hit a major global outage of infrastructure. In two years of regularly uploading 8Gb/200 files a fortnight (at the least), I've never needed custom retry logic.
- trevyn 7y agoIf you are seeing 50ms TTFB between B2’s only datacenter (in California) and Australia, there is something wrong with your methodology or you have discovered FTL communication.
- berkut 7y agoI've been seeing pretty bad upload failures (probably around 30%) for uploading hundreds of 30-40 MB files per month to B2 from New Zealand since I started using B2 over a year ago. And I'm not convinced it's connectivity issues, as I can SCP/FTP the same files to servers in the UK... When I test using an actual software client (Cyberduck) to do the same thing to B2, I see pretty much the same behaviour: retries are needed, and the total upload size (due to the retries) is generally ~20% larger than the size of the files.
- derivablefunc 7y agoSame, thank alone is very useful takeaway for me :).
- jasonlingx 7y agoSome possibly simpler alternatives to consider: Google Photos, Netlify, Gitlab/Github Pages
- deleted 7y ago[deleted]
- techntoke 7y agoThere are actually a couple pretty serious limitations of the Google Photos API: https://tip.rclone.org/googlephotos/#limitations https://tip.rclone.org/googlephotos/#limitations
- anderspitman 7y agoDo GitHub pages integrate nicely with the large file support for storing binary media?
- hinkley 7y agoHow are we feeling about Cloudflare stability versus AWS these days?
- floatingatoll 7y agoThey rarely (or never?) go down at the same time for any reason, other than the standard Internet BGP drama that all providers are at risk of and have no control over.
- svnpenn 7y agoWow I was just able to drop Google Drive for basic storage with this and rclone. Thank you!
- anderspitman 7y agoWe're you able to use Google Drive for web image hosting somehow?
- rasz 7y agoimages, pirated videos, you name it https://openloadmovies.bz/movies/men-in-black-international-2019/ https://openloadmovies.bz/movies/men-in-black-international-... Google seem not to care
- InvaderFizz 7y agoEdit: See replies below. Cloudflare CEO says this use case is fine. This is a cool project and something I will probably use for some hobby projects. I would caution against it for anything more than a hobby project as it violates the Cloudflare TOS: > 2.8 Limitation on Non-HTML Caching > The Service is offered primarily as a platform to cache and serve web pages and websites. Unless explicitly included as a part of a Paid Service purchased by you, you agree to use the Service solely for the purpose of serving web pages as viewed through a web browser or other application and the Hypertext Markup Language (HTML) protocol or other equivalent technology. Use of the Service for the storage or caching of video (unless purchased separately as a Paid Service) or a disproportionate percentage of pictures, audio files, or other non-HTML content, is prohibited. https://www.cloudflare.com/terms/ https://www.cloudflare.com/terms/ For something small, they won't care. If your images make the front page of reddit, you might get shut down.
- eastdakota 7y agoThat’s only for our traditional service. For Workers the ToS is different. Don’t see anything troubling about this project!
- InvaderFizz 7y agoWorkers are being used to do some URL rewriting. The main point of this article is to use a Cloudflare cache-everything rule and use that caching to create a free image host. From the article: > I'd heavily recommend adding a page-rule to set the "cache level" to "everything", and "edge cache TTL" to a higher value like 7 days if your files aren't often changing.
- steve19 7y agoThe guy you replied to is the CEO of cloudflare. If he says it's OK then I'm pretty sure it's OK!
- eastdakota 7y ago
- deleted 7y ago[deleted]
- abafazi 7y agoOr just use Imgur, or Discord
- bufferoverflow 7y agoImgur deletes photos after a while.
- CherryJimbo 7y agoAnd compresses them heavily.
- Markoff 7y ago> Backblaze has a 10GB free file limit, and then charges $0.005/GB/Month thereafter. is this true, I can have 110GB cloud storage for 0.5$ per month? it sounds TGTBT
- oakslab 7y agohttps://www.backblaze.com/b2/cloud-storage-pricing.html https://www.backblaze.com/b2/cloud-storage-pricing.html One way they can achieve that pricing is by using consumer drives, instead of enterprise drives. See https://www.backblaze.com/blog/vault-cloud-storage-architecture/ https://www.backblaze.com/blog/vault-cloud-storage-architect...
- sliken 7y agoThat doesn't seem completely fair. Much like blaming google for using normal commodity servers compared to Altavista using high end enterprise hardware. What matters is the reliability of the system, not some random part. Backblaze is quite transparent about how they do things. They publish their drive reliability numbers (including brand/model numbers), storage pod design, and how their sharding/redundancy works. Seems like most cloud storage vendors just say "We do object storage right handwave and we have lots of 9s". Backblaze says they shard your data into 20 pieces onto 20 servers and can recover with any 17 of those pieces. More details at https://www.backblaze.com/blog/reed-solomon/ https://www.backblaze.com/blog/reed-solomon/ Sure that's not enough redundancy for some, but at least you know what to expect and can plan accordingly. I've not see any other cloud vendor do that. Please post URLs for similar info from other companies.
- bufferoverflow 7y ago1TB OneDrive is $80/year ($6.66/month). So 110GB is $0.73/month. Not that far from 50 cents.
- Splines 7y agoThat 1TB you always pay for, while Backblaze is pay-per-use.
- homero 7y agoI didn't realize workers had a free plan. I avoided trying it for a while.
- StavrosK 7y agoThey added it very recently.
- gozzoo 7y agoFrom what I'm seeing workers are used only for URL rewriting. This can be acheived much simpler with page rules.
- CherryJimbo 7y agoWorkers are also used for basic CORS headers, and stripping some other unnecessary headers. They're definitely not required, but I don't believe you can do URL rewriting with page rules; redirects, sure, but not rewriting.
- kaivi 7y agoBackblaze is cheap, but if you're uploading millions og files, beware -- there is no way to just nuke/empty a bucket with a click of a button. If you're not keeping filename references in an external database, you are left to sequentially scan and remove files in batches of 1000 in a single thread. Support could not help, and it took me months to empty a bucket that way.
- tty7 7y agoThat doesn't really make any sense, Backblaze were not limiting you to a single thread - you were...
- kaivi 7y agoYou do need access to an index/DB of all files in a bucket in order to delete them in parallel. Otherwise you're stuck paginating with the B2 API.
- PopeDotNinja 7y agoI believe S3 is the same way.
- kaivi 7y agoS3 has an "Empty bucket" button, unlike B2.
- brianwski 7y agoDisclaimer: I work at Backblaze. > no way to empty a bucket. Backblaze currently recommends you do this by writing a “Lifecycle rule” to hide/delete all files in the bucket, then let Backblaze empty the bucket for you on the server side in 24 hours: https://www.backblaze.com/b2/docs/lifecycle_rules.html https://www.backblaze.com/b2/docs/lifecycle_rules.html
- deleted 7y ago[deleted]
- ebg13 7y agoIronically, none of the images on the page are loading right now because of "Error 1101 Worker threw exception". So, you know, caveat emptor.
- CherryJimbo 7y agoThat was entirely my bad. I was moving a few things around.
- bitmedley 7y agoUsing CloudFlare workers to clean up the URLs is neat but it seems like it would be really easy to reach the limits of the free tier. https://workers.cloudflare.com/docs/reference/runtime/limits/ https://workers.cloudflare.com/docs/reference/runtime/limits... You get 100,000 request per month and up to 1,000 requests in a 10 minute timeframe. So if you have a page with 10 images on it and you get 100 people visiting that page within a 10 minute timeframe, you will use up all of your free tier and all new visitors will get a 1015 error. For paid plans you must pay at least $5 and get 10 million requests included and additional requests are 50 cents per million.
- CherryJimbo 7y agoYou get 100,000 requests per day, not per month. The burst limits are definitely a concern for heavy traffic, but for just $5 you can remove the burst limits entirely, as you mention.
- StavrosK 7y agoHere's the script, edited to cache files forever: https://www.pastery.net/tdszux/ https://www.pastery.net/tdszux/ I wrote a simple uploader script that adds a random ID to each upload so they don't clash, but this will work fine regardless.