23 ms·
Backblaze B2 Cloud Storage Now Has S3 Compatible APIs
- avolcano 6y agoHuh, I thought it already had this! Must have mixed it up with a different object storage service (maybe DigitalOcean?). I've been using B2 for backup storage for some personal projects. It doesn't necessarily do anything "better" than S3 from what I've seen, but never having to log into AWS's dashboard is a reward enough on its own. They do have a command-line client that's a quick PIP install, so you can do something like: b2 upload-file bucket-name /path/to/file remote-filename Which is, of course, nice for backups.
- Hamuko 6y ago>Huh, I thought it already had this! Same since it seems to be on the new storage service launch checklist right after "buy hard drives".
- neurostimulant 6y agoI really wish the B2 client support uploading file from Unix pipe. It would be nice to be able to archive a huge directory into a tar.bz archive and directly pipe the result into the B2 client without having to save the archive into disk first. Currently I have to save the tar.bz archive to disk first before uploading to balance. Took several hours to do so (huge spinning disks array, not as fast as ssd), while uploading to B2 is blazing fast. Saving the archive to ramdrive essentially solved this, but as the data grows I don't have enough memory to spare anymore for a ram drive that can fit the whole archive.
- jorams 6y agoCan you use process substitution? b2 upload_file bucket <(tar -cj huge-directory) archive.tar.bz2 The argument the command sees will be something like "/dev/fd/42", and the shell will provide the output of tar through that file.
- neurostimulant 6y agoThanks for the idea! I'm going to try this.
- neurostimulant 6y agoDoes process substitution actually wrote the content to disk first or not? The information on internet I found seem to be conflicting on this. If it's actually writing the data into disk first, then it's probably won't solve my problem (limited disk i/o). Afaik writing to pipe won't result in saving the data to disk temporarily. I guess the only way to know is to try it out on my system and see how it performs.
- Ineentho 6y agoIf process substitution doesn't work, shouldn't /dev/stdin work? I haven't tried it, but as long as b2 doesn't try to check the file size before uploading I don't see why it wouldn't work: b2 upload_file bucket /dev/stdin < file
- ADefenestrator 6y agoIt definitely doesn't write it to disk first. It's basically a pipe() under the hood, but exposed as a file descriptor. Downside is that seeking doesn't work, but that shouldn't affect your case.
- neurostimulant 6y agoSo apparently process substitution doesn't work. The b2 client is probably trying to read file size or something, keep failing with "ERROR: Invalid upload source: /dev/fd/xx". Maybe their api requires knowing the filesize upfront instead of allowing "streaming" upload.
- nielsole 6y agoHere is the reasoning why they didn't have "s3 compatibility" before: https://www.backblaze.com/blog/design-thinking-b2-apis-the-hidden-costs-of-s3-compatibility/ https://www.backblaze.com/blog/design-thinking-b2-apis-the-h...
- yamrzou 6y agoSo how did they manage to get rid of those hidden costs? Or is the new S3 compatible API more expensive?
- toomuchtodo 6y agoIt's possible they're eating the cost of the load balancing necessary to multiplex from their backends to client requests, which should theoretically stoke an increase in business due to reduced switching costs (and what timing, with an economic contraction likely pushing cost reductions at those needing cloud storage). Disclaimer: Happy Backblaze Mac client and B2 customer, no other affiliation. EDIT: @yev: I took the signal out after the sibling reply :) Appreciate the responses as always. Please stay awesome.
- fthead9 6y agoFrom Gleb (CEO) comment on the blog. "Yes - B2 is still a more cost efficient API as it allows customers to connect directly to the final storage location. However, we have built a highly cost efficient load balancing system as we always have - using software to optimize inexpensive hardware - and are swallowing the additional costs for our customers."
- atYevP 6y agoYev here -> saw my Yev-signal. That's right, Gleb actually answered that question on our blog (https://www.backblaze.com/blog/backblaze-b2-s3-compatible-api/#comment-4900843623 https://www.backblaze.com/blog/backblaze-b2-s3-compatible-ap...) but we're eating the cost. Om nom nom.
- deleted 6y ago
- mgamache 6y agoS3 is now the standard for cloud storage APIs? Not sure if that's good or bad. I guess competitors have to reduce switching costs.
- geniium 6y agoThat's a good question to ask. S3 has become so common that it's going toward that road. It's like brand names that become so common that people use no more the material name, but the brand.
- tiborsaas 6y agoIt took me quite some time to realize that TRX is actually a brand and not some abbreviation for the trainers :)
- advisedwang 6y agoI think it essentially always has been. It was there first, [GCP copied it](https://cloud.google.com/storage/docs/xml-api/overview https://cloud.google.com/storage/docs/xml-api/overview), [OpenStack has compatibility middleware](https://docs.openstack.org/swift/latest/middleware.html#module-swift.common.middleware.s3api.s3api https://docs.openstack.org/swift/latest/middleware.html#modu...).
- atYevP 6y agoYev here -> It's not so much a standard, though S3 is generally the most often used suite of APIs. 100s of integrations exist with our B2 Native APIs (https://www.backblaze.com/b2/integrations.html https://www.backblaze.com/b2/integrations.html), but a lot of folks only know how to write to S3 Compatible APIs and don't have the resources to write to multiple API suites, so this makes integration easier for them!
- mgamache 6y agoThat's how something becomes a standard... You are just responding to market realities.
- 6y ago
- dopamean 6y agoI have a decent sized music collection consisting a lot of lossless vinyl rips that I've made from my record collection. It totals around 200gigs at the moment but is growing weekly. I've been looking for somewhere to back this all up in the cloud and backblaze is looking most promising at the moment. Anyone here have any thoughts on where I should go with this?
- l3s2d 6y agoI would recommend restic with B2. Really simple to set up. I use it for nightly homedir backups for multiple machines. Currently at ~400GB and it's costing me < $3/month.
- qmmmur 6y agoDo you automate your process somehow? I'm trying to develop a strategy where if I open my laptop after a certain time it backups in the background. My go to for this kind of thing was hammerspoon but it won't execute the shell task in the background thread and completely locks my machine. I guess I could cron it?
- Fritsdehacker 6y agoI've made systemd profiles and timers to do restic backups. It's really clean and works well. Also much better testable than cron. It will most likely also work for more complex scenarios, like you're mentioning.
- qmmmur 6y agoAny chance you could clue me in on the scripts? Or point me to some resources to learn systemd? I've really disliked cron since having to use it for some auto push CI/CD stuff.
- ac29 6y agoI just set this up last week, and it works well: https://fedoramagazine.org/automate-backups-with-restic-and-systemd/ https://fedoramagazine.org/automate-backups-with-restic-and-... edit: one gotcha is that using "user" systemd units, they will only run when the user is logged in. So, for a personal device like a laptop, this is fine, but for a server, it might not do what you're thinking. For the server use case, you probably want to enable linger for that utility user, so that units will run even with no active logged in sessions: # loginctl enable-linger username
- WalterBAmaQ 6y agoGreat. How about rsync.net-compatible (i.e. bog-standard, vendor-neutral) "APIs"?
- atYevP 6y agoYev here -> Anyone can write to the B2 Native API or our S3 Compatible API, we have tons of integrations that do it, here's a list -> https://www.backblaze.com/b2/integrations.html https://www.backblaze.com/b2/integrations.html
- solarkraft 6y agoI don't think that's what gp meant. Why not use a standard protcol like SFTP?
- toomuchtodo 6y agoBecause the S3 API has become a more popular standard.
- anon102010 6y agowhat python blob storage libraries are using "SFTP"? How is that the "standard"? I literally have NEVER seen SFTP being used for blob storage in any python project - is this a real thing somewhere?
- amiga-workbench 6y agoI don't see how you could cram all of S3's functionality into sftp? How would you configure a lifecycle policy for a file for example? Or generate a signed URL? It seems to me you would only get a very narrow subset of the functionality.
- solarkraft 6y agoI see, I suppose there wasn't a free standard before and S3's API just inofficially became one.
- hartator 6y agoActually excited by this. I was benchmarking S3 vs B2 vs others 2 years ago and I had to give up on B2 because its implementation for performance was so much more difficult. (88 lines vs 36 lines for all others in Ruby)
- simplyinfinity 6y agoSo how many times a month do you have to implement this to be reasonable compared to the cost of the storage?
- hartator 6y agoThis is not an implementation cost issue. It was just super hard to make the code perform well. Like you have to manage client sessions on your side and chose optimizations on your side. Like you have to spread things manually. Which is hard to do. Whereas S3 is maximising your bandwidth with no custom code required. It's not really S3 compatibility that was needed but B2 API wasn't good.
- prirun 6y agoHashBackup (author here) was one of the 1st if not the first B2 integration. I didn't find the B2 API any more difficult to use than the S3 API. It has the same functionality with similar kinds of API requests. The only significant difference is that you have to request an upload URL and download URL, and requests can sometimes return a code to get a new URL when a vault is full or overloaded. There is a price/performance trade-off: B2 has higher request latency than S3, no matter where you are (my experience), but they also are 5x cheaper on storage costs, 10x cheaper on download bandwidth, and have no price gimmicks like minimum object sizes or minimum object lifetimes like many other services (S3 IA for example). To make up for B2's request latency it is more important to issue requests from multiple threads, especially for short-running requests like removing files. Another key difference is that B2 always uses SSL whereas S3 can be accessed without SSL with little security impact because each S3 request is individually signed with a secret key. Setting up an SSL connection is more overhead, so another key to performance is to reuse connections. Both of these suggestions apply to S3 as well, just more to B2 because of the latency difference.
- numbsafari 6y agoI would absolutely love to replace my use of S3 with B2 as a backup for data stored elsewhere. Personally, I would much rather this storage to to a service that only does storage, rather than everything else that AWS does, so I don't have to worry about anything strange happening in a cloud service I don't use every day. When they first launched B2, I inquired about ability to enter into a BAA (Business Associates Agreement) for HIPAA compliance and was told that it wasn't "on the roadmap". It sounds like B2 has come a long way on the compliance side. Would be great if they were open to this.
- atYevP 6y agoYev here -> Double good news for you this morning: we're now signing BAAs for B2 Cloud Storage ;-) Just contact sales and they can get you sorted out!
- numbsafari 6y agoThat's great to hear! I'll definitely be reaching out.
- waffle_ss 6y agoJust a year ago B2 couldn't do server-side file copying.[1] If you wanted to rename or move a file you had to re-upload the whole thing (not great for large multi-gigabyte files)! That ruled them out of consideration for storing my personal backups. Glad to see they've since fixed that, and with this update are clearly continuing to improve ergonomics. I'll have to give B2 a fresh look. [1]: https://github.com/Backblaze/B2_Command_Line_Tool/issues/175 https://github.com/Backblaze/B2_Command_Line_Tool/issues/175
- atYevP 6y agoYev here -> thanks! We're constantly working on making the platform better, and copy-file was definitely a widely requested feature! That plus S3 Compatibility, for folks who wanted to integrated with B2 Cloud Storage but didn't have the resources to write code to our B2 Native API.
- rarrrrrr 6y agoI've been using B2 to disrupt a bunch of ugly & entrenched vendors in the price sensitive K12 market. Thanks for building it. :)
- budmang 6y agoCan you share what you do with B2? (Are you hosting educational content? Something else?)
- atYevP 6y agoAh that's awesome! I'd love to more know! How are you using B2 in general, and does this make it easier for you? If you want to leave a note here, or you can send it to: b2feedback@backblaze.com!
- ajinkyapatil 6y agoare there any plans to host public datasets like aws pds ?
- ksec 6y agoBackblaze is also the founding member of Bandwidth Alliance, meaning getting those B2 via Cloudflare is essentially free. So you are only paying for Storage. ( Correct me if I am on this one ) I wonder why doesn't ALL non HyperScale Cloud Vendors, like Linode and DO provide one click third party backup to B2. You should always store offsite backup somewhere. And B2 is perfect.
- Hamuko 6y agoHow does that work? Can I just dumb two terabytes of video into Backblaze B2, setup a Cloudflare account and have people watch those videos with it costing me only $10 a month? Because that doesn't sound right.
- hartator 6y agoCurious as well.
- varikin 6y agoI believe you still pay Cloudflare costs, but the traffic between Cloudflare and B2 is free on both platforms. But it might be worth double-checking the fine print.
- Hamuko 6y agoIsn't the Cloudflare CDN included in the free plan as well?
- shockinglytrue 6y agoCloudFlare's free plan can and does end at any moment, I wouldn't rely on it for any serious application
- georgyo 6y agoIt is, but they cap out the file size they will cache on the free plan to be 512MB. So you would need to chunk up your videos to get free bandwidth.
- idrock 6y agoOk - beer and a pizza to anyone getting Rails Active Storage up and running with the new API... if I can get to it this weekend I'll post back
- whalesalad 6y agoThis is great. Their current API requires you to identify a unique host to send data to, so you’re constantly performing a metric ton of DNS queries. Until I white listed the base domain it was the #1 client of my Pihole installation by multiple orders of magnitude.
- guenthert 6y agoThe DNS resolver library of your client is allowed to cache the IP address for a given hostname for up to TTL. If it does so, the cost should be negligible.
- deleted 6y ago[deleted]
- brianwski 6y agoDisclaimer: I work at Backblaze. > The DNS resolver library of your client is allowed to cache the IP address for a given hostname for up to TTL Not only that, but one mistake a lot of developers made early on was asking for a location to upload for every upload. That was NEVER the intention. In fact that annoys our servers also. Developers are supposed to request a location to upload ONCE, and then upload to that location for hours, or even DAYS. Unless you have a bug in that software, it really shouldn't come anywhere close to being a high runner in DNS. We're talking 9 or 10 requests per day, at most, if you are unlucky. Feel free to reach out to our support if you aren't seeing that!
- aftbit 6y agoAlthough that implies that you need to cache that location to upload somewhere and request it from your own cache. Also, you need to write error-handling code (which will only fire rarely, so is hard to test) to deal with redirects or fall back to re-fetching the location if it changes. These things are not particularly difficult, but they require additional mind-space to accommodate. Most developers will just do the simplest thing that works, performance be damned. If the simplest path is slow, they'll just remember "B2 is slow", no matter how unfair that is.
- hemancuso 6y agoAs a developer that supports B2 (I write ExpanDrive) I think it’s great that they are moving on from an API that doesn’t expose any extra value. That being said, I wish B2 performance was better. Throughput is dramatically slower than S3.
- eyegor 6y agoWhat region are you moving from/to? Last I checked, b2 only exists in datacenters on the US west coast.
- hemancuso 6y agoIt remains the case even if you’re only a few ms away.
- budmang 6y ago(backblaze ceo here) We also have a region in Europe: https://www.backblaze.com/blog/announcing-our-first-european-data-center/ https://www.backblaze.com/blog/announcing-our-first-european...
- karambir 6y agoPlease consider an Asia/Pacific data center. I am from India and my company was not able to use B2 due to high response times even from European DC. Even a DC in Singapore will be helpful for us. - Thankful Personal Backup Customer
- hannibalhorn 6y agoJust signed up to try it out - would really like to see some form of two factor auth that isn't SMS based. TOTP and/or FIDO U2F.
- ac29 6y agoI've been using TOTP with B2 for ages. I think SMS is just to set up the account.
- hannibalhorn 6y agoAh, the copy next to "Turn on Two Factor" sure sounded like SMS only, but sure enough, it gives you the option to use TOTP later on. Thanks!
- jedberg 6y agoThis is huge because it means you can use things like S3 Fuse to mount your storage. Which means you can use it to extend your local disk, or run your own backups, or whatever. Amusingly the price to store 1.2TB of data is the same as the cost of their backup plan, so if your disk is smaller than that, you could save a few bucks running your own backups. Until you have to restore (from what I can tell restores are free on their backup plans but would cost money on the S3 plan).
- DavideNL 6y ago> Which means you can use it to .... or run your own backups You could, but if i read correctly (s3fs-fuse limitations): "random writes or appends to files require rewriting the entire file". So changing 1 bit of a 10GB file, means re-uploading 10GB. https://github.com/s3fs-fuse/s3fs-fuse#limitations https://github.com/s3fs-fuse/s3fs-fuse#limitations
- gaul 6y agoThis changed in 1.86 and I updated the README as follows: > random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy Now changing one bit means re-uploading 5 MB, the minimum S3 part size.
- tgtweak 6y agoOnly if the blackblaze implementation supports put byte range... Not supported by default.
- mappu 6y agorclone has long had a Fuse mount feature using the original B2 API.
- jedberg 6y agoSure, but S3 Fuse is more mature, more stable, more feature complete, has a lot more usage and therefore a lot more visibility for possible bugs, especially corruption bugs.
- heipei 6y agoNow that we're talking about B2, has or is anyone using them for latency-sensitive small-file object storage? I'm about to take the plunge and set up benchmarks, my use-case is that I want to store and serve ~ 500k small files (30b-1MB) per day to website visitors. So far B2 support has told me that it shouldn't be a problem, and early benchmarking indicates the same, just curious if anyone had stories from the trenches.
- willcodeforfoo 6y agoWe use B2 to store images on Vintage Aerial (https://vintageaerial.com https://vintageaerial.com), both high res scans and all kinds of thumbnail sizes. It is... a little slower than I'd like but with Cloudfront in front it has been manageable. I love tips from Backblaze on how to increase performance there beyond caching to CF.
- haywirez 6y agoI'm seeing horrible TTFB from Europe. Most files are fast but sometimes a request is stuck for 10s or more...
- heipei 6y agoIs that to their US datacenter or the European (Amsterdam) one? So far the European one has been pretty snappy for me, ping is ~20ms from my home ISP (Germany) and TTFB is good enough that I can instantly saturate my 300MBit home ISP line with concurrency of ~ 500 downloading small objects (5b-1MB), but I haven't looked at min/max/avg/median yet. BIG gotcha btw: You have to choose between US and EU when you create your B2 account! You can't have buckets in the other location, so that means you'll need two accounts if you want to do that.
- kstrauser 6y agoAs a Synology user, please let this mean that Hyper Backup can work with B2 now (or at least soon).
- kevstev 6y agoI was trying to see if this was now better than Glacier- and aside from the SLA's being much better in terms of retrieval, I am not sure they make sense for a backup use case- where you are only really planning on downloading that data back down in a worst case scenario. It may depend on what your incremental backups look like as well- mine are negligible- a dump of a few GB of photos after holidays, other records are tiny. Glacier pricing in us-east is .0004 vs .0005 for B2. There is always pricing obfuscation with cloud, but AFAICT, there is no need to move off Glacier for a backup use-case.
- cdumler 6y agoMy two cents is that there is no reason to _use_ Glacier as a backup strategy. Glacier's cost come for restoration: the more you restore and the faster you want to restore it, the more quickly costs rise. It's far better suited for a collection where you're pretty much most of it will never be restored but what you'll need to restore you don't know. Think video, art, music assets for projects. B2's retrieval is far, far lower cost and completely immediate for restoring an entire backup back to a server. If you're not careful that extra .5 cents you save will really cost you on a full restore.
- Dylan16807 6y agoIf you can wait a few hours, the better comparison is probably Glacier Deep Archive, which is not $4/TB/month but $1/TB/month. Amazon wants to charge you $90/TB* to get data to the outside world, compared to B2's $10, but you can mitigate it in various ways. At the low end that's using a lightsail instance as a VPN, depending whether you think the TOS allows that. At the high end it's paying flexify.io $40 to move your data to B2, then paying B2 $10. There might be other ways to improve S3 egress costs. It's a very hard thing to search for. I only learned about flexify from this post. So if you have to restore less than half of your data each year, Glacier Deep storage will save you money. It's worth considering, unlike normal Glacier which is almost entirely downside. * There's also a $2.50/TB fee to get things out of Glacier, but that's dwarfed by the other costs.
- rb808 6y agoIs there a cheap s3 compatible service that is less reliable? I dont want to pay for redudancy Eg its my backups I can handle a 3% chance that my data is lost as long as I find out about it.
- brian_herman__ 6y agoThere is no mention of the durability guarantees that s3 has.
- bigtones 6y agoTheir durability is eleven 9's - 99.999999999% That's the same as Amazon S3. https://help.backblaze.com/hc/en-us/articles/218485257-B2-Resiliency-Durability-and-Availability https://help.backblaze.com/hc/en-us/articles/218485257-B2-Re...
- brianwski 6y agoDisclaimer: I work at Backblaze. > There is no mention of the durability guarantees that s3 has. I wrote this blog post doing some of our math around this: https://www.backblaze.com/blog/cloud-storage-durability/ https://www.backblaze.com/blog/cloud-storage-durability/ But here is the thing: if you value your data, like if you will really go out of business if you lose it, then you should store three copies with AT LEAST two separate vendors. No matter how reliable any one vendor is, "stuff can happen" like your credit card is declined and the vendor deletes all of it. I would recommend you use two separate vendors like Amazon S3 and Backblaze B2, and use two separate credit cards that expire on different cycles. I believe the credentials for login should be different on those two accounts, and the same one employee shouldn't have the credentials to both. Because one disgruntled employee should NOT have the ability to put you out of business. If you want some other thoughts, here is a blog post Backblaze wrote called the "3-2-1 Backup Strategy": https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
- shanemhansen 6y agoI'm curious what their load balancing layer looks like. There's alot of interesting options. (Disclaimer: I've worked in the CDN and the storage space in the past) If their load balancer is smart enough it can call the dispatcher, and make use of something like https://zaiste.net/nginx_x_accel_header/ https://zaiste.net/nginx_x_accel_header/ to figure out where to forward the request. Unfortunately this still requires uploads be proxied through the dispatcher. You could get crazy and involve a CDN (akamai or cloudflare or fastly) that could do some smart logic, especially if you can emit your dispatcher as a lookup table that's updated frequently. I don't know what bandwidth costs would be for that though. Probably high. It's an interesting problem space and I'd love to talk to these folks about it.
- ADefenestrator 6y agoHi! Backblaze employee who did some of the LB stuff here. It's relatively standard/straightforward. There's a L4 load balancing layer using IPVS and ECMP-via-BGP, then a custom application that does the actual proxying/forwarding to the appropriate vault.
- willcodeforfoo 6y agoThis is great news... there are lots more good clients for S3 than B2, and implementing one is less than trivial because of some special considerations B2 had in the beginning (namely: uploading directly to a pod.) I see this isn't available for old buckets, is there a straightforward way to duplicate a bucket to make it compatible or do you have to use something like rclone?
- budmang 6y ago(backblaze ceo here) Yes, easy to move the data to a compatible bucket using our B2 CLI or Transmit: https://help.backblaze.com/hc/en-us/articles/360047120614-How-to-move-Data-from-an-Existing-Bucket-to-a-new-S3-Compatible-Bucket https://help.backblaze.com/hc/en-us/articles/360047120614-Ho...
- sida 6y agoCan Amazon actually patent their API (per the google vs oracle case) - basically like prevent other vendors to provide S3 APIs so that Amazon can lock in users. I am not a lawyer. So this is a genuine / dumb question.
- brianwski 6y agoDisclaimer: I work at Backblaze. I'm also not a lawyer. :-) > Can Amazon actually patent their API (per the google vs oracle case) - basically like prevent other vendors to provide S3 APIs so that Amazon can lock in users. Most likely yes. Backblaze plans going forward are to fully, uncompromisingly maintain our original native B2 APIs for a few reasons including this concern. It's probably up to Amazon whether they want to boot all 3rd parties off their S3 API. Backblaze has a viable fallback if that occurs. I hope for customer's sake Amazon doesn't declare war in that fashion. If Amazon decides on this path, internally at Backblaze we have discussed immediately doing the opposite - declaring for all of time anybody can copy our B2 APIs. Remember, our APIs are technically superior to the S3 APIs. They are lower cost to implement, and are shockingly easier to use for developers. They don't make all the mistakes S3 made. We had the luxury of learning from all their mistakes over the years. :-)
- sida 6y agoThis is kind of scary that companies can patent an interface. So google cloud is actually expected to also have this potential legal time bomb? Amazon can sue you and retroactively force you to pay them right? So all they need to do is to wait for the alternatives to become popular
- swyx 6y agothats very short term thinking though. you win the battle but lose the war by being so partner-hostile. amazon has thousands of partners pay to join it at re:invent for a reason.
- anderspitman 6y agoI suspect a move like that would trigger a cloudpocalpyse that would actually be beneficial in the mid- and long-term.
- mdevere 6y agoEvery day I have to use something like 4gb of data to let Backblaze sync. This is despite the fact that I might only have created/changed 100mb worth of files since the previous day's sync.
- IvanK_net 6y agoIt reminds me a moment three years ago, when I asked Dropbox to make their API similar to Google Drive, as they basically provide the same service. https://github.com/dropbox/dropbox-api-spec/issues/3#issuecomment-320685313 https://github.com/dropbox/dropbox-api-spec/issues/3#issueco... It is just awful to see, how everyone tries to reinvent the wheel and not to be compatible with anyone else.
- tyingq 6y agoFear of lawsuits related to copying APIs may also be a factor. See https://en.wikipedia.org/wiki/Google_v._Oracle_America https://en.wikipedia.org/wiki/Google_v._Oracle_America
- throw_away 6y agoBackblaze is clearly violating Oracle's copyrighted copy of Amazon's S3 API: https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Task...
- haywirez 6y agoThat's awesome, but I really want to see lightning fast response times and TTFB... Second pain point is the number of retries needed for uploading a large batch of small files. Those are the main reasons I'm still considering migrating away. I really wish I shouldn't as otherwise I love the pricing and the philosophy. Edit: also think DigitalOcean Spaces and B2 might be better off merging together, or Spaces being a whitelabel B2 in disguise (both are part of BWA).
- brianwski 6y agoDisclaimer: I work at Backblaze. > I really want to see lightning fast response times and TTFB (Time To First Byte Served) If a file is "cold" (nobody has requested it in the last 24 hours) then it needs to be reconstructed from the Backblaze Vaults and there is a little delay. After that, it should serve pretty fast for the following requests (off of a caching layer with SSDs). In the end, Backblaze B2 is a good solution for some customers, and not ideal for others. If your application requires blinding speed, like sub 1 millisecond serve times, Backblaze B2 may not be perfect for you. But how often is that the case? Certainly not when fetching a web page, or storing a backup for a year, right? In those cases a small delay is FINE. This is an example web page served by Backblaze B2 here, how does it load for you? https://f001.backblazeb2.com/file/ski-epic-c/full/2015_scotland_will_macdonald_birthday_in_duns_castle/index.html https://f001.backblazeb2.com/file/ski-epic-c/full/2015_scotl... Fast? Slow? How is it? For comparison, my regular hosting provider serving the same web page here: https://www.ski-epic.com/2015_scotland_will_macdonald_birthday_in_duns_castle/index.html https://www.ski-epic.com/2015_scotland_will_macdonald_birthd... Personally I can't tell any difference. I still look silly in a kilt in both versions. :-) > Second pain point is the number of retries needed for uploading a large batch of small files. It really shouldn't take any retries, or geez, at VERY MOST something like less than 1% - why is that an issue? Software should handle the tiny failure rate. I'm honestly curious, we want to know why people aren't choosing our solution!!
- heipei 6y agoI understand you're probably not in a position to say anything about it, but I'd love to see the "little delay" when reconstructing a file qualified somewhat. Are we talking < 5s or < 10s? What do the percentiles for restore latency look like? How does file size play into it? This, for me, is one of the biggest unknowns right now since it's not easy to create a test benchmark for this case (i.e. upload a bunch of stuff and let it sit idle for at least 24 hours, hoping it will be expired from the caching layer).
- Waterluvian 6y agoOkay dumb it down for Monday Me. Does this mean I can read from and write to my B2 storage using AWS S3 libraries (like the CLI, Python, and Node libs)?
- jedberg 6y agoYes.
- mahesh_rm 6y agoWould S3cmd cli work as well?
- gjs278 6y agoyes
- zimpenfish 6y agoI tried s3cmd according to their blog post without success. Just kept complaining that the access key was invalid. Which is a shame because `s3cmd` is much easier to use than `b2`.
- atYevP 6y agoYev here -> make sure you ping our b2feedback@backblaze.com address and let us know about that experience, we're writing it all down and keeping tabs on what's not working as intended.
- zimpenfish 6y agoThanks for following up! I eventually got it working - I'd followed the example too closely and had a rogue `us-west-002` left in the config which broke things because I'm apparently on `us-west-000`. But I'll drop an email anyway because I can't see an easy way to see what region you're in other than visually parsing the endpoint URL.
- benbro 6y agoIs B2 suitable for streaming video files? Can I stream the same file to 1,000 viewers at the same time?
- budmang 6y ago(backblaze ceo here) B2 is a great origin store for your video files. If you're streaming to lots of viewers, using a CDN with Backblaze B2 is optimal. We partnered with Cloudflare as a founding member of the Bandwidth Alliance so you can store your videos with B2 and transit them for free to Cloudflare, which can serve to your viewers.
- GeneticGenesis 6y agoPlease correct me if I'm wrong, but my understanding was that Cloudflare should not be used to deliver video files unless using Cloudflare's "stream" product, IE specifically this [1]. [1]: https://community.cloudflare.com/t/cloudflare-how-not-to-violate-the-terms-of-audio-video-static-html-content/101075 https://community.cloudflare.com/t/cloudflare-how-not-to-vio...
- fabiandesimone 6y agoWould love to confirm this.
- bithavoc 6y agoI migrated a client from Cloudinary($1k+ /mo) to B2, a Go+ImageMagick program running in DigitalOcean and Cloudflare CDN for a total of $60 /mo. It’s been running for two years now, B2 has been incredibly reliable.
- atYevP 6y agoYev here -> That's awesome to hear! Glad we can make things more affordable for you and that it's working great!
- idrock 6y agoGiven all the Cloudflare discussion - Cloudflare webinar with Backblaze coming up next week: https://www.brighttalk.com/webcast/14807/405472 https://www.brighttalk.com/webcast/14807/405472
- jszymborski 6y agoOoo, even more reason to set-up a NextCloud instance now! Previously, it wasn't really practical to set-up B2 as external storage because you'd need to also set up a compat layer.
- christefano 6y agoI set this up yesterday, and it was a breeze. Just had to be sure to omit the B2 external storage folder from the backups on my Nextcloud server. Now only if Virtualmin (YC ‘08) supported virtual server backups to S3-compatible B2 cloud storage… There’s an open ticket for this at https://www.virtualmin.com/node/65024 https://www.virtualmin.com/node/65024
- unilynx 6y agoI've looked at B2 from time to time, but doing database blob storage over S3 or to disk and backing up database and files over rsync made us stick to our existing technology (eg Transip cloud storage which also charged 10 EUR/month per 2TB). One thing we didn't look forward to was having to reimplement cataloging and garbage collection for all of disk, S3 and B2, so we just stuck to a rsync hardlinking solution (which makes incremental backups painless) having access to primary storage and cheap backup storage using the same S3 API will make us reconsider that and will probably make it worth the effort to dump our rsync-based solution for B2.
- jjice 6y agoI was actually looking at B2 vs S3 literally 2 days ago and went S3 for the universal API. Luckily, it was a personal project and I I can probably migrate everything very quickly. This is a killer feature, and I bet this will convince a lot of people to move to Backblaze.
- rkrzr 6y agoDoes Backblaze offer strong consistency for files? The killer feature of Google Cloud Storage in my eyes is its ability to be strongly consistent, if you set the right HTTP headers. This is not possible for Amazon S3, which is always eventually consistent and makes it unusable for many use cases where you need to be able to guarantee that customers will always see the newest version of a file.
- kcolford 6y agoI know that the underlying backbaze b2 is strongly consistent because of how they shard it. You get a different download endpoint depending on your bucket corresponding to the data centre. Not sure how they implemented their S3 endpoint though so that will be interesting.
- nilayp 6y agoNilay from Backblaze here. Yes - B2 is strongly consistent. When you upload an object using either the B2 Native or S3 API - the object is persisted to the final resting place before the upload completes. Therefore, you can list/download the file immediately after your upload completes.
- ing33k 6y agoUsed B2 heavily until recently as a origin server for a CDN. Few weeks ago we saw a spike in 502 / 504 responses. When I contacted their customer supported , I was pointed to the following URL where they explain in detail how they handle these errors https://www.backblaze.com/blog/b2-503-500-server-error/ https://www.backblaze.com/blog/b2-503-500-server-error/ Essentially they are not considered as errors and expect the client to retry loading the file. This approach won't work in our use case.
- Shakahs 6y agoIf you are using Cloudflare you could use a Worker script to automatically retry the origin pull and only cache it on success.
- tgtweak 6y agoYou can definitely get read errors and your application should be aware of this and handle this use case. Amazon's own S3 is not immune to this and after years of running spark jobs which shard output across thousands of files on s3 (and load from the same) you'll see these underlying http errors on a daily scale even intra-region. Even if you're doing multi region s3 replication you'll run into this for external clients semi-occasionally.
- TickleSteve 6y agoSo, you're relying on the API being 100% reliable? no errors?
- ing33k 6y agoI not expecting 100 % reliability. but when we get a 503 response it should be considered an an error and acknowledged by the provider that it's an error. in my use case, we were using a CDN which was configured to pull files from B2. When B2 responds with a 503/500 I have no control on the retry mechanism. The error rate was around 5-10%
- nacs 6y agoRetrying a failed network request seems like a normal thing to do -- whether its a network timeout, server error, or whatever random hiccup happened.
- polskibus 6y agoIs there an open source S3-compatible component that I could rollout on prem?
- christophilus 6y agoSwank. One of the reasons I'm not using Backblaze is because I couldn't find a way to generate a private url which allowed secure upload from the browser. It only allowed (so far as I can tell) a private url that had access to an entire bucket. If they've got an S3 compatibility layer now, this problem is solved. I'm gonna invest some time on this tomorrow.
- deleted 6y ago[deleted]
- S3raph 6y agohappy customer of backblaze. Love how transparent they are with everything (especially the Harddisk statistics) and how the CEO takes time to respond to a lot questions only confirms how down to earth they are.
- deleted 6y ago[deleted]
- ghawkescs 6y agoAny chance of adding Azure storage compatible APIs in the future?
- gramakri 6y agoFantastic news. B2 Storage is one of the most requested backup storage backend for us.
- ckdarby 6y agoAnother step towards Amazon acquiring them.
- Aeolun 6y agoOh man, I hope not... I enjoy my independent B2.
- manigandham 6y agoJust switched to Wasabi last week for better pricing and S3 interface... but great to see this.
- sparrc 6y agoDoes this mean I can use awscli to interact with b2 by specifying some backblaze server with --endpoint-url? What is the endpoint I would use?
- cbo100 6y agoWhen you create the bucket it shows a url along with the keys. Not sure how unique that URL is, looking at the structure it could depend what data centre your bucket gets created in.
- tgtweak 6y agoI remember when they had all their servers in one room and the redundancy boiled down to erasure encoding in single servers. They've been doing incredible work in the open (storage server design, hardware reliability data, etc) and I'm really happy they've grown to where they are today.
- kcdipesh 6y agoGreat news. Only a few days ago I was trying to figure out ways to use minio to use backblaze as mattermost cloud storage which needs to be s3 compatible. I expect that will work straight now. Have anyone already tried this integration?
- artellectual 6y agoThis is wonderful news for me. I host a video on demand site Codemy.net and all the original source videos are on backblaze. Originally I had to write a library to connect to the backblaze api. Now I look forward to using the existing aws client libraries, one less thing I have to maintain.
- atYevP 6y agoYev here -> That's awesome to hear! Ease of use is one of the things that we strive for at Backblaze and I'm glad that the S3 Compatible APIs are going to unlock some use-cases and make things easier for people that don't have the bandwidth to maintain different codepaths!
- joshuaellinger 6y agoAny plans for an Azure Blob API?
- siscia 6y agoDid somebody actually tried to use the S3 API? It seems like they are not working for me.