13 ms·

Comparing AWS S3 with Cloudflare R2: Price, Performance and User Experience

226 points by randomint64 2y ago

theryanteodoro 2y ago
love a good comparison!
kevlened 2y ago
It's not mentioned, but important to note, that R2 lacks object versioning. https://community.cloudflare.com/t/r2-object-versioning-and-replication/524025 https://community.cloudflare.com/t/r2-object-versioning-and-...
- UltraSane 2y ago
  Ouch. Object versioning is one of the best features of object storage. It provides excellent protection from malware and human error. My company makes extensive use of versioning and Object Lock for protection from malware and data retention purposes.
  - CharlesW 2y ago
    As @yawnxyz mentioned, versioning is straightforward to do via Workers (untested sample: https://gist.github.com/CharlesWiltgen/84ab145ceda1a972422a85e6ab49a162 https://gist.github.com/CharlesWiltgen/84ab145ceda1a972422a8...), and you can also configure things so any deletes and other modifications must happen through Workers.
    - UltraSane 2y ago
      Interesting, thanks!
- yawnxyz 2y ago
  I built a thin Cloudflare workers script for object versioning and it works great
  - jw1224 2y ago
    Is this something you’d consider sharing? I know many of us would find it really useful!
    - yawnxyz 2y ago
      sure! I'll clean it up a bit and show it on HN. For some reason I figure Cloudflare would have built that by now?!
JOnAgain 2y ago
I _love_ articles like this. Hacker News peeps, please make more!
jzelinskie 2y ago
This is a great comparison and a great step towards pressure to improve cloud service pricing. The magic that moves the region sounds like a dealbreaker for any use cases that aren't public, internet-facing. I use $CLOUD_PROVIDER because I can be in the same regions as customers and know the latency will (for the most part) remain consistent. Has anyone measured latencies from R2 -> AWS/GCP/Azure regions similar to this[0]? Also does anyone know if the R2 supports the CAS operations that so many people are hyped about right now? [0]: https://www.cloudping.co/grid https://www.cloudping.co/grid
- xhkkffbf 2y ago
  This really is a good article. My only issue is that it pretends that the only competition is between Cloudflare and AWS. There are several other low rent storage providers that offer an S3 compatible API. It's also worth looking at Backblaze and Wasabi, for instance. But I don't want to take anything away from this article.
  - sliken 2y ago
    I tinkered with Wasabi, but their minimum bucket lifetime made it not work out for me.
jsheard 2y ago
Is R2 egress actually free, or is it like CFs CDN egress which is "free" until they arbitrarily decide you're using it too much or using it for the wrong things so now you have to pay $undisclosed per GB?
- shivasaxena 2y ago
  I would say don't run a casino on cloudflare
  - deleted 2y ago
    [deleted]
  - MortyWaves 2y ago
    I am also surprised that 4chan is using Cloudflare captcha and bot protection
    - byyll 2y ago
      What is surprising about that? Cloudflare also provides services to terrorists, CSAM websites and more.
      - telgareith 2y ago
        Nice job painting CF as the had guy. They do NOT provide services to such, again and again they have terminated such for breach of TOS and cooperated with the legal system.
        MassiveQuasar 2y ago
        [flagged]
  - dmd 2y ago
    Good to know. Please make an uncontroversial list of all the human activities that you think shouldn't be allowed on cloudflare (or perhaps in general). Then we can all agree to abide by it, and human conflict will end!
breckognize 2y ago
To measure performance the author looked at latency, but most S3 workloads are throughput oriented. The magic of S3 is that it's cheap because it's built on spinning HDDs, which are slow and unreliable individually, but when you have millions of them, you can mask the tail and deliver multi TBs/sec of throughput. It's misleading to look at S3 as a CDN. It's fine for that, but it's real strength is backing the world's data lakes and cloud data warehouses. Those workloads have a lot of data that's often cold, but S3 can deliver massive throughout when you need it. R2 can't do that, and as far as I can tell, isn't trying to. Source: I used to work on S3
- JoshTriplett 2y ago
  Yeah, I'd be interested in the bandwidth as well. Can R2 saturate 10/25/50 gigabit links? Can it do so with single requests, or if not, how many parallel requests does that require?
  - moralestapia 2y ago
    Yes, they absolutely can [1]. 1: https://blog.cloudflare.com/how-cloudflare-auto-mitigated-world-record-3-8-tbps-ddos-attack/ https://blog.cloudflare.com/how-cloudflare-auto-mitigated-wo...
    - fragmede 2y ago
      Cloudflare's paid DDoS protection product being able to soak up insane L3/4 DDoS attacks doesn't answer the question as to whether or not the specific product, R2 from Cloudflare which has free egress is able to saturate a pipe. Cloudflare has the network to do that, but they charge money to do so with their other offerings, so why would they give that to you for free? R2 is not a CDN.
      - moralestapia 2y ago
        [flagged]
deleted 2y ago
[deleted]
suryao 2y ago
Great article. Do you have throughput comparisons? I've found r2 to be highly variable in throughput, especially with concurrent downloads. s3 feels very consistent, but I haven't measured the difference.
- pier25 2y ago
  I'm also interested in upload speeds. I've seen complaints of users about R2 having erratic upload speeds.
vlovich123 2y ago
Very good article and interesting read. I did want to clarify some misconceptions I noted while reading (working from memory so hopefully I don’t get anything wrong myself). > As explained here, Durable Objects are single threaded and thus limited by nature in the throughput they can offer. R2 bucket operations do not use single threaded durable objects but did a one off thing just for R2 to let it run multiple instances even. That’s why the limits were lifted in the open beta. > they mentioned that each zone's assets are sharded across multiple R2 buckets to distribute load which may indicated that a single R2 bucket was not able to handle the load for user-facing traffic. Things may have improve since thought. I would not use this as general advice. Cache Reserve was architected to serve an absurd amount of traffic that almost no customer or application will see. If you’re having that much traffic I’d expect you to be an ENT customer working with their solutions engineers to design your application. > First, R2 is not 100% compatible with the S3 API. One notable missing feature are data-integrity checks with SHA256 checksums. This doesn’t sound right. I distinctly remember when this was implemented for uploading objects. Sha-1 and sha-256 should be supported (don’t remember about crc). For some reason it’s missing from the docs though. The trailer version isn’t supported and likely won’t be for a while though for technical reasons (the workers platform doesn’t support http trailers as it uses http1 internally). Overall compatibility should be pretty decent. The section on “The problem with cross-datacenter traffic” seems to be flawed assumptions rather than data driven. Their own graphs only show that while public buckets have some occasional weird spikes it’s pretty constantly the same performance while the S3 API has more spikeness and time of day variability is much more muted than the CPU variability. Same with the assumption on bandwidth or other limitations of data centers. The more likely explanation would be the S3 auth layer and the time of day variability experienced matches more closely with how that layer works. I don’t know enough of the particulars of this author’s zones to hypothesize but the s3 with layer was always challenging from a perf perspective. > This is really, really, really annoying. For example you know that all your compute instances are in Paris, and you know that Cloudflare has a big datacenter in Paris, so you want your bucket to be in Paris, but you can't. If you are unlucky when creating your bucket, it will be placed in Warsaw or some other place far away and you will have huge latencies for every request. I understand the frustration but there are very good technical and UX reasons this wasn’t done. For example while you may think that “Paris datacenter” is well defined, it isn’t for R2 because unlike S3 your metadata is stored regionally across multiple data centers whereas S3 if I recall correctly uses what they call a region which is a single location broken up into multiple availability zones which are basically isolated power and connectivity domains. This is an availability tradeoff - us-east-1 will never go offline on Cloudflare because it just doesn’t exist - the location hint is the size of the availability region. This is done at both the metadata and storage layers too. The location hint should definitely be followed when you create the bucket but maybe there are bugs or other issues. As others noted throughput data would also have been interesting.
nickjj 2y ago
One thing to think about with S3 is there's use cases where the price is very low which the article didn't mention. For example maybe you have ~500 GB of data across millions of objects that has accumulated over 10 years. You don't even know how many reads or writes you have on a monthly basis because your S3 bill is $11 while your total AWS bill is orders of magnitude more. If you're in a spot like this, moving to R2 to potentially save $7 or whatever it ends up being would end up being a lot more expensive from the engineering costs to do the move. Plus there's old links that might be pointing to a public S3 object which would break if you moved them to another location such as email campaign links, etc..
- philistine 2y ago
  Even simpler: I'm using Glacier Deep Archive for my personal backups, and I don't see how R2 would be cheaper for me.
  - telgareith 2y ago
    Have you priced the retrieval cost? You quickly run into high 3 and then 4 figures worth of bandwidth.
    - philistine 2y ago
      Retrieval? For an external backup? If I need to restore and my local backup is completely down, it either means I lost two drives (very unlikely) or the house is a calcinated husk and at this point I'm insured. And let's be honest. If the house burns down, the computers are the third thing I get out of there after the wife and the dog. My external backup is peace of mind, nothing more. I don't ever expect to need it in my lifetime.
    - seized 2y ago
      Yes, but if it's your third location of 3-2-1 then it can also make sense to weigh it against data recovery costs on damaged hardware. I backup to Glacier as well. For me to need to pull from it (and pay that $90/TB or so) means I've lost more than two drives in a historically very reliable RAIDZ2 pool, or lost my NAS entirely. I'll pay $90/TB over unknown $$$$ for a data recovery from burned/flooded/fried/failed disks.
bassp 2y ago
Only tangentially related to the article, but I’ve never understood how R2 offers 11 9s of durability. I trust that S3 offers 11 9s because Amazon has shown, publicly, that they care a ton about designing reliable, fault tolerant, correct systems (eg Shardstore and Shuttle) Cloudflare’s documentation just says “we offer 11 9s, same as S3”, and that’s that. It’s not that I don’t believe them but… how can a smaller organization make the same guarantees? It implies to me that either Amazon is wasting a ton of money on their reliability work (possible) or that cloudflare’s 11 9s guarantee comes with some asterisks.
- rat9988 2y ago
  What makes you think it did cost aws that much moneu at their scale to achieve 11 9s that cloudflare cannot afford it?
  - bassp 2y ago
    Minimally, the two examples I cited: Shardstore and Shuttle. The former is a (lightweight) formally verified key value store used by S3, and the latter is a model checker for concurrent rust code. Amazon has an entire automated reasoning group (researchers who mostly work on formal methods) working specifically on S3. As far as I’m aware, nobody at cloudflare is doing similar work for R2. If they are, they’re certainly not publishing! Money might not be the bottleneck for cloudflare though, you’re totally right
    - zild3d 2y ago
      S3 has touted 11 9's for many years, so before shardstore definitely. The 11 9's is for durability, which is really more about the redundancy setup, erasure coding, etc. (https://cloud.google.com/blog/products/storage-data-transfer/understanding-cloud-storage-11-9s-durability-target https://cloud.google.com/blog/products/storage-data-transfer...) fwiw availability is 4 9's (https://aws.amazon.com/s3/storage-classes/ https://aws.amazon.com/s3/storage-classes/)
      - bassp 2y ago
cube2222 2y ago
R2 and its pricing is quite fantastic. We’re using it to power the OpenTofu Provider&Modules Registry[0][1] and it’s honestly been nothing but a great experience overall. [0]: https://registry.opentofu.org https://registry.opentofu.org [1]: https://github.com/opentofu/registry https://github.com/opentofu/registry Disclaimer: CloudFlare did sponsor us their business plan so we got access to higher-tier functionality
deanCommie 2y ago
The innovator's dilemma is really interesting. Whenever a new incumbent gets on the scene offering the same thing as some entrenched leader only better, faster, and cheaper, the standard response is "Yeah but it's less reliable. This may be fine for startups but if you're <enterprise|government|military|medical|etc>, you gotta stick with the tried tested and true <leader>" You see this in almost every discussion of Cloudflare, which seems to be rapidly rebuilding a full cloud, in direct competition with AWS specifically. (I guess it wants to be evaluated as a fellow leader, not an also-ran like GCP/Azure fighting for 2nd place) The thing is, all the points are right. Cloudflare IS different - by using exclusively edge networks and tying everything to CDNs, it's both a strength and a weakness. There's dozens of reasons to be critical of them and dozens more to explain why you'd trust AWS more. But I can't help but wonder that surely the same happened (i wasn't on here, or really tech-aware enough) when S3 and EC2 came on the scene. I'm sure everyone said it was unreliable, uncertain, and had dozens of reasons why people should stick with (I can only presume - VMWare, IBM, Oracle, etc?) This is all a shallow observation though. Here's my real question, though. How does one go deeper and evaluate what is real disruption and what is fluff. Does Cloudflare have something that's unique and different that demonstrates a new world for cloud services I can't even imagine right now, as AWS did before it. Or does AWS have a durable advantage and benefits that will allow it to keep being #1 indefinitely? (GCP and Azure, as I see it, are trying to compete on specific slices of merit. GCP is all-in on 'portability', that's why they came up with Kubernetes to devalue the idea of any one public cloud, and make workloads cross-platform across all clouds and on-prem. Azure seems to be competitive because of Microsoft's otherwise vertical integration with business/windows/office, and now AI services). Cloudflare is the only one that seems to show up over and over again and say "hey you know that thing that you think is the best cloud service? We made it cheaper, faster, and with nicer developer experience." That feels really hard to ignore. But also seems really easy to market only-semi-honestly by hand-waving past the hard stuff at scale.
- everfrustrated 2y ago
  Cloudflares architecture is driven purely by their history of being a CDN and trying to find new product lines to generate new revenue streams to keep share price up. You wouldn't build a cloud from scratch in this way.
deleted 2y ago
[deleted]
orf 2y ago
My experience: I put parquet files on R2, but HTTP Range requests were failing. 50% of the time it would work, and 50% of the time it would return all of the content and not the subset requested. That’s a nightmare to debug, given that software expects it to work consistently or not work at all. Seems like a bug. Had to crawl through documentation to find out the only support is on Discord (??), so I had to sign up. Go through some more hoops and eventually get to a channel where I received a prompt reply: it’s not an R2 issue, it’s “expected behaviour due to an issue with “the CDN service”. I mean, sure. On a technical level. But I shoved some data into your service and basic standard HTTP semantics where intermittently not respected: that’s a bug in your service, even if the root cause is another team. None of this is documented anywhere, even if it is “expected”. Searching for [1] “r2 http range” shows I’m not the only one surprised Not impressed, especially as R2 seems ideal for serving Parquet data for small projects. This and the janky UI plus weird restrictions makes the entire product feel distinctly half finished and not a serious competitor. 1. https://www.google.com/search?q=r2+http+range&ie=UTF-8&oe=UTF-8&hl=en-gb&client=safari https://www.google.com/search?q=r2+http+range&ie=UTF-8&oe=UT...
- saurik 2y ago
  > given that software expects it to work consistently or not work at all I mean... that's wrong? If you come across such software, do you at least file a bug?
  - orf 2y ago
    Of course not, and it’s completely correct behaviour: if a server advertises it supports Range requests for a given URL, it’s expected to support it. Garbage in, garbage out. It’s not clear how you’d expect to handle a webserver trying to send you 1Gb of data after you asked for a specific 10kb range other than aborting.
    - saurik 2y ago
      "Conversely, a client MUST NOT assume that receiving an Accept-Ranges field means that future range requests will return partial responses. The content might change, the server might only support range requests at certain times or under certain conditions, or a different intermediary might process the next request." -- RFC 9110
MobileVet 2y ago
One thing that I haven't seen discussed in the comments is the inherent vulnerability of S3 pricing. Like all things AWS, if something goes sideways, you are suddenly on the wrong side of a very large bill. For instance, someone can easily blow your egress charges through the roof by making a massive amount of requests for your assets hosted there. While Cloudflare may reach out and say 'you should be on enterprise' when that happens on R2, the fact they also handle DDoS and similar attacks as part of their offering means the likelihood of success is much lower (as is the final bill).
- sroussey 2y ago
  Cloudflare has DDOS roots and it plays well here.
- akira2501 2y ago
  Typically you would use S3 with CloudFront for hosting. S3 provides no protections because it's meant to be a durable and global service. CloudFront provides DDoS and other types of protection while making it easy to get prepaid bandwidth discounts.
  - danielheath 2y ago
    Just one data point, but adding Cloudflare to our stack (in front of "CloudFront with bandwidth discounts") took about $30k USD per year off our bandwidth bill.
  - jgalt212 2y ago
    Yes, obviously. But just as obviously is there's rarely an easy and safe path with AWS. By default, R2 is easy, safe, and cheaper.
  - kmos17 2y ago
    In my experience the AWS waf and ddos mitigation are really expensive (min $40k per year contract) and are missing really basic ddos handling capabilities (last I evaluated it they did not have the ability to enforce client js validation which can be very effective against some bot networks). Maybe it has evolved since but Cloudflare enterprise was cheaper and more capable out of the box.
viraptor 2y ago
IAM gets only a tiny mention as not present, therefore making R2 simpler. But also... IAM is missing and a lot of interesting use cases are not possible there. No access by path, no 2fa enforcing, no easy SSO management, no blast radius limits - just would you like a token which can write a file, but also delete everything? This is also annoying for their zone management for the same reason.
- eddd-ddde 2y ago
  Usually you'd delegate that to a CF worker. Your logic can be fully implemented by you. No single user gets a direct access token, but rather uses your preferred auth mechanism.
  - viraptor 2y ago
    Sure, you could. But that means every team effectively writing their own custom authz instead of a common description language. And then paying extra per call for it.
pier25 2y ago
> you can't chose the location of your R2 bucket! Yeah this is really annoying. That and replication to multiple regions is the reason we're not using R2. Global replication was a feature announced in 2021 but still hasn't happened: > R2 will replicate data across multiple regions and support jurisdictional restrictions, giving businesses the ability to control where their data is stored to meet their local and global needs. https://www.cloudflare.com/press-releases/2021/cloudflare-announces-r2-storage/ https://www.cloudflare.com/press-releases/2021/cloudflare-an...
tlarkworthy 2y ago
I've benchmarked R2 and S3 and S3 is well ahead in terms of latency especially on ListObject requests. I think R2 has come kind of concurrency limit as concurrent ListObject requests seem to to have increase failure rate when serving simultaneous requests I have a few of the S3-like wired up live over the internet you can try yourself in your browser. Backblaze is surprisingly performant which I did not expect (S3 is still king though) https://observablehq.com/@tomlarkworthy/mps3-vendor-examples https://observablehq.com/@tomlarkworthy/mps3-vendor-examples
- kmos17 2y ago
  thanks for sharing, that’s a good data point in comparing those services.
postatic 2y ago
I do mostly CRUD apps with Laravel and Vue. Nothing too complicated. Allows users to post stuff with images and files. I’ve moved ALL of my files from S3 to R2 in the past 2 years. It’s been slow as any migrations are but painless. But most importantly for an indie dev like me the cost became $0.
snihalani 2y ago
>Generally, R2's user experience is way better and simpler than S3. As always with AWS, you need 5 certifications and 3 months to securely deploy a bucket. +1
asteroidburger 2y ago
“Here’s a bunch of great things about CloudFlare R2 - and please buy my book about it” leaves a bad taste in my mouth. Also, has CF improved their stance around hosting hate groups? They have strongly resisted pressure to stop hosting/supporting hate sites like 8chan and Kiwifarms, and only stopped reluctantly.
- gjsman-1000 2y ago
  I don’t have to support 8chan or KiwiFarms to say that Cloudflare has absolutely no role in policing the internet. The job of policing the internet is for the police. If it’s illegal, let them investigate.
  - asteroidburger 2y ago
    There is a difference between policing the internet and supplying resources and services to known bad actors. Their job isn’t to investigate and punish harassment and criminal behavior, but they certainly don’t have to condone it via their support.
    - gjsman-1000 2y ago
      > known bad actors If they are known bad actors, let the police do the job of policing the internet. Otherwise, all bad actors are ultimately arbitrarily defined. Who said they are known bad actors? What does that even mean? Why does that person determining bad actors get their authority? Were they duly elected? Or did one of hundreds of partisan NGOs claim this? Who elected the NGO? Does PETA get a say on bad actors? Be careful what you wish for. In some US States, I am sure the attorney general would send a letter saying to shut down the marijuana dispensary - they're known bad actors, after all. They might not win a lawsuit, but winning the support of private organizations would be just as good. > they certainly don’t have to condone it via their support Wow, what a great argument. Hacker News supports all arguments here by tolerating people speaking and not deleting everything they could possibly disagree with. Or maybe, providing a service to someone, should not be seen as condoning all possible uses of the service. Just because water can be used to waterboard someone, doesn't mean Walmart should be checking IDs for water purchasers. Just because YouTube has information on how to pick locks, does not mean YouTube should be restricted to adults over 21 on a licensed list of people entrusted with lock-picking knowledge.
karmakaze 2y ago
At one company we were uploading videos to S3 and finding a lot of errors or stalls in the process. That led to evaluating GCP and Azure. I found that Azure had the most consistent (least variance) in upload durations and better pricing. We ended up using GCP for other reasons like resumable uploads (IIRC). AWS now supports appending to S3 objects which might have worked to avoid upload stalls. CloudFront for us at the time was overpriced.
kansi 2y ago
I have tried to find a CDN provider which would offer access control similar to Cloudfront's signed cookies but failed to find something that would match it. This is a major drawback with these providers offering S3 style bucket storage because most of time you would want to serve the content from a CDN and offloading access control to CDN via cookies makes life so much easier. You only need to set the cookies for the user's session once and they are automatically sent (by the web browser) to the CDN with no additional work needed
- saurik 2y ago
  This is supported by Google Cloud using literally the same wording: https://cloud.google.com/cdn/docs/using-signed-cookies https://cloud.google.com/cdn/docs/using-signed-cookies As far as I can tell, this feature is also supported by Akamai here: https://techdocs.akamai.com/property-mgr/docs/cookie-authz https://techdocs.akamai.com/property-mgr/docs/cookie-authz I am pretty sure you can implement this on CDNetworks using eval_func: https://docs.cdnetworks.com/en/cdn/docs/recipes/secure-delivery https://docs.cdnetworks.com/en/cdn/docs/recipes/secure-deliv... With AWS Cloudfront, I'd think you--worst case--pull out Lambda@Edge?
  - donavanm 2y ago
    Cloudfront “signed cookie” auth should “just work”: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-cookies.html https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope... IIRC its essentially the same as the aws style signed urls and header bearer token auth. I _think_ lambda@edge is only relevant if you want to do the initial sig generation in the cdn instead of your api/app endpoint. Edit: actually GP mentioned Cloudfront already, so yes works as theyre asking for AFAICT
denysvitali 2y ago
No mention of Backblaze's B2? It's cheaper than these two at just 6$/TB
- russelg 2y ago
  It's also dogshit slow, and only available in 2 regions. At least you can choose the region, unlike R2...
  - pdimitar 2y ago
    How slow we're talking about?
dxxvi 2y ago
I have 250GB on S3. To get them out and store in R2, AWS will charge me 9 cents * 250 ~ $24: ouch.
- mythz 2y ago
  Thanks to the EU AWS now offers "Free data transfer out to internet when moving out of AWS" [1] [1] https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-internet-when-moving-out-of-aws/ https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-i...
matteocontrini 2y ago
I don't know what people use object storage for, but R2 is missing lots of features and it's not a replacement for S3 in many cases. For example: no regions, no replication (and no AZs either), limited lifecycle management, no versioning, no MFA protection, no intelligent tiering, no customer encryption, no IAM, etc.
- remram 2y ago
  Apologies for the tangent, but I always wondered: Why does anyone use "customer encryption" on the server rather than customer side?
mherrmann 2y ago
Reading around online, it seems egress is free until Cloudflare asks for big bucks or threatens to terminate your service. It's probably fine for hobbyists or small web sites. But as a business, I would never trust something that is "free". There has to be a catch somewhere. A business contact of mine asked Cloudflare for a quote for setting up a CDN for 7B requests per month. They dragged him through five sales calls to eventually deliver an offer with request costs 30% above CloudFront's public pricing. He said the costs per GB were ok though. The cheapest reliable CDN I've found is bunny.net. Unlike OP, who is selling a book about Cloudflare, I have no conflict of interest by recommending Bunny, other than being a customer. I serve around 100 TB per month through them, at $0.005/GB. You can put it in front of a bucket to have cheap (globally distributed) egress with all the benefits of your favorite object storage provider. In my case, the buckets lie on Linode/Akamai. Bunny uses CDN77's infrastructure, which I have also heard good things about.
- sieabahlpark 2y ago
  [dead]
Havoc 2y ago
S3 is still hdd? Cached at least?

226 points by randomint64 2y ago

love a good comparison!

It's not mentioned, but important to note, that R2 lacks object versioning. https://community.cloudflare.com/t/r2-object-versioning-and-replication/524025 https://community.cloudflare.com/t/r2-object-versioning-and-...

UltraSane 2y ago
Ouch. Object versioning is one of the best features of object storage. It provides excellent protection from malware and human error. My company makes extensive use of versioning and Object Lock for protection from malware and data retention purposes.
- CharlesW 2y ago
  As @yawnxyz mentioned, versioning is straightforward to do via Workers (untested sample: https://gist.github.com/CharlesWiltgen/84ab145ceda1a972422a85e6ab49a162 https://gist.github.com/CharlesWiltgen/84ab145ceda1a972422a8...), and you can also configure things so any deletes and other modifications must happen through Workers.
  - UltraSane 2y ago
    Interesting, thanks!
yawnxyz 2y ago
I built a thin Cloudflare workers script for object versioning and it works great
- jw1224 2y ago
  Is this something you’d consider sharing? I know many of us would find it really useful!
  - yawnxyz 2y ago
    sure! I'll clean it up a bit and show it on HN. For some reason I figure Cloudflare would have built that by now?!

JOnAgain 2y ago

I _love_ articles like this. Hacker News peeps, please make more!

jzelinskie 2y ago

This is a great comparison and a great step towards pressure to improve cloud service pricing. The magic that moves the region sounds like a dealbreaker for any use cases that aren't public, internet-facing. I use $CLOUD_PROVIDER because I can be in the same regions as customers and know the latency will (for the most part) remain consistent. Has anyone measured latencies from R2 -> AWS/GCP/Azure regions similar to this[0]? Also does anyone know if the R2 supports the CAS operations that so many people are hyped about right now? [0]: https://www.cloudping.co/grid https://www.cloudping.co/grid

xhkkffbf 2y ago
This really is a good article. My only issue is that it pretends that the only competition is between Cloudflare and AWS. There are several other low rent storage providers that offer an S3 compatible API. It's also worth looking at Backblaze and Wasabi, for instance. But I don't want to take anything away from this article.
- sliken 2y ago
  I tinkered with Wasabi, but their minimum bucket lifetime made it not work out for me.

jsheard 2y ago

Is R2 egress actually free, or is it like CFs CDN egress which is "free" until they arbitrarily decide you're using it too much or using it for the wrong things so now you have to pay $undisclosed per GB?

shivasaxena 2y ago
I would say don't run a casino on cloudflare
- deleted 2y ago
  [deleted]
- MortyWaves 2y ago
  I am also surprised that 4chan is using Cloudflare captcha and bot protection
  - byyll 2y ago
    What is surprising about that? Cloudflare also provides services to terrorists, CSAM websites and more.
    - telgareith 2y ago
      Nice job painting CF as the had guy. They do NOT provide services to such, again and again they have terminated such for breach of TOS and cooperated with the legal system.
      - MassiveQuasar 2y ago
        [flagged]
- dmd 2y ago
  Good to know. Please make an uncontroversial list of all the human activities that you think shouldn't be allowed on cloudflare (or perhaps in general). Then we can all agree to abide by it, and human conflict will end!

breckognize 2y ago

To measure performance the author looked at latency, but most S3 workloads are throughput oriented. The magic of S3 is that it's cheap because it's built on spinning HDDs, which are slow and unreliable individually, but when you have millions of them, you can mask the tail and deliver multi TBs/sec of throughput. It's misleading to look at S3 as a CDN. It's fine for that, but it's real strength is backing the world's data lakes and cloud data warehouses. Those workloads have a lot of data that's often cold, but S3 can deliver massive throughout when you need it. R2 can't do that, and as far as I can tell, isn't trying to. Source: I used to work on S3

JoshTriplett 2y ago
Yeah, I'd be interested in the bandwidth as well. Can R2 saturate 10/25/50 gigabit links? Can it do so with single requests, or if not, how many parallel requests does that require?
- moralestapia 2y ago
  Yes, they absolutely can [1]. 1: https://blog.cloudflare.com/how-cloudflare-auto-mitigated-world-record-3-8-tbps-ddos-attack/ https://blog.cloudflare.com/how-cloudflare-auto-mitigated-wo...
  - fragmede 2y ago
    Cloudflare's paid DDoS protection product being able to soak up insane L3/4 DDoS attacks doesn't answer the question as to whether or not the specific product, R2 from Cloudflare which has free egress is able to saturate a pipe. Cloudflare has the network to do that, but they charge money to do so with their other offerings, so why would they give that to you for free? R2 is not a CDN.
    - moralestapia 2y ago
      [flagged]

deleted 2y ago

[deleted]

suryao 2y ago

Great article. Do you have throughput comparisons? I've found r2 to be highly variable in throughput, especially with concurrent downloads. s3 feels very consistent, but I haven't measured the difference.

pier25 2y ago
I'm also interested in upload speeds. I've seen complaints of users about R2 having erratic upload speeds.

vlovich123 2y ago

Very good article and interesting read. I did want to clarify some misconceptions I noted while reading (working from memory so hopefully I don’t get anything wrong myself). > As explained here, Durable Objects are single threaded and thus limited by nature in the throughput they can offer. R2 bucket operations do not use single threaded durable objects but did a one off thing just for R2 to let it run multiple instances even. That’s why the limits were lifted in the open beta. > they mentioned that each zone's assets are sharded across multiple R2 buckets to distribute load which may indicated that a single R2 bucket was not able to handle the load for user-facing traffic. Things may have improve since thought. I would not use this as general advice. Cache Reserve was architected to serve an absurd amount of traffic that almost no customer or application will see. If you’re having that much traffic I’d expect you to be an ENT customer working with their solutions engineers to design your application. > First, R2 is not 100% compatible with the S3 API. One notable missing feature are data-integrity checks with SHA256 checksums. This doesn’t sound right. I distinctly remember when this was implemented for uploading objects. Sha-1 and sha-256 should be supported (don’t remember about crc). For some reason it’s missing from the docs though. The trailer version isn’t supported and likely won’t be for a while though for technical reasons (the workers platform doesn’t support http trailers as it uses http1 internally). Overall compatibility should be pretty decent. The section on “The problem with cross-datacenter traffic” seems to be flawed assumptions rather than data driven. Their own graphs only show that while public buckets have some occasional weird spikes it’s pretty constantly the same performance while the S3 API has more spikeness and time of day variability is much more muted than the CPU variability. Same with the assumption on bandwidth or other limitations of data centers. The more likely explanation would be the S3 auth layer and the time of day variability experienced matches more closely with how that layer works. I don’t know enough of the particulars of this author’s zones to hypothesize but the s3 with layer was always challenging from a perf perspective. > This is really, really, really annoying. For example you know that all your compute instances are in Paris, and you know that Cloudflare has a big datacenter in Paris, so you want your bucket to be in Paris, but you can't. If you are unlucky when creating your bucket, it will be placed in Warsaw or some other place far away and you will have huge latencies for every request. I understand the frustration but there are very good technical and UX reasons this wasn’t done. For example while you may think that “Paris datacenter” is well defined, it isn’t for R2 because unlike S3 your metadata is stored regionally across multiple data centers whereas S3 if I recall correctly uses what they call a region which is a single location broken up into multiple availability zones which are basically isolated power and connectivity domains. This is an availability tradeoff - us-east-1 will never go offline on Cloudflare because it just doesn’t exist - the location hint is the size of the availability region. This is done at both the metadata and storage layers too. The location hint should definitely be followed when you create the bucket but maybe there are bugs or other issues. As others noted throughput data would also have been interesting.

nickjj 2y ago

One thing to think about with S3 is there's use cases where the price is very low which the article didn't mention. For example maybe you have ~500 GB of data across millions of objects that has accumulated over 10 years. You don't even know how many reads or writes you have on a monthly basis because your S3 bill is $11 while your total AWS bill is orders of magnitude more. If you're in a spot like this, moving to R2 to potentially save $7 or whatever it ends up being would end up being a lot more expensive from the engineering costs to do the move. Plus there's old links that might be pointing to a public S3 object which would break if you moved them to another location such as email campaign links, etc..

philistine 2y ago
Even simpler: I'm using Glacier Deep Archive for my personal backups, and I don't see how R2 would be cheaper for me.
- telgareith 2y ago
  Have you priced the retrieval cost? You quickly run into high 3 and then 4 figures worth of bandwidth.
  - philistine 2y ago
    Retrieval? For an external backup? If I need to restore and my local backup is completely down, it either means I lost two drives (very unlikely) or the house is a calcinated husk and at this point I'm insured. And let's be honest. If the house burns down, the computers are the third thing I get out of there after the wife and the dog. My external backup is peace of mind, nothing more. I don't ever expect to need it in my lifetime.
  - seized 2y ago
    Yes, but if it's your third location of 3-2-1 then it can also make sense to weigh it against data recovery costs on damaged hardware. I backup to Glacier as well. For me to need to pull from it (and pay that $90/TB or so) means I've lost more than two drives in a historically very reliable RAIDZ2 pool, or lost my NAS entirely. I'll pay $90/TB over unknown $$$$ for a data recovery from burned/flooded/fried/failed disks.

bassp 2y ago

Only tangentially related to the article, but I’ve never understood how R2 offers 11 9s of durability. I trust that S3 offers 11 9s because Amazon has shown, publicly, that they care a ton about designing reliable, fault tolerant, correct systems (eg Shardstore and Shuttle) Cloudflare’s documentation just says “we offer 11 9s, same as S3”, and that’s that. It’s not that I don’t believe them but… how can a smaller organization make the same guarantees? It implies to me that either Amazon is wasting a ton of money on their reliability work (possible) or that cloudflare’s 11 9s guarantee comes with some asterisks.

rat9988 2y ago
What makes you think it did cost aws that much moneu at their scale to achieve 11 9s that cloudflare cannot afford it?
- bassp 2y ago
  Minimally, the two examples I cited: Shardstore and Shuttle. The former is a (lightweight) formally verified key value store used by S3, and the latter is a model checker for concurrent rust code. Amazon has an entire automated reasoning group (researchers who mostly work on formal methods) working specifically on S3. As far as I’m aware, nobody at cloudflare is doing similar work for R2. If they are, they’re certainly not publishing! Money might not be the bottleneck for cloudflare though, you’re totally right
  - zild3d 2y ago
    S3 has touted 11 9's for many years, so before shardstore definitely. The 11 9's is for durability, which is really more about the redundancy setup, erasure coding, etc. (https://cloud.google.com/blog/products/storage-data-transfer/understanding-cloud-storage-11-9s-durability-target https://cloud.google.com/blog/products/storage-data-transfer...) fwiw availability is 4 9's (https://aws.amazon.com/s3/storage-classes/ https://aws.amazon.com/s3/storage-classes/)
    - bassp 2y ago

cube2222 2y ago

R2 and its pricing is quite fantastic. We’re using it to power the OpenTofu Provider&Modules Registry[0][1] and it’s honestly been nothing but a great experience overall. [0]: https://registry.opentofu.org https://registry.opentofu.org [1]: https://github.com/opentofu/registry https://github.com/opentofu/registry Disclaimer: CloudFlare did sponsor us their business plan so we got access to higher-tier functionality

deanCommie 2y ago

The innovator's dilemma is really interesting. Whenever a new incumbent gets on the scene offering the same thing as some entrenched leader only better, faster, and cheaper, the standard response is "Yeah but it's less reliable. This may be fine for startups but if you're <enterprise|government|military|medical|etc>, you gotta stick with the tried tested and true <leader>" You see this in almost every discussion of Cloudflare, which seems to be rapidly rebuilding a full cloud, in direct competition with AWS specifically. (I guess it wants to be evaluated as a fellow leader, not an also-ran like GCP/Azure fighting for 2nd place) The thing is, all the points are right. Cloudflare IS different - by using exclusively edge networks and tying everything to CDNs, it's both a strength and a weakness. There's dozens of reasons to be critical of them and dozens more to explain why you'd trust AWS more. But I can't help but wonder that surely the same happened (i wasn't on here, or really tech-aware enough) when S3 and EC2 came on the scene. I'm sure everyone said it was unreliable, uncertain, and had dozens of reasons why people should stick with (I can only presume - VMWare, IBM, Oracle, etc?) This is all a shallow observation though. Here's my real question, though. How does one go deeper and evaluate what is real disruption and what is fluff. Does Cloudflare have something that's unique and different that demonstrates a new world for cloud services I can't even imagine right now, as AWS did before it. Or does AWS have a durable advantage and benefits that will allow it to keep being #1 indefinitely? (GCP and Azure, as I see it, are trying to compete on specific slices of merit. GCP is all-in on 'portability', that's why they came up with Kubernetes to devalue the idea of any one public cloud, and make workloads cross-platform across all clouds and on-prem. Azure seems to be competitive because of Microsoft's otherwise vertical integration with business/windows/office, and now AI services). Cloudflare is the only one that seems to show up over and over again and say "hey you know that thing that you think is the best cloud service? We made it cheaper, faster, and with nicer developer experience." That feels really hard to ignore. But also seems really easy to market only-semi-honestly by hand-waving past the hard stuff at scale.

everfrustrated 2y ago
Cloudflares architecture is driven purely by their history of being a CDN and trying to find new product lines to generate new revenue streams to keep share price up. You wouldn't build a cloud from scratch in this way.

deleted 2y ago

[deleted]

orf 2y ago

My experience: I put parquet files on R2, but HTTP Range requests were failing. 50% of the time it would work, and 50% of the time it would return all of the content and not the subset requested. That’s a nightmare to debug, given that software expects it to work consistently or not work at all. Seems like a bug. Had to crawl through documentation to find out the only support is on Discord (??), so I had to sign up. Go through some more hoops and eventually get to a channel where I received a prompt reply: it’s not an R2 issue, it’s “expected behaviour due to an issue with “the CDN service”. I mean, sure. On a technical level. But I shoved some data into your service and basic standard HTTP semantics where intermittently not respected: that’s a bug in your service, even if the root cause is another team. None of this is documented anywhere, even if it is “expected”. Searching for [1] “r2 http range” shows I’m not the only one surprised Not impressed, especially as R2 seems ideal for serving Parquet data for small projects. This and the janky UI plus weird restrictions makes the entire product feel distinctly half finished and not a serious competitor. 1. https://www.google.com/search?q=r2+http+range&ie=UTF-8&oe=UTF-8&hl=en-gb&client=safari https://www.google.com/search?q=r2+http+range&ie=UTF-8&oe=UT...

saurik 2y ago
> given that software expects it to work consistently or not work at all I mean... that's wrong? If you come across such software, do you at least file a bug?
- orf 2y ago
  Of course not, and it’s completely correct behaviour: if a server advertises it supports Range requests for a given URL, it’s expected to support it. Garbage in, garbage out. It’s not clear how you’d expect to handle a webserver trying to send you 1Gb of data after you asked for a specific 10kb range other than aborting.
  - saurik 2y ago
    "Conversely, a client MUST NOT assume that receiving an Accept-Ranges field means that future range requests will return partial responses. The content might change, the server might only support range requests at certain times or under certain conditions, or a different intermediary might process the next request." -- RFC 9110

MobileVet 2y ago

One thing that I haven't seen discussed in the comments is the inherent vulnerability of S3 pricing. Like all things AWS, if something goes sideways, you are suddenly on the wrong side of a very large bill. For instance, someone can easily blow your egress charges through the roof by making a massive amount of requests for your assets hosted there. While Cloudflare may reach out and say 'you should be on enterprise' when that happens on R2, the fact they also handle DDoS and similar attacks as part of their offering means the likelihood of success is much lower (as is the final bill).

sroussey 2y ago
Cloudflare has DDOS roots and it plays well here.
akira2501 2y ago
Typically you would use S3 with CloudFront for hosting. S3 provides no protections because it's meant to be a durable and global service. CloudFront provides DDoS and other types of protection while making it easy to get prepaid bandwidth discounts.
- danielheath 2y ago
  Just one data point, but adding Cloudflare to our stack (in front of "CloudFront with bandwidth discounts") took about $30k USD per year off our bandwidth bill.
- jgalt212 2y ago
  Yes, obviously. But just as obviously is there's rarely an easy and safe path with AWS. By default, R2 is easy, safe, and cheaper.
- kmos17 2y ago
  In my experience the AWS waf and ddos mitigation are really expensive (min $40k per year contract) and are missing really basic ddos handling capabilities (last I evaluated it they did not have the ability to enforce client js validation which can be very effective against some bot networks). Maybe it has evolved since but Cloudflare enterprise was cheaper and more capable out of the box.

viraptor 2y ago

IAM gets only a tiny mention as not present, therefore making R2 simpler. But also... IAM is missing and a lot of interesting use cases are not possible there. No access by path, no 2fa enforcing, no easy SSO management, no blast radius limits - just would you like a token which can write a file, but also delete everything? This is also annoying for their zone management for the same reason.

eddd-ddde 2y ago
Usually you'd delegate that to a CF worker. Your logic can be fully implemented by you. No single user gets a direct access token, but rather uses your preferred auth mechanism.
- viraptor 2y ago
  Sure, you could. But that means every team effectively writing their own custom authz instead of a common description language. And then paying extra per call for it.

pier25 2y ago

> you can't chose the location of your R2 bucket! Yeah this is really annoying. That and replication to multiple regions is the reason we're not using R2. Global replication was a feature announced in 2021 but still hasn't happened: > R2 will replicate data across multiple regions and support jurisdictional restrictions, giving businesses the ability to control where their data is stored to meet their local and global needs. https://www.cloudflare.com/press-releases/2021/cloudflare-announces-r2-storage/ https://www.cloudflare.com/press-releases/2021/cloudflare-an...

tlarkworthy 2y ago

I've benchmarked R2 and S3 and S3 is well ahead in terms of latency especially on ListObject requests. I think R2 has come kind of concurrency limit as concurrent ListObject requests seem to to have increase failure rate when serving simultaneous requests I have a few of the S3-like wired up live over the internet you can try yourself in your browser. Backblaze is surprisingly performant which I did not expect (S3 is still king though) https://observablehq.com/@tomlarkworthy/mps3-vendor-examples https://observablehq.com/@tomlarkworthy/mps3-vendor-examples

kmos17 2y ago
thanks for sharing, that’s a good data point in comparing those services.

postatic 2y ago

I do mostly CRUD apps with Laravel and Vue. Nothing too complicated. Allows users to post stuff with images and files. I’ve moved ALL of my files from S3 to R2 in the past 2 years. It’s been slow as any migrations are but painless. But most importantly for an indie dev like me the cost became $0.

snihalani 2y ago

>Generally, R2's user experience is way better and simpler than S3. As always with AWS, you need 5 certifications and 3 months to securely deploy a bucket. +1

asteroidburger 2y ago

“Here’s a bunch of great things about CloudFlare R2 - and please buy my book about it” leaves a bad taste in my mouth. Also, has CF improved their stance around hosting hate groups? They have strongly resisted pressure to stop hosting/supporting hate sites like 8chan and Kiwifarms, and only stopped reluctantly.

gjsman-1000 2y ago
I don’t have to support 8chan or KiwiFarms to say that Cloudflare has absolutely no role in policing the internet. The job of policing the internet is for the police. If it’s illegal, let them investigate.
- asteroidburger 2y ago
  There is a difference between policing the internet and supplying resources and services to known bad actors. Their job isn’t to investigate and punish harassment and criminal behavior, but they certainly don’t have to condone it via their support.
  - gjsman-1000 2y ago
    > known bad actors If they are known bad actors, let the police do the job of policing the internet. Otherwise, all bad actors are ultimately arbitrarily defined. Who said they are known bad actors? What does that even mean? Why does that person determining bad actors get their authority? Were they duly elected? Or did one of hundreds of partisan NGOs claim this? Who elected the NGO? Does PETA get a say on bad actors? Be careful what you wish for. In some US States, I am sure the attorney general would send a letter saying to shut down the marijuana dispensary - they're known bad actors, after all. They might not win a lawsuit, but winning the support of private organizations would be just as good. > they certainly don’t have to condone it via their support Wow, what a great argument. Hacker News supports all arguments here by tolerating people speaking and not deleting everything they could possibly disagree with. Or maybe, providing a service to someone, should not be seen as condoning all possible uses of the service. Just because water can be used to waterboard someone, doesn't mean Walmart should be checking IDs for water purchasers. Just because YouTube has information on how to pick locks, does not mean YouTube should be restricted to adults over 21 on a licensed list of people entrusted with lock-picking knowledge.

karmakaze 2y ago

At one company we were uploading videos to S3 and finding a lot of errors or stalls in the process. That led to evaluating GCP and Azure. I found that Azure had the most consistent (least variance) in upload durations and better pricing. We ended up using GCP for other reasons like resumable uploads (IIRC). AWS now supports appending to S3 objects which might have worked to avoid upload stalls. CloudFront for us at the time was overpriced.

kansi 2y ago

I have tried to find a CDN provider which would offer access control similar to Cloudfront's signed cookies but failed to find something that would match it. This is a major drawback with these providers offering S3 style bucket storage because most of time you would want to serve the content from a CDN and offloading access control to CDN via cookies makes life so much easier. You only need to set the cookies for the user's session once and they are automatically sent (by the web browser) to the CDN with no additional work needed

saurik 2y ago
This is supported by Google Cloud using literally the same wording: https://cloud.google.com/cdn/docs/using-signed-cookies https://cloud.google.com/cdn/docs/using-signed-cookies As far as I can tell, this feature is also supported by Akamai here: https://techdocs.akamai.com/property-mgr/docs/cookie-authz https://techdocs.akamai.com/property-mgr/docs/cookie-authz I am pretty sure you can implement this on CDNetworks using eval_func: https://docs.cdnetworks.com/en/cdn/docs/recipes/secure-delivery https://docs.cdnetworks.com/en/cdn/docs/recipes/secure-deliv... With AWS Cloudfront, I'd think you--worst case--pull out Lambda@Edge?
- donavanm 2y ago
  Cloudfront “signed cookie” auth should “just work”: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-cookies.html https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope... IIRC its essentially the same as the aws style signed urls and header bearer token auth. I _think_ lambda@edge is only relevant if you want to do the initial sig generation in the cdn instead of your api/app endpoint. Edit: actually GP mentioned Cloudfront already, so yes works as theyre asking for AFAICT

denysvitali 2y ago

No mention of Backblaze's B2? It's cheaper than these two at just 6$/TB

russelg 2y ago
It's also dogshit slow, and only available in 2 regions. At least you can choose the region, unlike R2...
- pdimitar 2y ago
  How slow we're talking about?

dxxvi 2y ago

I have 250GB on S3. To get them out and store in R2, AWS will charge me 9 cents * 250 ~ $24: ouch.

mythz 2y ago
Thanks to the EU AWS now offers "Free data transfer out to internet when moving out of AWS" [1] [1] https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-internet-when-moving-out-of-aws/ https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-i...

matteocontrini 2y ago

I don't know what people use object storage for, but R2 is missing lots of features and it's not a replacement for S3 in many cases. For example: no regions, no replication (and no AZs either), limited lifecycle management, no versioning, no MFA protection, no intelligent tiering, no customer encryption, no IAM, etc.

remram 2y ago
Apologies for the tangent, but I always wondered: Why does anyone use "customer encryption" on the server rather than customer side?

mherrmann 2y ago

Reading around online, it seems egress is free until Cloudflare asks for big bucks or threatens to terminate your service. It's probably fine for hobbyists or small web sites. But as a business, I would never trust something that is "free". There has to be a catch somewhere. A business contact of mine asked Cloudflare for a quote for setting up a CDN for 7B requests per month. They dragged him through five sales calls to eventually deliver an offer with request costs 30% above CloudFront's public pricing. He said the costs per GB were ok though. The cheapest reliable CDN I've found is bunny.net. Unlike OP, who is selling a book about Cloudflare, I have no conflict of interest by recommending Bunny, other than being a customer. I serve around 100 TB per month through them, at $0.005/GB. You can put it in front of a bucket to have cheap (globally distributed) egress with all the benefits of your favorite object storage provider. In my case, the buckets lie on Linode/Akamai. Bunny uses CDN77's infrastructure, which I have also heard good things about.

sieabahlpark 2y ago
[dead]

Havoc 2y ago

S3 is still hdd? Cached at least?