11 ms·
Dodging S3 Downtime with Nginx and HAProxy
- cagataygurturk 10y agoNeat. You should be saving bandwidth costs also.
- mattrobenolt 10y agoAs mentioned, we are saving a lot in S3 transfer cost. :)
- deleted 10y ago[deleted]
- nodesocket 10y agoHeads up a simple yet production ready NGINX location block to proxy to a public s3 bucket looks like: # matches /s3/* location ~* /s3/(.+)$ { set $s3_host 's3-us-west-2.amazonaws.com'; set $s3_bucket 'somebucketname' proxy_http_version 1.1; proxy_ssl_verify on; proxy_ssl_session_reuse on; proxy_set_header Connection ''; proxy_set_header Host $s3_host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Authorization ''; proxy_hide_header x-amz-id-2; proxy_hide_header x-amz-request-id; proxy_buffering on; proxy_intercept_errors on; resolver 8.8.4.4 8.8.8.8; resolver_timeout 10s; proxy_pass https://$s3_host/$s3_bucket/$1; } Adding NGINX caching on-top of this is pretty trivial. Also, heads up, in the directive proxy_cache_path, they should consider enabling "use_temp_path". This directive instructs NGINX to write them to the same directories where they will be cached. We recommend that you set this parameter to off to avoid unnecessary copying of data between file systems. use_temp_path was introduced in NGINX version 1.7.10 and NGINX Plus R6. use_temp_path=off Also, they should enable "proxy_cache_revalidate". This saves on bandwidth, because the server sends the full item only if it has been modified since the time recorded in the Last-Modified header. proxy_cache_revalidate on;
- mattrobenolt 10y agoIn our case, we don't need to ever revalidate. We store things forever since our file blobs are immutable.
- bpicolo 10y agoImmutable blobs are really the right choice with s3, as it's eventually-consistent (when using it as a blob store anyway. If you're hosting a static site or similar it's a bit tricky to immutableize and not necessarily worth the effort).
- mattrobenolt 10y agoWe even go a step further, and our blobs are 100% content addressable. :) So caching is super easy for us.
- bpicolo 10y agoYep, file shas are a great choice. UUIDs are typically fine too. One sort of weird case is if I have an image key (sha-based) and want to store thumbnail sizes: 'bae6ff187e4c491e5de9cfa3b039ce7da8255798' makes sense as a base key, but really I want bae6ff187e4c491e5de9cfa3b039ce7da8255798/400x400 for thumbnails rather than storing individual thumbnail shas, hah.
- sandGorgon 10y agoor.. use cloudfront. It will probably be much cheaper to use cloudfront than the instance scaling required as your traffic increases.
- eli 10y agoIs CloudFront expected to have better uptime than s3? One argument for self hosting the proxy is that I don't care if s3 is working when my server is down anyway.
- sfeng 10y agoAnother good option is to store your files in a second cloud storage service, and use CDN or DNS failover.
- mattrobenolt 10y agoThis would unfortunately be slower, since it's not within our datacenter and on our network, and significantly more expensive. Probably at least 100x more expensive for the bandwidth than S3 by itself.
- koolba 10y ago> This proxy service had been running for a week while we watched our bandwidth and S3 bill drop, but we had an unexpected exchange yesterday morning: [pic showing S3 offline] Talk about great timing!
- StreamBright 10y agoThis is actually a great idea. I like datadoge on the pic. What sort of bot is that? Does anybody know?
- mattrobenolt 10y agoThat's DataDog. :)
- movedx 10y agoHave you considered using S3 bucket replication and having your application's logic fail over to the replication target in the event of a regional failure? The former is a checkbox, the latter is 30 minutes of coding (in my experience.)
- mattrobenolt 10y agoS3 bucket replication is a bit flawed, and doesn't buy us anything on top of what we implemented. And would cost double for storage. Plus more complex application logic which I wanted to avoid. Tools like haproxy are pretty good at this. With S3 replication, you still have a primary/replica setup in which only one of them can accept writes, but you can accept reads from both. So we'd gain HA between multiple regions, but we wouldn't solve our original goals: speed. The round trip to S3 was too slow for us.
- movedx 10y agoVery cool. I did read the article but I missed that speed was important, so the custom solution was an excellent choice. Another idea might be to use Varnish for the caching layer, but I haven't compared Varnish to NginX in many years so the gap has probably been closed now? Good work. I've stuffed this one in the back pocket for future use.
- mattrobenolt 10y agoI have tons of experience with Varnish and a long history there. Varnish is really bad for this since it's memory only. I wanted to use 750GB of disk space, not the 32GB of RAM we had. 750GB of RAM is significantly more expensive. And in our case, performance between reading some bytes from disk vs memory isn't significant. A disk seek is still many many orders of magnitude faster than a round trip to Amazon. With that said, Varnish does offer the ability to use mmaped files, but the performance is really appalling out of the box, and just not worth it. Varnish is way better if you want strictly in memory cache. Another benefit of nginx is the cache won't be dumped if the process restarts, unlike Varnish.
- 10y ago
- openasocket 10y agoThis is a neat way to deal with it, and has numerous other benefits, but I thought I'd add another thing to try: deploying your data in multiple regions. You can set up a secondary bucket in a secondary region, and configure your primary bucket to replicate data to the secondary bucket automatically. And then set up your infrastructure to switch to the secondary bucket for read operations should there be a problem with the primary. With large amounts of data the costs could add up, but it has the advantage that you can still serve all read requests during a regional outage, not just the reads that are in the cache. Obviously this method has cost associated with it, so you should probably only do this if you need complete data availability.
- mattrobenolt 10y agoOr we can do what we did and save money instead and gain the guarantees. Plus achieve our original goal of performance by bringing the bytes inside our datacenter instead of going out through public internet.
- openasocket 10y agoYeah, what I mentioned does nothing to improve performance, just thought I'd mention it as something to think about. For your use case S3 replication doesn't really give you extra availability. In practice I would likely use your caching method, and use S3 replication in addition, if I had a large dataset that doesn't cache well. Or if I absolutely needed to maintain write availability, in which case I would use bi-directional S3 replication.
- mattrobenolt 10y agoDefinitely. If our dataset was absolutely massive and we couldn't hold a reasonable amount on disk, it'd make more sense. Fortunately, we are getting a 90+% hit ratio out of a very small amount of space relative to the size of our bucket. And yeah, we have 0 resiliance for write data here. Again, fortunately, we can afford this tradeoff since the amount of uploads is significantly lower and much less critical for us.
- 10y ago
- c17r 10y agoI've built similar proxy/cache setups with just nginx. Is HAProxy there to get more insight into things, like the slack notification? Or does it serve another purpose?
- mattrobenolt 10y agoIt directs traffic conditionally based on if our cache server is up or not, falling back to S3 directly. In theory, we could use nginx for this, but nginx isn't as good as a generic load balancer since it doesn't have as rich of insight into the status of upstreams. Also yes, the nice benefit of being able to do alerting based on uptimes and whatnot. We already have haproxy running here, so it was just adding another frontend to support this.
- c17r 10y agoThanks for the info. Is the next step locally cache uploads so that works while S3 is down?
- mattrobenolt 10y agoThat's another problem that'd require much more work than just a few hours of hacking around. :)
- newobj 10y agoAlternate title: "Replacing S3 downtime for vastly greater amounts of your own downtime" What is the name for this phenomenon where folks think they can out-available a thing that has multiple engineers singularly dedicated to nothing more than its availability /and/ operation? It it just hubris? Surely there must be a more clinical name.
- openasocket 10y agoIf the caching server fails, the infrastructure automatically switches to use S3 directly.
- mattrobenolt 10y agoYeah, one of our goals was to not add a new/weaker point of failure. We gracefully fall back to S3 directly if our cache server is down without a hiccup. So there is no operational overhead of this additional cog. If the server has a failure, we'd go back to slightly degraded performance by talking across the country until we brought it back online.
- wimagguc 10y agoMaybe I missed this in the docs, but why isn't HAProxy considered a new point of failure?
- mattrobenolt 10y agoIt's running on localhost to each server. So the failure event here is that somehow haproxy process would explode with the rest of the server being fine. It's much much more likely that a whole machine will die instead or a network issue between machines, etc.
- wimagguc 10y agoSure, that's well understood. Being a low-risk point of failure, isn't it still a new one? It does come with setup and maintenance costs, test scenarios etc., so it's only fair to recognise this as a risk.
- mnutt 10y agoOne thing that complicates easy replication is the rules around CNAMEing your S3 bucket to a subdomain you control. You have to name the bucket subdomain.your-domain.com.s3.amazonaws.com, then CNAME subdomain to s3.amazonaws.com. Your replicated bucket needs a unique name, so when you want to fail over it's unfortunately not just a matter of changing a DNS entry to point to the alternate bucket. Of course, if you have complete control of the client you can just change the hostname. But if you have references scattered throughout html files, you'll likely want a reverse proxy in front of S3.
- matt_wulfeck 10y agoIf you're making range requests against S3 objects, look no further than Varnish as a reverse S3 proxy. More details on accomplishing this here: https://moz.com/devblog/how-to-cache-http-range-requests/ https://moz.com/devblog/how-to-cache-http-range-requests/
- krishnasrinivas 10y agoIf the client sends incorrect Authorization you will still serve from the cache. This is insecure.
- mattrobenolt 10y agoThis is a trusted, internal, private network. The only one who could do this is the application itself, or something rogue on our network. If something were running rogue on our network, there'd be worse things it could get access to.
- krishnasrinivas 10y agoI see, if this is private network then this is a nice simple solution for caching. We plan to implement S3-caching in minio [https://minio.io https://minio.io] (i.e it will authenticate the requests and also do caching) in case you'd be interested for public facing caching proxies.
- mattrobenolt 10y agoYep, it's definitely possible to go this route as well. We just didn't have to.
- AlexCoventry 10y agoHow does authorization and access control interact with the proxy? Does it first check authorization with S3 and cache the result, use a parallel ACL, or just allow access to anything by anyone?
- mattrobenolt 10y agoGood question. Authorization is passed along upstream to S3, but we don't re-check authorization when serving a cache hit. In our case, this is a fine tradeoff since our network is private and trusted.
- mattrobenolt 10y agoTo be more clear, there are other things we could do here if we didn't inherently trust our network and the things running there.
- AlexCoventry 10y agoThanks, I understand.
- noway421 10y agoEssentially a fluke and not a general case people should follow.
- mattrobenolt 10y agoDisagree. Considering the original goal had nothing to do with expecting S3 to go down, it just happened to be super useful during this incident. We get more more benefits even without S3 breaking.
- cakoose 10y agoOP is a little abrupt, but it isn't downplaying the system's usefulness to you guys; just warning other readers that this isn't a general solution for, as the post is titled, "dodging S3 downtime". It's a "fluke" precisely because it "just happened" to help with something it wasn't designed for. It's not uncommon for applications to have MRU access patterns and be able to keep partially functioning during partial data availability. For these applications, a cache will lower costs and mitigate S3 outages. It would have been nice for the article to give the criteria up front.
- mattrobenolt 10y agoIs paragraph 2 of the post not upfront enough? I clearly stated our goals of the project, and the fact that this also saves us through the incident was a great side effect. But I never set out to mitigate S3 failures like this.
- cakoose 10y ago> Is paragraph 2 of the post not upfront enough? I clearly stated our goals of the project That paragraph says your connection to S3 is slow. Your solution doesn't fix general S3 slowness. However, it does make S3 slowness less of a problem for your application. That's why it would have been useful to describe how your application accesses S3, so others can quickly determine if their applications would also benefit from a similar solution. For example: "90% of our S3 reads are of blobs that were read in the last 30 days." Many applications might upload data to S3, then quickly download it for processing. For those applications, this solution won't work. > But I never set out to mitigate S3 failures like this. We know. It's just that some people might read the title and think this is a more general solution than it is. OP was just warning people against that.
- jaymichael 10y agoFor object sync between on-premise and S3, AWS offers Storage Gateway. It supports IAM, encryption, bandwidth optimization, and local cache for hot objects with availability built in. https://aws.amazon.com/storagegateway/details/ https://aws.amazon.com/storagegateway/details/
- mattrobenolt 10y agoI'm afraid to know how much that costs.
- jaymichael 10y agoPricing is here: https://aws.amazon.com/storagegateway/pricing https://aws.amazon.com/storagegateway/pricing As with most services, you pay as you go and only for what you use. The prices are aligned with typical AWS storage, request volume, and data transfer costs.
- asher_ 10y agoThere have been a lot of articles and suggestions for mitigating S3's single-region architecture (design flaw?) since the outage. One solution that I haven't seen much of is to just use a service that gives you multi-region without any extra work, such as https://cloud.google.com/storage/docs/storage-classes#multi-regional https://cloud.google.com/storage/docs/storage-classes#multi-...
- mattrobenolt 10y agoThis doesn't solve our performance issues that I originally set out to address.
- cavisne 10y agoThe problem with solutions like these is you see global outages, like googles global VM outage last year. Also from the public description this sounds like one big system, ie the second "region" may not be a public Google Cloud region. Just as much chance of an outage,
- jlgaddis 10y agoLots of criticism in this thread by folks who are missing the point. @mattrobenoit -- neat idea and the 70% savings in bandwidth is awesome. The side effect of helping mitigate the S3 issue for you was a sweet little bonus!
- mattrobenolt 10y agoThank you. <3
- mattrobenolt 10y agoAlso, should note, it's about a 98% savings in bandwidth, but cost savings explicitly was 70% since we have to factor in the cost of running this new server.
- mixedbit 10y agoCan you disclose which provider do you use to host the servers? Do you have a fixed bandwidth or pay per GiB of transfer?
- mattrobenolt 10y agoSoftLayer. All internal bandwidth is free. Public network is a fixed allocation of bytes per server, then overages.
- Hydraulix989 10y agoCloud services like S3 are making programmers too complacent these days. Back in the day, every production web site had something like this "HAProxy" in place. Now, S3 goes down, and everybody has their thumbs up their asses because that wasn't covered in the Rails bootcamp they went to.