19 ms·
Have you considered using S3 bucket replication and having your application's logic fail over to the replication target in the event of a regional failure? The
by movedx 10y ago
Have you considered using S3 bucket replication and having your application's logic fail over to the replication target in the event of a regional failure? The former is a checkbox, the latter is 30 minutes of coding (in my experience.)
- mattrobenolt 10y agoS3 bucket replication is a bit flawed, and doesn't buy us anything on top of what we implemented. And would cost double for storage. Plus more complex application logic which I wanted to avoid. Tools like haproxy are pretty good at this. With S3 replication, you still have a primary/replica setup in which only one of them can accept writes, but you can accept reads from both. So we'd gain HA between multiple regions, but we wouldn't solve our original goals: speed. The round trip to S3 was too slow for us.
- movedx 10y agoVery cool. I did read the article but I missed that speed was important, so the custom solution was an excellent choice. Another idea might be to use Varnish for the caching layer, but I haven't compared Varnish to NginX in many years so the gap has probably been closed now? Good work. I've stuffed this one in the back pocket for future use.
- mattrobenolt 10y agoI have tons of experience with Varnish and a long history there. Varnish is really bad for this since it's memory only. I wanted to use 750GB of disk space, not the 32GB of RAM we had. 750GB of RAM is significantly more expensive. And in our case, performance between reading some bytes from disk vs memory isn't significant. A disk seek is still many many orders of magnitude faster than a round trip to Amazon. With that said, Varnish does offer the ability to use mmaped files, but the performance is really appalling out of the box, and just not worth it. Varnish is way better if you want strictly in memory cache. Another benefit of nginx is the cache won't be dumped if the process restarts, unlike Varnish.
- openasocket 10y agonit: S3 has bi-directional replication, so both of the buckets can accept writes.
- mattrobenolt 10y agoI'll have to look into this. I wasn't aware. Either way, I don't think that'd replace our current setup since the original intent wasn't to increase our availability. But good to know!
- user5994461 10y agoOr just use Google, their S3 has an option where it's replicate worldwide and available in all zones.
- anthony_franco 10y agoWas this setup useful during the S3 outage yesterday? Wanted to double check before I implement it.
- mattrobenolt 10y agoYes. That was, in fact, the entire point of the blog post.
- anthony_franco 10y agoRight, but I was referring to the comment regarding the S3 bucket replication.
- mattrobenolt 10y agoAh, I can't comment about that since we obviously don't use it, but in theory, yes.