24 ms·
Using a disk-based Redis clone to reduce AWS S3 bill
- porker 5y agoNot getting how this helps. How do they know if the data is stale in SSDB? If the data is immutable I get that caching it locally speeds things up, but then couldn't they merely mirror files and check if the file exists?
- welder 5y agoThe majority of code stats (the data being stored) is received for the current day. Older data still comes in, but in that case we just update the SSDB cache after writing the new data to S3. The reason we don't use SSDB as the main source of truth is because S3 provides replication and resilience.
- nikisweeting 5y agoAren’t both of the problems posed solvable with stock redis? Using an append only file for persistence and enabling virtual memory support so the dataset can grow beyond RAM size.
- jiofih 5y agoVirtual memory was deprecated some time ago, there is no support for larger than RAM data anymore.
- xtracto 5y agoWhat do you mean by deprecated? As far as I know, Swap partition/files are alive and kicking in Linux. There are even recommendations of the amount of swap you should setup depending on the available RAM.
- welder 5y agoThey're talking about the Redis Virtual Memory feature not Linux Swap: https://redis.io/topics/virtual-memory https://redis.io/topics/virtual-memory
- nikisweeting 5y agoAh TIL, that’s too bad.
- klohto 5y agoDeprecated in favor of Redis Flash (behind an enterprise subscription)
- tebruno99 5y agoAll problems are solvable by switching to AWS EC2 which doesn't cost money to transfer Data out of S3 to begin with.
- CameronBanga 5y agoWish they would have given an idea of how much this helped improve their AWS bill. Hard to judge the value of this implementation without knowing the $$$ saved.
- welder 5y agoOur S3 costs were between $400/mo. - $800/mo. After SSDB it's more stable around $200/mo, so a conservative number is $200/mo savings. A related comment about our costs: https://news.ycombinator.com/item?id=26740911 https://news.ycombinator.com/item?id=26740911
- justinclift 5y agoOut of curiosity, have you compared SSDB via MinIO (as another commenter mentions)? Asking because we use MinIO, and for us (a small online place with a single actually active MinIO server) it's been pretty good + super stable.
- welder 5y agoHadn't heard of MinIO. We looked into another S3 caching solution that someone linked in comments here, but it came down to we already used SSDB in production. Also, a personal preference of mine is to keep all logic in the same layer: Python. For ex: We use SQLAlchemy as the source of truth for our relational database schema, and we use Alembic to manage database schema changes in Python. We even shard our Postgres tables in Python [1]. It fits with my preference to keep this caching logic also in Python, by reading/writing directly to the cache and S3 from the Python app and workers. [1] https://github.com/sqlalchemy/sqlalchemy/issues/6055#issuecomment-801132820 https://github.com/sqlalchemy/sqlalchemy/issues/6055#issueco...
- staticassertion 5y agoCan Minio act as a caching proxy? Or do you do something manual like publishing to minio first, and then later publishing to S3 in batch or something? edit: Found this https://docs.min.io/docs/minio-gateway-for-s3.html https://docs.min.io/docs/minio-gateway-for-s3.html Awesome! So minio can actually act as a caching proxy for S3. Very cool.
- jiofih 5y agoSince they are using Redis as a simple key-value store, I wonder if they looked into something like LMDB which is disk-backed and has even better performance?
- tyingq 5y agoHadn't heard of SSDB before, looks promising, active, variety of users, broad client support. Off the top of my head, if I were trying to front AWS S3 from DigitalOcean, I'd have gone with MinIO and their AWS S3 gateway[1]. It appears to be purpose built for this exact kind of problem. [1] https://docs.min.io/docs/minio-gateway-for-s3.html https://docs.min.io/docs/minio-gateway-for-s3.html
- juancampa 5y agoIt seems to be popular only in China [0] 0: http://ssdb.io/docs/users.html http://ssdb.io/docs/users.html
- edoceo 5y agoso?
- aravindet 5y agoYou might face a language barrier when trying to find answers from the community: For example it looks like most of the discussion on the issues and pull requests are in Chinese. I do not believe Google Translate does well with technical terms. (Of course this is a concern only if you don't read Chinese.)
- brap 5y agoLast I checked, DigitalOcean has a "Spaces" feature for file storage, which is compatible with the AWS S3 API (and they even suggest using the AWS client library). I always found it odd that we can easily port apps, databases, etc., from one cloud to another, but for file storage/CDN it's always some proprietary solution like S3. AFAIK open source solutions never really took off.
- tyingq 5y agoThe article mentions they looked at Spaces, but it was too slow.
- d10zqwe 5y agohttps://github.com/MarshallWace/cachenator https://github.com/MarshallWace/cachenator
- welder 5y agoVery cool! Thanks for sharing, I hadn't seen this before. Edit: Actually I looked at this before using SSDB and it was overly complicated for my use case. We went with SSDB because we already used Redis and SSDB, were very familiar with them, and had already worked out any pitfalls with using them. Wish we had found this sooner, thanks!
- latch 5y agoWhy not an http proxy with disk caching, e.g. location / { proxy_http_version 1.1; proxy_set_header Connection ''; proxy_set_header ..... proxy_cache s3_zone; proxy_pass https://s3 } If you wanted to scale this to multiple proxies, without having each one be a duplicate, do a consistent hash of the URL and proxy it to the owner. We used to do this to cache the first chunk of videos. Encoded with `-movflags faststart` this typically ensured that the moov chunk was cached at an edge, which dramatically decreased the wait time of video playback (supporting arbitrary seeking of large/long video file)
- welder 5y agoBecause we invalidate some SSDB keys in various cases, for ex: like old code stats data incoming, renaming a project, etc. Http proxies with etag seem like a good public-facing cache option, but our caching is internal to our DigitalOcean servers. Here's more info on our infra, which might help with our decision: https://wakatime.com/blog/46-latency-of-digitalocean-spaces-vs-aws-s3 https://wakatime.com/blog/46-latency-of-digitalocean-spaces-...
- latch 5y agonginx has proxy_cache_purge (1), but I agree that nginx doesn't expose a particularly good API around its cache. Plus, the awkwardness of doing disk IO from OpenResty (unless this has been improved), is a drag. Maybe this is something envoy or the newer breads do better. (1) http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_purge http://nginx.org/en/docs/http/ngx_http_proxy_module.html#pro...
- otterley 5y agoConsider using opaque symbols for your object/cache identifiers instead of using meaningful literals. That way, you can simply update the identifier mapping (Project A cache prefix moves from ajsdf09yu8sbvoihjasdg -> klajnsdg9fasf8avby), and the old values will naturally be evicted from the cache as they become less frequently accessed. In my experience, having to perform cache invalidation is usually a sign of design immaturity. Senior engineers have been through this trial before :-)
- ericls 5y agoIf you are caching files to something that is saved to a disk based file system, why not just use the filesystem?
- welder 5y agoThat's kind of what we're doing, until we start using SSDB Clustering. Filesystem as cache would be good for a local same-server cache, but it doesn't scale well when used over the network.
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- dieters 5y agoScylla also exposes Redis API, disk-based as well: https://scylla.docs.scylladb.com/master/design-notes/redis.html https://scylla.docs.scylladb.com/master/design-notes/redis.h... . With a good NVMe disk the latency is still expected to be good enough for most workloads.
- lykr0n 5y agoWow. Scylla does Cassandra, DynamoDB, and Redis. Wonder what will be next.
- PeterCorless 5y agoThere was an analysis done on the Redis API in Scylla back in 2019 (https://siddharthc.medium.com/redis-on-nvme-with-scylladb-5e12afd38dbc https://siddharthc.medium.com/redis-on-nvme-with-scylladb-5e...). The implementation, while still experimental, has come a long way since. I would be very interested to hear additional feedback from Redis users.
- mpermar 5y agoThis article seems to take it on Redis as it unproperly supports this use case, but Redis does not seem to be a right choice for a cheap vertically scalable k-v store. I didn't know SSSD but really you could have chosen amongst many other k-v stores or even memcache with disk support. I guess the point is that someone chose Redis and soon realized that RAM was not going to be the cheapest store for 500Gb of data. But that is not Redis' fault.
- nyc_pizzadev 5y agoDefinitely a strong use case for a caching proxy like Varnish or Nginx. The post describes using Redis as an application cache which means your application needs custom logic. A cache lookup before going to S3, and then it needs to write back out to cache when done. Simple, but possibly a lot more work than necessary. A caching proxy does away with this, you make a single call to the proxy, which behaves identically to S3, and it does all the work for you. No application changes needed other than changing the S3 endpoint address. Also, much greater performance because you are potentially cutting out multiple costly round trips.
- welder 5y agoCache invalidation is much easier with SSDB than Nginx or Varnish. Http proxies are good for public facing uses but not a good use case here.
- haggy 5y agoHow do you figure? Http proxies support various common cache invalidation strategies such as hash-TTL and ETags
- SahAssar 5y agoETags are not cache invalidation. ETags are a way for a client to validate the cache in the client, while cache invalidation for a http proxy would involve telling the proxy that it should discard a cached file. --- Basically: cache validation: Check if a file matches a cached version cache invalidation: Preemptively tell a cache to discard a cached file --- For example: Client A requests file X. Proxy caches file X. File X changes in the upstream. Client B requests file X. Proxy does not know the file changed upstream, so it sends a stale version. --- In this case either the proxy needs to revalidate each request with upstream (which is expensive) or we need some way to tell the proxy that it should discard it's cache for file X since it has changed. Nginx plus supports this but the free version does not: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_purge https://nginx.org/en/docs/http/ngx_http_proxy_module.html#pr...
- deleted 5y ago[deleted]
- sunilkumarc 5y agoInteresting! On a different note, recently I was looking to learn AWS concepts through online courses. After so much of research I finally found this e-book on Gumroad which is written by Daniel Vassallo who has worked in AWS team for 10+ years. I found this e-book very helpful as a beginner. This book covers most of the topics that you need to learn to get started: If someone is interested, here is the link :) https://gumroad.com/a/238777459/MsVlG https://gumroad.com/a/238777459/MsVlG
- 3pt14159 5y ago> According to benchmarks, writes to SSDB are slightly slower but reads are actually faster than Redis! How is this possible? Surely reads from RAM are faster than disk?
- welder 5y agoThe magic of Google's LevelDB? Seriously though, I don't know and I'm not sure if it's an accurate benchmark. I just know our statsd metrics show it performing just as fast as official Redis on much larger data sets, so that benchmark is probably correct.
- throwaway823882 5y agoI don't know Redis' design, but all sorts of things can slow down a program's reading from memory, from CPU cache misses to mutexes, garbage collection, CPU utilization, or something else (I don't know Redis' design). Whereas the page cache and disk cache can be very efficient, without anything but more reads/writes to get in your way. Or maybe the program reading from memory is single-threaded, and the program reading from disk isn't.
- avinassh 5y ago> How is this possible? Surely reads from RAM are faster than disk? even I am so surprised. I will try to run the benchmarks myself and see. but... can anyone really explain how can a disk based solution be faster than RAM
- specialist 5y agore SSDB: is there an ELI5 for doing disk writes "correctly"? And maybe something like Jespen for testing database persistence? Ages ago, I just used fsync() and called it good. After reading Dan Luu's article, I'm pretty sure I'd never figure it out on my own. "Files are hard" https://danluu.com/file-consistency/ https://danluu.com/file-consistency/
- welder 5y agoBecause files are hard, a lot of projects use a file system library called LevelDB/RocksDB [1] They interact with the library instead of directly with files. CockroachDB and SSDB are examples of projects using LevelDB. [1] https://en.wikipedia.org/wiki/RocksDB https://en.wikipedia.org/wiki/RocksDB
- aclatuts 5y agoI think minio might have been better to use than SSDB? It is a proxy specific for object storage and can do local file caching. https://docs.min.io/docs/minio-gateway-for-s3.html https://docs.min.io/docs/minio-gateway-for-s3.html
- welder 5y agoYep, someone suggested MinIO in the comments https://news.ycombinator.com/item?id=26956330 https://news.ycombinator.com/item?id=26956330
- segmondy 5y agoMinio is not a caching layer, it's pretty much spinning up your own object storage. I have used it and SSDB, and SSDB is a better drop in replacement for redis. Minio is a drop in for S3 if you don't want to pay for S3 and not serving a large enough data.
- welder 5y agoOh, in that case I still prefer using S3 because it scales and has built-in redundancy and reliability.
- segmondy 5y agoYup, nothing will match the redundancy and reliability of S3. If you use Minio and have hard drive issue, you now need to restore. If you really want to cut down on S3 cost some mor then you might want to use Wasabi or Backblaze with Minio as the S3 layer in front. If you're profitable tho, the peace of mind of S3 is worth it than trying to save another $100.
- tebruno99 5y agoI want to hate AWS as much as the next guy, but this is nearly the dumbest thing I've seen trending lately. They are paying money for something that is FREE to save a dime. AWS S3 -> AWS EC2 bandwidth is FREE. To pay money to send the data to DO EC2 and then build a Redis cache to save a few dimes on EC2 that the D.O. Redis cluster now spends is... just wow.
- booi 5y agowell you gotta spend money to save money...
- welder 5y agoTotally get that. The journey here wasn't straightforward. I've tried dual-writing in production to many databases including Cassandra, RethinkDB, CockroachDB, TimescaleDB, and more. Haven't tried ClickHouse/VictoriaMetrics, but probably won't now because S3 scales beautifully. The main reason not using EC2 is compute and attached SSD IOPs costs. This balance of DO compute and AWS S3 is the best combination so far.
- dang 5y agoCan you please not post shallow dismissals and put other people's work down like this? Putdowns tend to get upvoted, and then they sit at the top of threads, letting off toxic fumes and making this a nasty place. We're trying to avoid that here. (I've downweighted this subthread now.) I'm not sure why name-calling and meanness attract upvotes the way they do—it seems to be an unfortunate bug in how upvoting systems interact with the brain—but HN members need to realize that posts like this exert a strong conditioning effect on the community. It's always possible to rephrase a comment like this as, for example, a curious question—you just have to remember that maybe you're not 100% aware of every consideration that went into someone's work. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- tebruno99 5y agoFair enough, I should work on changing My tone. Thanks for the feedback.
- AtlasBarfed 5y agoSo... Redis is a read-primary system, but this is faster than Redis using SSDB/LevelDB for reads? That's a bombshell that deserves the graphs to back it up, way hotter of a take than the article.
- welder 5y agoYep, it's true SSDB is a hidden gem that deserves more exposure. LevelDB/RocksDB has been around for a while with benchmarks. Here's a related reply about it from earlier https://news.ycombinator.com/item?id=26957339 https://news.ycombinator.com/item?id=26957339
- throwaway823882 5y agoDoes LevelDB/RocksDB still randomly corrupt your data?
- secondcoming 5y agoLast time I looked at twemproxy it looked like abandonware
- dhruvrrp 5y agoHow would SSDB compare to using etcd? They both are disk backed key-value stores, but SSDB doesn't seem to be under active maintenance, with most of the recent merges being about fixing build issues.
- welder 5y agoThe main difference is you can use existing code/libraries without changing anything if you're already using Redis. That was a big win for us, since we could just swap out the Redis instances that suffered from low hit rates (because the data didn't fit in RAM and was purged) for SSDB easily.
- dikei 5y agoetcd is not suitable for high volume, it's built for data consistency, not performance.
- jrochkind1 5y agoHm. If a disk-based redis clone is actually more performant than redis... what are the reasons to use redis instead of SSDB? Why is anyone using redis instead of SSDB? Should I stop?
- welder 5y agoSSDB doesn't support every single Redis command, it's only a subset. Most people don't even know about SSDB. There's power behind the Redis snowball. We've switched to SSDB, but with that we lost the Redis community.
- zymhan 5y ago> a disk-based redis clone is actually more performant than redis Who says it's faster? This is supposed to make things cheaper.
- plater 5y agoThe articles says so.
- crazypython 5y agoJust use a CDN like CloudFront or Cloudflare. S3 is for storage, not retrieval. Building your own poorly distributed not at edge psuedo-CDN won't help as using a real edge CDN. Edge CDNs are closer.
- reph2097 5y agoJust run a server.
- kaydub 5y agoYou should really just foot the bill for AWS. This is going to be technical debt for you later on that will need to be paid. As a software/systems engineer I'll walk away from anywhere doing stuff like this to save a few dollars. I've worked a lot of janky places and now I work somewhere that goes the extra mile and puts that little bit extra in to do things right the first time. I know the on-going costs of doing stuff like this as well as the fact that you will have to clean it up at some point. EDIT: Not necessarily YOU will have to clean it up at some point, but the next guy most likely will.
- welder 5y agoIt's just me nobody else is working on WakaTime, so it's ok. Also, S3 scales beautifully and has been the best database decision ever. Built-in reliability and redundancy. SSDB has also been wonderful. It's just as easy and powerful as Redis, without the RAM limitation. I've had zero issues with SSDB regarding maintenance and reliability, as long as you increase your ulimit [1] and run ssdb-cli compact periodically [2]. I've tried dual-writing in production to many databases including Cassandra, RethinkDB, CockroachDB, TimescaleDB, and more. So far, this setup IS the right solution for this problem. [1] https://wakatime.com/blog/47-maximize-your-concurrent-web-server-connections https://wakatime.com/blog/47-maximize-your-concurrent-web-se... [2] ssdb-cli compact runs garbage collection and needs to be run every day/week/month depending on your writes load, or you'll eventually run out of disk space. Check the blog post for my crontab automating ssdb-cli compact.
- philihp 5y agoI don't think there's anything wrong with this. The correct answer is not always "outsource to AWS"; dependence on AWS (and any external dependency, especially those you have to pay for) is also tech debt. Everything you do is tech debt, and since this is just you, building it yourself is not only respectable (here, have an upvote!) but could also perhaps be something you end up doing 2-3 years down the line, and you've just gotten ahead of yourself.
- heliodor 5y agoFor many startups, that code that you mentioned as technical debt will be deleted en masse as the business pivots and adapts. For startups, moving fast and getting 80% done in 20% of the time is the correct choice. It's frustrating to look at the result as a software developer, but the engineer is there to serve the business, which pays the salary. As a software engineer, I very much disliked working in engineering teams that were engineering for the sake of engineering. It quickly devolves into meaningless arguments and drama.
- selcuka 5y agoI am wondering as to why you opted to use DigitalOcean instead of LightSail. Any technical reasons?
- welder 5y agoLightSail didn't exist yet when we switched to DigitalOcean.
- mikesabbagh 5y agoyou are decreasing your bill at the cost of your reliability. You are moving from offering the reliability of S3 to a single node reliability of your reverse proxy. Maybe if u have the technological knowledge, start your own block storage using Rook/Ceph object storage on DO. This will reduce your bill even further, and if u know what u r doing, u can improve the reliability
- welder 5y agoNo, the SSDB cache can go offline and the app just reads from S3 directly. I appreciate everyone trying to suggest better ways to solve this, but know that I've gone through many solutions dual-writing in production, comparing latency and throughput on production loads, and this is the best so far.
- mikesabbagh 5y agoAh ok Then if it has the option to read from S3 directly, then it seems legit :)
- Black101 5y agoIf you worry about cost, maybe you should avoid AWS?
- welder 5y agoYep, WakaTime uses DigitalOcean for almost everything, except S3 is just too good and the price is worth it.
- chrislusf 5y ago(I work on SeaweedFS) How about using SeaweedFS? https://github.com/chrislusf/seaweedfs https://github.com/chrislusf/seaweedfs With your dedicated server, the latency is consistent, No API/network cost. Extra data can be tiered to S3. SeaweedFS as a Key-Large-Value store https://github.com/chrislusf/seaweedfs/wiki/Filer-as-a-Key-Large-Value-Store https://github.com/chrislusf/seaweedfs/wiki/Filer-as-a-Key-L... Cloud Tiering https://github.com/chrislusf/seaweedfs/wiki/Cloud-Tier https://github.com/chrislusf/seaweedfs/wiki/Cloud-Tier
- welder 5y agoLooks slightly better for our use case than MinIO with optimization for many small files, I'll look more into it. Thanks!