4 ms·
Do people really trust Redis for something like this? I feel like it's sort of pointless to pair Redis with S3 like this, and it'd be better to see benchmarks w
by staticassertion 9mo ago
Do people really trust Redis for something like this? I feel like it's sort of pointless to pair Redis with S3 like this, and it'd be better to see benchmarks with metadata stores that can provide actual guarantees for durability/availability.
Unfortunately, the benchmarks use Redis. Why would I care about distributed storage on a system like S3, which is all about consistency/durability/availability guarantees, just to put my metadata into Redis?
It would be nice to see benchmarks with another metadata store.
- tuhgdetzhh 9mo agoI think they should replace Redis with Valkey or even better use rocksdb.
- fpoling 9mo agoBut how would RocksDB work with S3? It needs support for append that generic S3 buckets do not provide and for checkpoints and backup it assumes the support for hardlinks that S3 does not have at all.
- alexhornby 9mo ago[dead]
- tuhgdetzhh 9mo agoSeaweedFS has a RocksDB metadata backend for instance.
- bastawhiz 9mo agoRedis is as reliable as the storage you persist it to. If you're running Redis right, it's very reliable. Not S3 reliable, though. But if you need S3 reliable, you would turn to something else. I expect that most folks looking at this are doing it because it means: 1. Effectively unbounded storage 2. It's fast 3. It's pretty darn cheap 4. You can scale it horizontally in a way that's challenging to scale other filesystems 5. All the components are pretty easy to set up. Many folks are probably already running S3 and Redis.
- danpalmer 9mo agoRedis isn't durable unless you drastically reduce the performance. Filesystems are pretty much by definition durable.
- bastawhiz 9mo agoEnabling the WAL doesn't make Redis slow. It's slower than the default, but it's still exceptionally fast. > Filesystems are pretty much by definition durable. Where do you think Redis persists its data to
- danpalmer 9mo agoRedis using the WAL for durable data can only be as fast as you can fsync on every write, just like every other database. The time to fsync is going to dominate the write path for Redis, and you'll sacrifice almost all of Redis's speed to achieve it. If you're running Redis in always fsync mode it I'd suggest there are better storage systems to use. > Where do you think Redis persists its data to This is sort of the point, Redis isn't for persistence. I mean it can, you can use the WAL, it's handy for not having to recompute when you relaunch Redis etc, but Redis is fundamentally an in-memory system and should be treated as such. It is designed for use-cases that necessitate in-memory performance, and that don't require durability.
- doctorpangloss 9mo ago> 4. You can scale it horizontally in a way that's challenging to scale other filesystems Easy to scale on RDS, along with everything else. But there’s no Kubernetes operator. Is there a better measure of “easy” or “challenging?” IMO no. Perhaps I am spoiled by CNPG.
- staticassertion 9mo ago> Redis is as reliable as the storage you persist it to. For a single node, if you tank performance by changing the configuration, sure. Otherwise, no, not really. I don't get why you'd want a file system that isn't durable, but to each their own.
- ycombinatrix 9mo agoIt says MySQL can be used instead of Redis for the metadata
- suavesu 9mo agoyes, support more 10 options, include redis, SQL-like DB, TiKV, FoundationDB, and more. see here -> https://juicefs.com/docs/community/databases_for_metadata https://juicefs.com/docs/community/databases_for_metadata
- onionjake 9mo agoWe developed Object Mount (formerly cunoFS) (https://www.storj.io/object-mount?hn=1 https://www.storj.io/object-mount?hn=1) specifically to not rely on any metadata storage other than S3 AND preserve 1:1 mapping of objects to files AND support for POSIX. We have a direct mode that uses LD_PRELOAD to keep everything in userspace so no FUSE overhead. This approach isn't right for every use case and juice might be a better fit for this sort of 'direct block store', but wanted to include it here for folks that might want something like Juice but without having to maintain a metadata store. (Disclosure: I work at Storj that develops Object Mount)
- ridruejo 9mo agoPretty cool
- lifty 9mo agoIs this open source? I am a happy Storj customer, would love to use it if it's open source.
- onionjake 9mo agoNot open source. What would be your use case?
- joshstrange 9mo agoI am currently looking for a way to take a legacy application that uses the filesystem as it's database and needs to support locking (flock) on FreeBSD and scale it horizontally (right now we only scale vertically and rebuilding from a corrupted FS and/or re-pulling our, backup, data from S3 takes too long if we lose a machine). We investigated NFS but the FreeBSD NFS performance was 1/10th or worse than on Linux and switching to Linux is not in the cards for us right now. Does Object Mount support file locks (flock specifically) and FreeBSD? I see some mention of FreeBSD but I can't find anything on locking. For context, we are working with a large number of small (<10KB if not <4KB) files normally.
- onionjake 9mo ago
- suavesu 9mo agoJuiceFS metadata engine comparison -> https://juicefs.com/docs/community/metadata_engines_benchmark https://juicefs.com/docs/community/metadata_engines_benchmar...
- staticassertion 9mo agoThanks, good to see these. Looks like these only show metadata operations though. But I guess that's probably enough to extrapolate that the system is going to be roughly 2-4x slower with other metadata stores.