uroni

3 ms·

1.
by uroni 11d ago
I implemented this (HS5). Etags match the AWS S3 ones as well.
2.
by uroni 11d ago
Ceph is pretty much the standard, I think. Very complex to manage, though, but that might just be necessary complexity for a distributed storage system.
3.
by uroni 11d ago
Not without snapshot currently. I guess I could collect all writes in the (optional) WAL file while a backup is running, then the files would be consistent.
4.
by uroni 11d ago
My https://github.com/uroni/hs5 is designed for this use case. One notable thing is that compared to MinIO (and others) it does not store the objects as individual files. I also have DuckDB directly integrated. The rea
5.
by uroni 3mo ago
That it keeps an infinite cache of malloc page allocations is annoying (the issue you referenced). I just removed that (after complaining on the mailing list about it). The performance advantage is probably negligible in many cases (since m
6.
by uroni 5mo ago
In my similar project (s3 compatible single-node storage) https://github.com/uroni/hs5 I do use proper fsync for data and metadata durability. But it can be turned of via switch. It is a pet peeve of mine that the defa
7.
by uroni 5mo ago
I build https://github.com/uroni/hs5 as replacement for single node use with a focus on high performance. I list other alternatives in the README there. Some short version: Ceph: Robust, widely used for multi-node depl
8.
by uroni 5mo ago
I'd worry about file create, write, then fsync performance with btrfs, but not about reliability or data-loss. But a quick grep across versitygw tells me they don't use Sync()/fsync, so not a problem... Any data loss occurrin
9.
by uroni 5mo ago
Yup, https://github.com/awslabs/git-remote-s3 (disclaimer: never used it)
10.
by uroni 6mo ago
For me it went into the multi-node direction, where I'd use Ceph anyway (or build on-top of an existing solid distributed database) if I needed it. Also think there is an abstraction mismatch with the object stores that store the objec
11.
by uroni 6mo ago
I made https://github.com/uroni/hs5 -- focus is on single node and high performance. So plenty of alternatives available.
12.
by uroni 7mo ago
There is a lot of software that directly implements the HTTP S3 API. That API is also documented by Amazon. E.g. the last implementation I saw was by DuckDB https://github.com/duckdb/duckdb-httpfs/blob/main&#x
13.
by uroni 7mo ago
AGPL is "a plague" by design (viral). It has the explicit goal that any improvements flow back to the community project and the virality is a necessary building block for this. It is an elegant solution to a tragedy of the commons
14.
by uroni 7mo ago
I never understood why one would use MinIO over Ceph for serious (multi-node) use. Sure, it might be easier to setup initially, but Ceph would be more likely to work. For the single node use-case, I'm working on https://gith
15.
Show HN: HS5 – Open Source fast single-node S3 compatible object storage(hs5.eu)
2 points by uroni 7mo ago|0 comments
16.
by uroni 8mo ago
I’ve used this technique in the past, and the problem is that the way some file systems perform the file‑offset‑to‑disk‑location mapping is not scalable. It might always be fine with 512 MB files, but I worked with large files and millions
17.
by uroni 10mo ago
It uses LMDB, so if the object mapping fits in memory that should be pretty optimal for reading, while using the build-in Linux page cache and not a separate one (important for testing use cases). For write/deletes it has a bit of writ
18.
by uroni 10mo ago
I'm not a contributor to Minio. This is its own separate thing. I do have a separate AGPL project (see github) where I have nearly all of the copyright and have looked into how one would be able to enforce this in the US at some point
19.
by uroni 10mo ago
I've been working on https://github.com/uroni/hs5 as a replacement with similar goals to early minio. The core is stable at this point, but the user/policy management and the web interface is still in the wor
20.
by uroni 1y ago
I'm working on something that might be suited for this use-case at https://github.com/uroni/hs5 (not ready for production yet). It would still need a resilience/cache layer like ZFS, though.