daviesliu

9 ms·

61.
by daviesliu 6y ago
Good question, we should put these in readme: 1. Unlinked file remain accessible, when it's unlink from same machine. 2. mmap is supported. 3. atomic rename is supported. 4. Is there atomic append in POSIX? 5. range lock is supported.
62.
by daviesliu 6y ago
In AWS, it's yes. The latency of first byte from S3 is about 20-30ms, close to what you can expect from HDD. Ceph is great, if you can master the complexity under the hood, MinIO + Redis + JuiceFS could be the easier answer for beginne
63.
by daviesliu 6y ago
We started to build JuiceFS since 2016, released it as a SaaS solution in 2017. After years of improvements, we released the core of JuiceFS recently, hopefully you will find it useful. I'm the founder of JuiceFS, would like to answer
64.
by daviesliu 6y ago
Good point. Most of file storage picked GPL, for example, Ceph, GlusterFS, MooseFS, so we followed them.
65.
by daviesliu 6y ago
Right now, the metadata is not replicated to S3, you can replicated it to another Redis, or backup the persisted RDB and AOF to S3.
66.
by daviesliu 6y ago
Redis can be persisted with RDB and AOF, can also be replicated to another machine. In the cloud, you don't need to worry about that, hosted Redis are ready to use. The is an ongoing effort [1] to improve the persistency and availabili
67.
by daviesliu 6y ago
Good, that's why we choose Redis for the metadata. When Redis is deployed in same DC or VPC, the latency could be about 0.2ms - 0.5ms. Later on, We will try the client cache of Redis, which could also reduce the latency for some metada
68.
by daviesliu 6y ago
Since POSIX is so complicated that we would not trust the test we wrote to prove the compatibility. So we use several third-party test suite, including fsx, pjdfstest, fsracer, flock. We also use third-party tools for benchmark, fio, mdtest
69.
by daviesliu 6y ago
The architect of JuiceFS is very close to GFS, which use a single node master for many years, even now. Since Redis is only responsible for metadata, a single node can serve hundreds of millions of files, and tens of thousands of IOPS, that
70.
by daviesliu 6y ago
The Kubernetes CSI driver will be released soon.
71.
by daviesliu 6y ago
The underlying S3 bucket still have intelligent tiering, you can also put life cycle rules on it.
72.
by daviesliu 6y ago
For small files (less than 4MB), the number of Get/Put/Delete is the same comparing using S3 directly. For larger file, each object in S3 is about 4MiB. Most of S3 Client will do the similar thing to GET/PUT small parts in pa
73.
by daviesliu 6y ago
Comparing to SeaweedFS, JuiceFS is more feature complete rather than basic read/write functionalities. The core of seaweedfs is to manage many small blobs, you can use SeaweedFS together with JuiceFS to have a full featured POSIX file
74.
by daviesliu 6y ago
I'd say Route 53 is more of an eventual consistent object store rather than a database :-)
75.
by daviesliu 6y ago
Comparing to seaweedfs, JuiceFS is more feature complete rather than basic read/write functionalities. The core of seaweedfs is to manage many small blobs, you can use SeaweedFS together with JuiceFS to have a full featured POSIX file
76.
by daviesliu 6y ago
The file format is described in https://github.com/juicedata/juicefs#architecture We will provide tools to dump the metadata as JSON, then you could recover your files using that.
77.
by daviesliu 6y ago
Good question, we have not try xfstests yet, will put it on TODO list.
78.
by daviesliu 6y ago
Founder of JuiceFS here, we have spent 4 years building it and open sourced it today.
79.
The Wise Woman's Helper(thewisewomanshelper.wordpress.com)
2 points by daviesliu 6y ago|0 comments
80.
by daviesliu 6y ago
Yes, JuiceFS splits files into 4MB blocks and put them into S3.
81.
by daviesliu 6y ago
JuiceFS supports GCS and can be used in GCP, we have customers use it for data scientists. JuiceFS is free if you don't have data more than 1TB. Discloser: Founder of JuiceFS here
82.
by daviesliu 6y ago
> interop is too valuable Good point, JuiceFS already provides this "transparent mode", which is called compatible mode.
83.
by daviesliu 6y ago
There are different when they are read/write from multiple clients in same time. HopsFS can use the meta service to synchronize with each other under low latency (about ms), but ObjectiveFS have to use S3 to do the synchronisation, whi
84.
by daviesliu 6y ago
JuiceFS can used in AWS EMR cluster, we have customers using JuiceFS to address the consistency and performance issues from S3. There is a blog post [1] talking about this use case. Unfortunately, we have not publish a English version, you
85.
by daviesliu 6y ago
We see similar results using JuiceFS [1], are glad to see that another player has moved on the same direction with us, since JuiceFS was released to public in 2017, and is available on most of public cloud vendor. The architect of JuiceFS i
86.
by daviesliu 6y ago
Juicesync support HDFS, can copy data using multiple machines, which is game killer if you have petabytes of data.
87.
by daviesliu 6y ago
Good point. Juicesync does support less service, but has better coverage for enterprise (Rclone does much better for individuals). Btw, we are on the way to open source JuiceFS.
88.
by daviesliu 6y ago
Juicesync is simper (fewer command line options) and faster (using multiple threads and machines) than rclone. disclaimer: worked on juicesync
89.
Juicesync: Parallelized Rsync for Cloud Storage(github.com)
28 points by daviesliu 6y ago|7 comments
90.
Ask HN: How do you backup your Mac?
1 points by daviesliu 7y ago|3 comments

More ›