daviesliu

62 ms·

31.
by daviesliu 4y ago
If you have good network, JuiceFS is the thing you are looking for. Multiple hosts can mount the same volume and it just works without conflicts. If the network is slow and unreliable, the experience will be bad, all fs operations will be b
32.
by daviesliu 4y ago
I live in Hangzhou, 150 miles away to Shanghai. There are a few positive COVID-19 cases in past weeks and we prepared to lockdown, but it did not come. Right now, there is a 48-hour-test policy, and we should have a negative COVID test with
33.
by daviesliu 5y ago
Yes, we can use `#amazon-s3` as the anchor, then the name could be long, for example, `#digitalocean-spaces-object-storage`), and it will not work in Chinese. Your suggestion is better than current one, we will use that, thanks!
34.
by daviesliu 5y ago
Docusaurus is great! We migrated from Sphinx to Docusaurus[1] recently, the navigation is much better than before. One thing that bothers us: we have not figure a way to name the anchor that both work in Github (`<span id='aws-s3&#x
35.
by daviesliu 5y ago
Founder of JuiceFS is here, we could like to here more feedback from community. JuiceFS have made lots of improvements since open sourced in Jan 2021[1]. For example, people were concerning on persistency of Redis, so we added PostgreSQL an
36.
JuiceFS v0.15 Was Released(github.com)
9 points by daviesliu 5y ago|2 comments
37.
by daviesliu 6y ago
Most of network file system provide open-after-close consistency, same to JuiceFS, it's mentioned here [1]: https://github.com/juicedata/juicefs#posix-compatibility
38.
by daviesliu 6y ago
As mentioned in other place, the perfect solution would be adding Raft to redis [1]. [1] https://github.com/RedisLabs/redisraft
39.
by daviesliu 6y ago
Redis is an in-memory database, so the persistency of data should be supported by using RDB and AOF together with replication and failover.
40.
by daviesliu 6y ago
These are scalable but very expensive in the clouds. They require a cluster of machines, the replicate the data across them, using either expensive EBS or local disk (a few larger instance to pick). Maintaining them well is another burden.
41.
by daviesliu 6y ago
JuiceFS access objects in S3 using an unique id to generate key, so the result of lifecycle will not change the way JuiceFS accessing it, except the StorageClass Glacier or Deep Arching, which is not accessible instantly, will hurt the user
42.
by daviesliu 6y ago
Since JuiceFS written in Go, GO SDK will be the first, then we can have other languages using CGO. We already have Java SDK internally, Python will be the next. But the way, A S3 gateway is on our roadmap, you can spin up an S3 gateway for
43.
by daviesliu 6y ago
Good to know, I'm always curious on Colossus.
44.
by daviesliu 6y ago
JuiceFS has a single node version working like this, it store in the metadata in SQLite and upload S3 in background, it's useful in some cases but not in general, you may lose recent metadata if the single node is broken. The difficult
45.
by daviesliu 6y ago
No, The Hadoop SDK is a connector to access JuiceFS, is built using JuiceFS client, will be open sourced later on, stay tuned.
46.
by daviesliu 6y ago
They are summaried here: https://github.com/juicedata/juicefs/blob/main/pkg/redis/red...
47.
by daviesliu 6y ago
Understood, the annoying part of NFS is that some operation can not be interrupted when the network is not healthy, and we have no way to kill it. We have put lots of effort to make the operations interruptible when either meta or object st
48.
by daviesliu 6y ago
Based on some information found on internet, Colossus is built on top of GFS, which use BigTable (or Spanner) as the meta store, and the BigTable still use GFS. The recent GFS may have multiple masters[1], but they are separate namespace, s
49.
by daviesliu 6y ago
Thanks, this is the goal we are targeting.
50.
by daviesliu 6y ago
Agreed, thanks for putting these number together. JuiceFS was initial designed for big data workload in Cloud, and we have tens of paying customer use it in this use case. For NAS use case, the latency would be slower when case missed, but
51.
by daviesliu 6y ago
Redisraft [1] is a good thing to expect. [1] https://github.com/RedisLabs/redisraft
52.
by daviesliu 6y ago
Thanks for verifying that. I have saw people asking SQL database support for s3ql, to make s3ql accessible from multiple machines, right now, JuiceFS could be a choice.
53.
by daviesliu 6y ago
Do you mean use JuiceFS as the place to store RDB or AOF? Yes, for backup. The fsync() is much slower than SSD(20ms vs 100us), so it may hurt performance.
54.
by daviesliu 6y ago
The whole idea was came from GFS: separate the metadata and data, load all the meta into memory, single meta server for simplicity, fixed-size chunk. The POSIX and FUSE stuff was learned from MooseFS, but changed to use read-only chunk, and
55.
by daviesliu 6y ago
Right now, the hosted Redis is the easiest one. If you run it on your own, please pay attention on the persistency options and HA solution, there should be plenty of article on these.
56.
by daviesliu 6y ago
> 1. Does JuiceFS support TLS encrypted Redis? Yes, please use `rediss://host:port/` > 2. Is there any access control and if there is who enforces it? You may specify port 0 to disable the non-TLS port completely. To en
57.
by daviesliu 6y ago
I saw Avere was recommended in GCP before, but never find the details, thanks for sharing that. It seems that Avere is close to ObjectiveFS, which also use S3 both for data and metadata. My guest is that Avere could require a cluster of nod
58.
by daviesliu 6y ago
Redis is used to persistent metadata, it can not be recovered from S3. The hosted Redis solution should already have failover solution for that, for example, AWS ElasticCache have multi-AZ failover[1]. [1] https://docs.aws.amazon
59.
by daviesliu 6y ago
There are covered now: https://github.com/juicedata/juicefs#posix-compatibility
60.
by daviesliu 6y ago
FUSE is not support by AWS Lambda, so we can't mount JuiceFS in Lambda. We can have a SDK to access JuiceFS from Lambda, similar to S3 SDK, when you need to use JuiceFS outside of Lambda. Same to Fargate, we can not mount JuiceFS in Fa

More ›