chrislusf

12 ms·

91.
by chrislusf 6y ago
Thanks! There are many problems with large number of small files. It's more efficient to batch them together.
92.
by chrislusf 6y ago
Agree! Yet I do try very hard to keep each layer separate from each other. So each layer can be "lightweight". :)
93.
by chrislusf 6y ago
In SeaweedFS, rebalancing data is via admin commands. The data nodes only act as just as storage devices.
94.
by chrislusf 6y ago
a) you are right. I was not in storage field. b) During writes, the strong consistency is required. The repair is by admin scripts, not by the data node themselves. This is to keep data nodes simple and scalable. You can also run the repair
95.
by chrislusf 6y ago
In SeaweedFS, replication is for hot data and erasure coding is for warm data.
96.
by chrislusf 6y ago
Thanks for educational comments! I need to update those old statements. It was started originally as an object store only. The blobs are read-modify-write, but a file can have many blobs. So the whole file does not need to be read-modify-wr
97.
SeaweedFS S3 performance test, with comparison to Minio(github.com)
2 points by chrislusf 6y ago|0 comments
98.
by chrislusf 6y ago
Hi, I am working on SeaweedFS. Just saw this. Here is my comparison with MooseFS. All distributed file systems are similar in many ways. For differences specifically to MooseFS, 1) MooseFS metadata is managed in memory. SeaweedFS metadata i
99.
SeaweedFS on HDD is 2 times better than MooseFS on SSD(github.com)
1 points by chrislusf 6y ago|0 comments
100.
by chrislusf 10y ago
Good observation. I will make the recovery much faster with gRPC calls, which can detect broken network much faster than http heartbeats.
101.
by chrislusf 10y ago
Actually there is a pull request to add redis support to list files. https://github.com/chrislusf/seaweedfs/pull/254 I just haven't have time to test it and make it work yet. If you want to, you can help
102.
by chrislusf 10y ago
The master nodes only have soft states. All actual states are stored on volume servers, together with the actual data. If one master node goes down, the other master nodes will elect a new leader and control the system.
103.
by chrislusf 10y ago
Good to know this! The metadata layer with NewSQL storage seems exactly what SeaweedFS is already doing.
104.
by chrislusf 10y ago
There are many ways to build a filer layer. Current implementation is just one way to do it. Adding POSIX or not is more of a design choice. The blob storage layer can support both ways.
105.
by chrislusf 10y ago
There is. I am not following your reasoning.
106.
by chrislusf 10y ago
Thanks for the write up! You are correct on most of the points. There are postgres backed filer added already, which can list files.
107.
by chrislusf 10y ago
If one volume replica disappears, the whole set of volume will become read-only. There are no magic to auto correct things, which I personally think is where problems start. If the client crashes, the writes would fail.
108.
by chrislusf 10y ago
Sorry for my wording, if it is misleading. I am only saying the SeaweedFS code does not try to do anything unnecessary, so you can extract most of the performance juice out of your hard drive or SSD.
109.
by chrislusf 10y ago
Not a simple re-write. My original plan is to break normal file system into composable pieces, and make the file system more scalable. The first piece is the key~blob storage, which consists of master and volume servers. The second piece is
110.
by chrislusf 10y ago
I am Chris the author. btw: This is not a new project. It has been around since 2011. Many people tested it and contributed on it. It may not be the tool for everyone. But I guess many companies will need a key~file store. There is also a &
111.
Join 2 CSV Files by Go and LuaJit(github.com)
1 points by chrislusf 10y ago|0 comments
112.
Show HN: Gleam – A Go-Based MapReduce System Supporting Luajit, Unix Pipe Tools(github.com)
6 points by chrislusf 10y ago|0 comments
113.
by chrislusf 11y ago
The "operations" package is more sort-of "official" client, used by SeaweedFS itself. The feature list is still growing. I prefer not to really have an official GO API before I can have time to polish the API design.
114.
by chrislusf 11y ago
SeaweedFS Distributed File System https://github.com/chrislusf/seaweedfs Seaweed-FS is a simple and highly scalable distributed file system. There are two objectives: to store billions of files! to serve the files fast
115.
Going Out of Business Sale. Now I know it is for real(mailp.in)
1 points by chrislusf 12y ago|0 comments
116.
Intercepted audio shows Ukraine rebels shot down the plane – Mailpin(mailp.in)
6 points by chrislusf 12y ago|0 comments
117.
[ANN] WeedFS v0.61 – A distributed scalable file system(groups.google.com)
2 points by chrislusf 12y ago|0 comments
118.
Apologies from New Relic – Email Mistake – Mailpin(mailp.in)
1 points by chrislusf 12y ago|1 comments
119.
Fwd: HipChat is now free for unlimited users. Awesome.(mailp.in)
2 points by chrislusf 12y ago|0 comments
120.
by chrislusf 12y ago
True... but it is a lot of trouble with a large number of EC2 instances. Anyway, have to deal with it.

More ›