chrislusf

8 ms·

1. 61.▲
   by chrislusf 5y ago

   I have the same syndrome. But I feel seaweedfs should be easy enough to be picked up. If you have any questions, just let me know.
2. 62.▲
   by chrislusf 5y ago

   It really depends on the use case. The project is still growing and there are different edge cases for each features, especially new ones. However, in general, I feel the project is structured layer-by-layer, and should be easy to fix the p
3. 63.▲
   by chrislusf 5y ago

   For this particular issue, it was fixed in a PR that checks whether a volume is "crowded" and creates new volumes before they fill up. Not really an issue any more.
4. 64.▲
   by chrislusf 5y ago

   Try SeaweedFS. It should be much more scalable than MinIO or CEPH. Large files are well supported.
5. 65.▲
   by chrislusf 5y ago

   Actually SeaweedFS already supports asynchronous backup to data sinks such as S3,GCP,Azure,etc. I did not heard of this "kitchen" sink before. :) But adding one more sink should be trivial.
6. 66.▲
   by chrislusf 5y ago

   Thanks! SeaweedFS has a linearly scalable architecture and performs much better. It is also very easy to run. Just run this: "docker run -p 8333:8333 chrislusf/seaweedfs server -s3"
7. 67.▲
   by chrislusf 5y ago

   Thanks! I am working on SeaweedFS. https://github.com/chrislusf/seaweedfs There are also SeaweedFS CSI Driver: https://github.com/seaweedfs/seaweedfs-csi-driver
8. 68.▲
   by chrislusf 5y ago

   (I work on SeaweedFS) How about using SeaweedFS? https://github.com/chrislusf/seaweedfs With your dedicated server, the latency is consistent, No API/network cost. Extra data can be tiered to S3. SeaweedFS as a Ke
9. 69.▲
   by chrislusf 5y ago

   I am working on SeaweedFS. It was originally designed to store images as Facebook Haystack paper, and should be ideal for your use case. See https://github.com/chrislusf/seaweedfs And it already supports S3 API, and ot
10. 70.▲
    by chrislusf 6y ago

    I am working on SeaweedFS, which supports S3 API for object store, and can also use Cassandra as the metadata db. Cassandra has been performing well for most SeaweedFS users. The article listed many known Cassandra characteristics and cited
11. 71.▲
    by chrislusf 6y ago

    I am receiving $337 per month as of now for https://github.com/chrislusf/seaweedfs Not something to be proud of if including the time spent to evolve the project.
12. 72.▲
    by chrislusf 6y ago

    I am working on SeaweedFS, which this simple key value store project is inspired by. This minikeyvalue is a nice simplification for George's own use case. SeaweedFS has filePath -> fileIds -> locateVolumeServers -> lookupOnVol
13. 73.▲
    Use the distributed file system as a Key-Large-Value store

    2 points by chrislusf 6y ago|0 comments
14. 74.▲
    by chrislusf 6y ago

    Do you have an example of this format: [timestamp] [filename:line_number] message k1=v1 k2=v2 ...
15. 75.▲
    by chrislusf 6y ago

    Thanks a lot! Switching to it!
16. 76.▲
    Any recommendation for structured logging library in Golang?

    2 points by chrislusf 6y ago|4 comments
17. 77.▲
    by chrislusf 6y ago

    Thanks for the insights! Are the tail latencies critical though? Messages are expected to be async and delivery latency usually are not hyper critical.
18. 78.▲
    by chrislusf 6y ago

    If messages are batched, would there be any performance advantages over Kafka/Pulsar/etc with thread-per-core architecture? The context switching cost would be amortized?
19. 79.▲
    by chrislusf 6y ago

    It depends on your project. I used Go since 2011 on multiple projects, two of them are http://github.com/chrislusf/seaweedfs and http://github.com/chrislusf/gleam . One is a distributed file system
20. 80.▲
    by chrislusf 6y ago

    see https://github.com/chrislusf/seaweedfs/wiki/Erasure-Coding-f...
21. 81.▲
    by chrislusf 6y ago

    Thanks! I am not good at Python at all and would not able to add Python specific support. However, SeaweedFS has gRPC APIs that you can use to do all file operations.
22. 82.▲
    by chrislusf 6y ago

    SeaweedFS does support erasure coding for warm data.
23. 83.▲
    by chrislusf 6y ago

    There are actually no limit for blob size. The clients decide the blob size. It is limited by available memory and concurrency. By default, the filer client uses 8MB. Each identifier is as <volume_id, file_key, cookie>. With volume_id
24. 84.▲
    by chrislusf 6y ago

    Of course you can go with faster filer stores. Default filer store would be enough for millions of files. It is the same speed as a common file listing. Actually I do not understand why MinIO is slow here.
25. 85.▲
    by chrislusf 6y ago

    Predictable latency? Hadoop Support? Too different to compare.
26. 86.▲
    by chrislusf 6y ago

    Oracle wants to lock Java, while Amazon wants to open up S3 API, so more people can use S3. On the other hand, SeaweedFS is has no API access fees and faster than S3 with your own hardware. Not sure what Amazon may do.
27. 87.▲
    by chrislusf 6y ago

    From a user who worked in Apple as a contractor, they have used SeaweedFS object storage to store build artifacts, grown to about 200 nodes. There are a few other companies, but not as famous as Apple. :)
28. 88.▲
    by chrislusf 6y ago

    yes. Just run "weed mount". Actually the same binary to start "weed server".
29. 89.▲
    by chrislusf 6y ago

    haha. Actually SeaweedFS started with a worse name. But some users warned me of how bad the name was in Japanese language. So I added "Sea" in front of the old name.
30. 90.▲
    by chrislusf 6y ago

    In SeaweedFS both metadata and data are portable. You can move metadata to any faster scalable data store. And/or move the data to any cheaper/faster/larger storage, in cloud or in your garage.