4 ms·

Deepseek is the real "open<something>" that the world needed. Via these three projects, Deepseek has addressed not only efficient AI but also distributed comput

Deepseek is the real "open<something>" that the world needed. Via these three projects, Deepseek has addressed not only efficient AI but also distributed computing: 1. smallpond: https://github.com/deepseek-ai/smallpond https://github.com/deepseek-ai/smallpond 2. 3fs: https://github.com/deepseek-ai/3FS https://github.com/deepseek-ai/3FS 3. deepep: https://github.com/deepseek-ai/DeepEP https://github.com/deepseek-ai/DeepEP

dkdcwashere 2y ago
thank goodness, we’ve had nothing open to do efficient distributed computing with for years!
- OutOfHere 2y ago
  At least there hasn't been anything for distributed DuckDB before it afaik. For anyone with a substantial DuckDB project, they might now go distributed without having to rewrite it in something else.
swyx 2y ago
how many companies will actually adopt 3FS now that it's open source? not a hater, just know that theres a lot of hurdles to adoption even if something if open source - for example not being an industry standard. i dont know a ton about this space - what is the main alternative?
- huntaub 2y ago
  For example, in AWS, you can get a similar FSx for Lustre file system for just 11% more cost, which could be worth it to avoid the management costs of running your own storage cluster.
- skeeter2020 2y ago
  to me this seems to target a pretty small audience: very big data and specific problem domains, you need killer devops chops, expensive & specialized infrastructure and a desire to build out on bleeding edge architecture. I'd suspect most with these characteristics will stick with what they've got, "medium Big Data" companies should probably go with hsoted services and the rest of use stick with a single node DuckDB.
  - 0cf8612b2e1e 2y ago
    Bingo. Very few organizations have petabytes of data on which they are trying to efficiently process for machine learning. Such organizations already have personnel and technology in place offering some kind of solution. Maybe this is an improvement, but it is quite unlikely to be offering new capabilities to such teams.
    - datadrivenangel 2y ago
      And the organizations that get large enough to be sad with DuckDB performance will have options like MotherDuck for cloud hosting

dkdcwashere 2y ago

thank goodness, we’ve had nothing open to do efficient distributed computing with for years!

OutOfHere 2y ago
At least there hasn't been anything for distributed DuckDB before it afaik. For anyone with a substantial DuckDB project, they might now go distributed without having to rewrite it in something else.

swyx 2y ago

how many companies will actually adopt 3FS now that it's open source? not a hater, just know that theres a lot of hurdles to adoption even if something if open source - for example not being an industry standard. i dont know a ton about this space - what is the main alternative?

huntaub 2y ago
For example, in AWS, you can get a similar FSx for Lustre file system for just 11% more cost, which could be worth it to avoid the management costs of running your own storage cluster.
skeeter2020 2y ago
to me this seems to target a pretty small audience: very big data and specific problem domains, you need killer devops chops, expensive & specialized infrastructure and a desire to build out on bleeding edge architecture. I'd suspect most with these characteristics will stick with what they've got, "medium Big Data" companies should probably go with hsoted services and the rest of use stick with a single node DuckDB.
- 0cf8612b2e1e 2y ago
  Bingo. Very few organizations have petabytes of data on which they are trying to efficiently process for machine learning. Such organizations already have personnel and technology in place offering some kind of solution. Maybe this is an improvement, but it is quite unlikely to be offering new capabilities to such teams.
  - datadrivenangel 2y ago
    And the organizations that get large enough to be sad with DuckDB performance will have options like MotherDuck for cloud hosting