17 ms·

We built another object storage

133 points by fractalbits 10mo ago

fractalbits 10mo ago
github page: https://github.com/fractalbits-labs/fractalbits-main https://github.com/fractalbits-labs/fractalbits-main
andai 10mo ago
HN's version of this title is unintentional comedy :)
_f9cu 10mo ago
Can you say, how is this diferent in terms of data structure over conventional one please?
dbacar 10mo ago
One can only hope this does not go to same direction like Minio once they gain momentum.
whinvik 10mo ago
Interesting. Have you seen any benefits of using io-uring. It seems io-uring is constatly talked about but no one seems to be really using it in anger.
- 6r17 10mo ago
  Io-uring has it's fair amount of CVEs ; I'm wondering if people are checking these out ; because the goal is not to just make something fast ; but fast & secure. It's a little bit of a grey area in my opinion for prod on public machines. Anyone has a counter view on this I'm genuinely curious maybe i'm over cautious ? ps : there are actually other faster and more secure options than io-uring but I won't spoil ;)
  - hansvm 10mo ago
    My understanding is that the iouring CVEs are about local privilege escalation, not being appropriately sandboxed, etc. If you're only running code you trust on machines with iouring enabled then you're fine (give or take "defense in depth"). Is that not accurate?
    - 6r17 10mo ago
      I really need to properly study the CVEs instead of making some surface judgement tbh - might have to take a look at it again
jamiesonbecker 10mo ago
These questions are meant to be constructively critical, but not hyper-critical: I'm genuinely interested and a big fan of open-source projects in this space: * In terms of a high-performance AI-focused S3 competitor, how does this compare to NVIDIA's AIstore? https://aistore.nvidia.com/ https://aistore.nvidia.com/ * What's the clustering story? Is it complex like ceph, requires K8s like AIstore for full functionality, or is it more flexible like Garage, Minio, etc? * You spend a lot of time talking about performance; do you have any benchmarks? * Obviously most of the page was written by ChatGPT: what percentage of the code was written by AI, and has it been reviewed by a human? * How does the object storage itself work? How is it architected? Do you DHT, for example? What tradeoffs are there (CAP, for example) vs the 1.4 gazillion alternatives? * Are there any front-end or admin tools (and screenshots)? * Can a cluster scale horizontally or only vertically (ie Minio) * Why not instead just fork a previous version of Minio and then put a high-speed metadata layer on top? * Is there any telemetry? * Although it doesn't matter as much for my use case as for others, what is the specific jurisdiction of origin? * Is there a CLA and does that CLA involve assigning rights like copyright (helps prevent the 'rug-pull' closing-source scenario)? * Is there a non-profit Foundation, goal for CNCF sponsorship or other trusted third-party to ensure that the software remains open source (although forks of prior versions mostly mitigates that concern)? Thanks!
- mrweasel 10mo ago
  > the page was written by ChatGPT I wonder in that's why it's all over the place. Meta engine written in Zig, okay, do I need to care? Gateway in Rust... probably a smart choice, but why do I need to be able to pick between web frameworks? > Most object stores use LSM-trees (good for writes, variable read latency) or B+ trees (predictable reads, write amplification). We chose a radix tree because it naturally mirrors a filesystem hierarchy Okay, so are radix tree good for write, and reads, bad for both, somewhere in between? What is "physiological logging"?
  - randallsquared 10mo ago
    A hybrid of physical logging, which is logging page-by-page changes, and logical logging, which is recording the activity performed at an intent level. If you do both of these, it's apparently "physiological", which I imagine was first conceived of as "physio-logical". I could only find references to this in database systems course notes, which may indicate something.
ChocolateGod 10mo ago
so they added a metadata engine to S3? How does that compare to something like JuiceFS.
Aperocky 10mo ago
So they built an object storage to replace filesystem. And in "Why Not Just Use a Filesystem?", the answer they gave is "the line is already blurring" and "industry is converging". The line maybe blurring but as mentioned is still a clear cut use case for file system - or if higher access speed is warranted, just slap more RAM to the system and cache them. It will still cost less even at current cost of RAM.
- zozbot234 10mo ago
  AIUI, one obvious difference between object storage and file system (beyond things like support for directories and file name lookups, which OP talks about already) is that an object storage has only atomic file store/replace, whereas a file system has to support arbitrary edits on both file content and directories/metadata.
  - Aperocky 10mo ago
    Yes, so file system is a superset of object storage, making this even less of a competition. It's easy to implement object storage on FS vs. the other way around.
- pjdesno 10mo ago
  Because (a) you have to mount a file system, so the user running the app needs permission to do that, and (b) It’s really hard to have a filesystem shared across untrusting admin domains. With S3 you just do an http request and you’re done. A lot of folks get hung up on the theoretical equivalence of things, and forget that their favorite solution may be flat out unworkable in practice for reasons that have nothing to do with the theoretical features they’re talking about.
  - Aperocky 10mo ago
    This is absolutely correct and the reason why S3 exist and are popular. However have you looked at the case being discussed here? There's a place for object storage and the pattern as discussed might even warrant being a cache or cache cluster.
oersted 10mo ago
Small objects and low latency. Why not use any of the great KV stores out there? Or a traditional database even. People use object storage for the low cost, not because it is a convenient abstraction. I suspect some people use the faster expensive S3 simply as a stopgap. Because they started with object storage, the requirements changed, it is no longer the right tool for the job but it is a hassle to switch, and AWS is taking advantage of their situation. I suppose that offering an alternative to those people for a non-extortionate price is a decent business model, but I am not sure how big that market is or how long it will last. And it's not really a question of better tech, I'm sure AWS could make it a lot cheaper if they wanted to. But object storage at the price of a database with the performance of a database, is just a database, and I doubt that quickly reinventing that wheel yielded anything too competitive.
- pjdesno 10mo ago
  Because people don’t. I’ve spent a bunch of time analyzing IBM’s publicly released Cloud Object Storage traces. Median object size is about 16K, mean is a megabyte or two. A decent number of tenants have mean object sizes under 100K. People use object storage for a bunch of reasons. In general you’re better off supporting what your users are doing than demanding that they rewrite their applications because you think they’re doing it all wrong.
kburman 10mo ago
I feel like this product is optimizing for an anti-pattern. The blog argues that AI workloads are bottlenecked by latency because of 'millions of small files.' But if you are training on millions of loose 4KB objects directly from network storage, your data pipeline is the problem, not the storage layer. Data Formats: Standard practice is to use formats like WebDataset, Parquet, or TFRecord to chunk small files into large, sequential blobs. This negates the need for high-IOPS metadata operations and makes standard S3 throughput the only metric that matters (which is already plentiful). Caching: Most high-performance training jobs hydrate local NVMe scratch space on the GPU nodes. S3 is just the cold source of truth. We don't need sub-millisecond access to the source of truth, we need it at the edge (local disk/RAM), which is handled by the data loader pre-fetching. It seems like they are building a complex distributed system to solve a problem that is better solved by tar -cvf
- jeremyjh 10mo ago
  Yeah I was a bit lost from the introduction. High performance object stores are "too expensive?" We live an era where I can store everything forever and query it in human scale time-frames at costs that are far less than what we paid for much worse technologies a decade ago. But I was thinking of datalakes, not vector stores or whatever they are trying to solve for AI.
- Scubabear68 10mo ago
  Loved your sentence at the end about tar -cvf. Every generation seems to have to learn the lesson about batching small inputs together to keep throughput up.
- hodgesrm 10mo ago
  > It seems like they are building a complex distributed system to solve a problem that is better solved by tar -cvf That doesn't work on Parquet or anything compressed. In real-time analytics you want to load small files quickly into a central location where they can be both queried and compacted (different workloads) at the same time. This is hard to do in existing table formats like Iceberg. Granted not everyone shares this requirement but it's increasingly important for a wide range of use cases like log management.
tsuru 10mo ago
Every time I hear hierarchical storage, I can't help but think "It's all coming back to MUMPS, isn't it?"
hansvm 10mo ago
Nice. I was looking at building an object store myself. It's fun to see what features other people think are important. I'm curious about one aspect though. The price comparison says storage is "included," but that hides the fact that you only have 2TB on the suggested instance type, bringing the storage cost to $180/TB/mo if you pay each year up-front for savings, $540/TB/mo when you consider that the durability solution is vanilla replication. I know that's "double counting" or whatever, but the read/write workloads being suggested here are strange to me. If you only have 1875GB of data (achieved with 3 of those instances because of replication) and sustain 10k small-object (4KiB) QPS as per the other part of the cost comparison, you're describing a world where you read and/or write 50x your entire storage capacity every month. I know there can be hot vs cold objects or workloads where most data is transient, but even then that still feels like a lot higher access amplification than I would expect from most workloads (or have ever observed in any job I'm allowed to write about publicly). With that in mind, the storage costs themselves actually dominate, and you're at the mercy of AWS not providing any solution even as cheap as 6x the cost of a 2-year amortized SSD (and only S3 comes close -- it's worse when you rent actual "disks," doubly so when they're high-performance).
websiteapi 10mo ago
it's always interesting to me how our profession keeps reimplementing the same sort of thing over and over and over again. is it just inherent to the ease in which our experiments can be conducted?
- pyrolistical 10mo ago
  It’s different when it’s a product/service offering. In this case it’s more competition. Good for us the consumer.
firesteelrain 10mo ago
How does this compare to Dell’s ObjectScale? We eliminated MinIO on vSAN in lieu of ObjectScale for on prem.
orliesaurus 10mo ago
I'm more interested in the design philosophy behind these projects than which benchmarks top the charts... A lot of the high performance S3 alternatives trumpet crazy IOPS numbers, but the devil is in how they handle metadata and consistency. FractalBits says it offers strong consistency and atomic rename ([Why We Built Another Object Storage (And Why It's Different)](https://fractalbits.com/blog/why-we-built-another-object-storage/#:~:text=Why%20We%20Built%20Another%20Object,Strong%20consistency%E2%80%94no%20eventual%20consistency%20surprises https://fractalbits.com/blog/why-we-built-another-object-sto...)), which makes it different from most eventual consistency S3 clones. That implies a full‑path indexing metadata engine (something they mention in a LinkedIn post). That’s a really interesting direction because it potentially avoids some of the inode bottlenecks you see in Ceph and MinIO. BUT the real question for me is long‑term sustainability. Running your own object store is a commitment. Who's maintaining it when the original team moves on? It's great to see new entrants with ideas, ALSO it would be reassuring if there were clear governance and a non‑profit steward at some point. I don't mind if something uses AI to draft marketing copy... as long as the code is readable, reviewed, and licensed in a way that keeps it open. The space is crowded, and differentiation often comes down to the less flashy stuff: operational tooling, monitoring, easy deployment across zones, and how it fails. I'm curious to see where this one goes.
YouAreWRONGtoo 10mo ago
[dead]
up2isomorphism 10mo ago
This is not an area an open source will work. Particularly your main target is on cloud customers. If you are confident with your work, you should not open your source because that’s the single leverage you have.
pyrolistical 10mo ago
They claim AI workflows require: 1. Small Objects at Scale 2. Latency Sensitivity 3. The Need for Directories I’m skeptical on the last one. They talk about rename performance as being the issue. I think what they mean is if you use path as the object key, if you rename a directory in the middle of a path, you need rename every object key that uses it. But to me that is just a poor usage of an object store. You should never “rename” object keys. Consider how git does it. If you rename a directory and diff it, the underlying object store didn’t rename any key. In fact all the files in the object stores are unchanged. Only the tree file changed, which maps paths to file hashes. While renames would get faster that way, it would increase latency to do a path to object key look up. I would like to see how fundamental the requirement to have directories are to AI workflows. I suspect it’s human “but I’m used to it” requirement
- munchbunny 10mo ago
  > I would like to see how fundamental the requirement to have directories are to AI workflows. In my experience, it's not that directories are inherently important, it's that an organization mechanism is, in the service of a few key problems: 1. Privacy and data handling requirements 2. Versioning 3. Partitioning 4. Probably some others I'm forgetting Hierarchical storage is a useful all-purpose tool for these things.
  - pyrolistical 10mo ago
    How many of those problems are not solved by independent (s3 concept of) buckets?
everfrustrated 10mo ago
Clearly AI slop. The links are hallucinations and don't work just 404.

133 points by fractalbits 10mo ago

fractalbits 10mo ago

github page: https://github.com/fractalbits-labs/fractalbits-main https://github.com/fractalbits-labs/fractalbits-main

andai 10mo ago

HN's version of this title is unintentional comedy :)

_f9cu 10mo ago

Can you say, how is this diferent in terms of data structure over conventional one please?

dbacar 10mo ago

One can only hope this does not go to same direction like Minio once they gain momentum.

whinvik 10mo ago

Interesting. Have you seen any benefits of using io-uring. It seems io-uring is constatly talked about but no one seems to be really using it in anger.

6r17 10mo ago
Io-uring has it's fair amount of CVEs ; I'm wondering if people are checking these out ; because the goal is not to just make something fast ; but fast & secure. It's a little bit of a grey area in my opinion for prod on public machines. Anyone has a counter view on this I'm genuinely curious maybe i'm over cautious ? ps : there are actually other faster and more secure options than io-uring but I won't spoil ;)
- hansvm 10mo ago
  My understanding is that the iouring CVEs are about local privilege escalation, not being appropriately sandboxed, etc. If you're only running code you trust on machines with iouring enabled then you're fine (give or take "defense in depth"). Is that not accurate?
  - 6r17 10mo ago
    I really need to properly study the CVEs instead of making some surface judgement tbh - might have to take a look at it again

jamiesonbecker 10mo ago

These questions are meant to be constructively critical, but not hyper-critical: I'm genuinely interested and a big fan of open-source projects in this space: * In terms of a high-performance AI-focused S3 competitor, how does this compare to NVIDIA's AIstore? https://aistore.nvidia.com/ https://aistore.nvidia.com/ * What's the clustering story? Is it complex like ceph, requires K8s like AIstore for full functionality, or is it more flexible like Garage, Minio, etc? * You spend a lot of time talking about performance; do you have any benchmarks? * Obviously most of the page was written by ChatGPT: what percentage of the code was written by AI, and has it been reviewed by a human? * How does the object storage itself work? How is it architected? Do you DHT, for example? What tradeoffs are there (CAP, for example) vs the 1.4 gazillion alternatives? * Are there any front-end or admin tools (and screenshots)? * Can a cluster scale horizontally or only vertically (ie Minio) * Why not instead just fork a previous version of Minio and then put a high-speed metadata layer on top? * Is there any telemetry? * Although it doesn't matter as much for my use case as for others, what is the specific jurisdiction of origin? * Is there a CLA and does that CLA involve assigning rights like copyright (helps prevent the 'rug-pull' closing-source scenario)? * Is there a non-profit Foundation, goal for CNCF sponsorship or other trusted third-party to ensure that the software remains open source (although forks of prior versions mostly mitigates that concern)? Thanks!

mrweasel 10mo ago
> the page was written by ChatGPT I wonder in that's why it's all over the place. Meta engine written in Zig, okay, do I need to care? Gateway in Rust... probably a smart choice, but why do I need to be able to pick between web frameworks? > Most object stores use LSM-trees (good for writes, variable read latency) or B+ trees (predictable reads, write amplification). We chose a radix tree because it naturally mirrors a filesystem hierarchy Okay, so are radix tree good for write, and reads, bad for both, somewhere in between? What is "physiological logging"?
- randallsquared 10mo ago
  A hybrid of physical logging, which is logging page-by-page changes, and logical logging, which is recording the activity performed at an intent level. If you do both of these, it's apparently "physiological", which I imagine was first conceived of as "physio-logical". I could only find references to this in database systems course notes, which may indicate something.

ChocolateGod 10mo ago

so they added a metadata engine to S3? How does that compare to something like JuiceFS.

Aperocky 10mo ago

So they built an object storage to replace filesystem. And in "Why Not Just Use a Filesystem?", the answer they gave is "the line is already blurring" and "industry is converging". The line maybe blurring but as mentioned is still a clear cut use case for file system - or if higher access speed is warranted, just slap more RAM to the system and cache them. It will still cost less even at current cost of RAM.

zozbot234 10mo ago
AIUI, one obvious difference between object storage and file system (beyond things like support for directories and file name lookups, which OP talks about already) is that an object storage has only atomic file store/replace, whereas a file system has to support arbitrary edits on both file content and directories/metadata.
- Aperocky 10mo ago
  Yes, so file system is a superset of object storage, making this even less of a competition. It's easy to implement object storage on FS vs. the other way around.
pjdesno 10mo ago
Because (a) you have to mount a file system, so the user running the app needs permission to do that, and (b) It’s really hard to have a filesystem shared across untrusting admin domains. With S3 you just do an http request and you’re done. A lot of folks get hung up on the theoretical equivalence of things, and forget that their favorite solution may be flat out unworkable in practice for reasons that have nothing to do with the theoretical features they’re talking about.
- Aperocky 10mo ago
  This is absolutely correct and the reason why S3 exist and are popular. However have you looked at the case being discussed here? There's a place for object storage and the pattern as discussed might even warrant being a cache or cache cluster.

oersted 10mo ago

Small objects and low latency. Why not use any of the great KV stores out there? Or a traditional database even. People use object storage for the low cost, not because it is a convenient abstraction. I suspect some people use the faster expensive S3 simply as a stopgap. Because they started with object storage, the requirements changed, it is no longer the right tool for the job but it is a hassle to switch, and AWS is taking advantage of their situation. I suppose that offering an alternative to those people for a non-extortionate price is a decent business model, but I am not sure how big that market is or how long it will last. And it's not really a question of better tech, I'm sure AWS could make it a lot cheaper if they wanted to. But object storage at the price of a database with the performance of a database, is just a database, and I doubt that quickly reinventing that wheel yielded anything too competitive.

pjdesno 10mo ago
Because people don’t. I’ve spent a bunch of time analyzing IBM’s publicly released Cloud Object Storage traces. Median object size is about 16K, mean is a megabyte or two. A decent number of tenants have mean object sizes under 100K. People use object storage for a bunch of reasons. In general you’re better off supporting what your users are doing than demanding that they rewrite their applications because you think they’re doing it all wrong.

kburman 10mo ago

I feel like this product is optimizing for an anti-pattern. The blog argues that AI workloads are bottlenecked by latency because of 'millions of small files.' But if you are training on millions of loose 4KB objects directly from network storage, your data pipeline is the problem, not the storage layer. Data Formats: Standard practice is to use formats like WebDataset, Parquet, or TFRecord to chunk small files into large, sequential blobs. This negates the need for high-IOPS metadata operations and makes standard S3 throughput the only metric that matters (which is already plentiful). Caching: Most high-performance training jobs hydrate local NVMe scratch space on the GPU nodes. S3 is just the cold source of truth. We don't need sub-millisecond access to the source of truth, we need it at the edge (local disk/RAM), which is handled by the data loader pre-fetching. It seems like they are building a complex distributed system to solve a problem that is better solved by tar -cvf

jeremyjh 10mo ago
Yeah I was a bit lost from the introduction. High performance object stores are "too expensive?" We live an era where I can store everything forever and query it in human scale time-frames at costs that are far less than what we paid for much worse technologies a decade ago. But I was thinking of datalakes, not vector stores or whatever they are trying to solve for AI.
Scubabear68 10mo ago
Loved your sentence at the end about tar -cvf. Every generation seems to have to learn the lesson about batching small inputs together to keep throughput up.
hodgesrm 10mo ago
> It seems like they are building a complex distributed system to solve a problem that is better solved by tar -cvf That doesn't work on Parquet or anything compressed. In real-time analytics you want to load small files quickly into a central location where they can be both queried and compacted (different workloads) at the same time. This is hard to do in existing table formats like Iceberg. Granted not everyone shares this requirement but it's increasingly important for a wide range of use cases like log management.

tsuru 10mo ago

Every time I hear hierarchical storage, I can't help but think "It's all coming back to MUMPS, isn't it?"

hansvm 10mo ago

Nice. I was looking at building an object store myself. It's fun to see what features other people think are important. I'm curious about one aspect though. The price comparison says storage is "included," but that hides the fact that you only have 2TB on the suggested instance type, bringing the storage cost to $180/TB/mo if you pay each year up-front for savings, $540/TB/mo when you consider that the durability solution is vanilla replication. I know that's "double counting" or whatever, but the read/write workloads being suggested here are strange to me. If you only have 1875GB of data (achieved with 3 of those instances because of replication) and sustain 10k small-object (4KiB) QPS as per the other part of the cost comparison, you're describing a world where you read and/or write 50x your entire storage capacity every month. I know there can be hot vs cold objects or workloads where most data is transient, but even then that still feels like a lot higher access amplification than I would expect from most workloads (or have ever observed in any job I'm allowed to write about publicly). With that in mind, the storage costs themselves actually dominate, and you're at the mercy of AWS not providing any solution even as cheap as 6x the cost of a 2-year amortized SSD (and only S3 comes close -- it's worse when you rent actual "disks," doubly so when they're high-performance).

websiteapi 10mo ago

it's always interesting to me how our profession keeps reimplementing the same sort of thing over and over and over again. is it just inherent to the ease in which our experiments can be conducted?

pyrolistical 10mo ago
It’s different when it’s a product/service offering. In this case it’s more competition. Good for us the consumer.

firesteelrain 10mo ago

How does this compare to Dell’s ObjectScale? We eliminated MinIO on vSAN in lieu of ObjectScale for on prem.

orliesaurus 10mo ago

I'm more interested in the design philosophy behind these projects than which benchmarks top the charts... A lot of the high performance S3 alternatives trumpet crazy IOPS numbers, but the devil is in how they handle metadata and consistency. FractalBits says it offers strong consistency and atomic rename ([Why We Built Another Object Storage (And Why It's Different)](https://fractalbits.com/blog/why-we-built-another-object-storage/#:~:text=Why%20We%20Built%20Another%20Object,Strong%20consistency%E2%80%94no%20eventual%20consistency%20surprises https://fractalbits.com/blog/why-we-built-another-object-sto...)), which makes it different from most eventual consistency S3 clones. That implies a full‑path indexing metadata engine (something they mention in a LinkedIn post). That’s a really interesting direction because it potentially avoids some of the inode bottlenecks you see in Ceph and MinIO. BUT the real question for me is long‑term sustainability. Running your own object store is a commitment. Who's maintaining it when the original team moves on? It's great to see new entrants with ideas, ALSO it would be reassuring if there were clear governance and a non‑profit steward at some point. I don't mind if something uses AI to draft marketing copy... as long as the code is readable, reviewed, and licensed in a way that keeps it open. The space is crowded, and differentiation often comes down to the less flashy stuff: operational tooling, monitoring, easy deployment across zones, and how it fails. I'm curious to see where this one goes.

YouAreWRONGtoo 10mo ago

[dead]

up2isomorphism 10mo ago

This is not an area an open source will work. Particularly your main target is on cloud customers. If you are confident with your work, you should not open your source because that’s the single leverage you have.

pyrolistical 10mo ago

They claim AI workflows require: 1. Small Objects at Scale 2. Latency Sensitivity 3. The Need for Directories I’m skeptical on the last one. They talk about rename performance as being the issue. I think what they mean is if you use path as the object key, if you rename a directory in the middle of a path, you need rename every object key that uses it. But to me that is just a poor usage of an object store. You should never “rename” object keys. Consider how git does it. If you rename a directory and diff it, the underlying object store didn’t rename any key. In fact all the files in the object stores are unchanged. Only the tree file changed, which maps paths to file hashes. While renames would get faster that way, it would increase latency to do a path to object key look up. I would like to see how fundamental the requirement to have directories are to AI workflows. I suspect it’s human “but I’m used to it” requirement

munchbunny 10mo ago
> I would like to see how fundamental the requirement to have directories are to AI workflows. In my experience, it's not that directories are inherently important, it's that an organization mechanism is, in the service of a few key problems: 1. Privacy and data handling requirements 2. Versioning 3. Partitioning 4. Probably some others I'm forgetting Hierarchical storage is a useful all-purpose tool for these things.
- pyrolistical 10mo ago
  How many of those problems are not solved by independent (s3 concept of) buckets?

everfrustrated 10mo ago

Clearly AI slop. The links are hallucinations and don't work just 404.