5 ms·
F3
- MetroWind 4mo agoLmao I read the readme and still don't know what it does.
- krzyk 4mo agoFile format for what? Text, graphics, compiled code?
- Arainach 4mo agoThis project README is not particularly useful: It doesn't explain what the project does (a file format for what? Name dropping other things I haven't heard of isn't useful) There are no examples. It links to a flatbuffer schema which is at least well commented, but is full of deep implementation details. The point is that within 2-3 minutes I'm not convinced why I care and still don't know enough about what this is to even think back to if if I encounter a scenario in the future where it would be useful. > designed with efficiency, interoperability, and extensibility in mind. It provides a data organization that rectifies the layout shortcomings of the last-generation formats like Parquet, This is all marketing speak that says nothing. > maintaining good interoperability and extensibility (a.k.a future-proof) via embedded Wasm decoders What does this even mean? Providing a decoder is no guarantee of futureproofness.
- adammarples 4mo agoTabular data, it wants to replace Parquet
- deleted 4mo ago[deleted]
- largbae 4mo agoThis could use a bit more "why". Shortcomings of Parquet are mentioned as overcome by this, which ones? Certainly not wide tool support... Why should one leave Parquet or ORC for this structure?
- altairprime 4mo agoThe ‘why’ is referenced in the bibliography at the end of the readme; this repo is not meant to be consumed standalone. Start with the paper instead: https://doi.org/10.1145/3749163 https://doi.org/10.1145/3749163
- skrtskrt 4mo agoYeah it seems like most of this can be handled by some more dev hours to Parquet
- dj_axl 4mo agoPaper mentions Parquet, ORC, Nimble, Lance, TSFile, Bullion, and BtrBlocks.
- dietr1ch 4mo agoI also had no idea what they were talking about, but there's good points about how hardware oblivious and somewhat global is Parquet around metadata. I found this post interesting, - https://medium.com/@reliabledataengineering/f3-the-future-proof-file-format-that-finally-gets-it-right-0e7f0ddd2e72 https://medium.com/@reliabledataengineering/f3-the-future-pr...
- thisisauserid 4mo agoGreat! I'll use it. In the "future." Nimble? Lance? Also in the future. Maybe. I'll use Parquet in the present.
- adammarples 4mo agoNo commits in 8 months?
- yung_lean 4mo agoYeah this was a research project, it doesn't look like this is getting any adoption
- owentbrown 4mo agoNice! The world can always use a better data format. I think you might get some traction if you post the advantages over parquet and other files directly on the readme, so that if someone goes to https://github.com/future-file-format/f3 https://github.com/future-file-format/f3 the see why they should try it. Mention the advantages and post metrics. Cherry pick the metrics! There's probably a good use case for this but, from the current readme, it's not clear who should use this and why.
- deleted 4mo ago[deleted]
- gavinray 4mo agoThis bit is quite genius, rather than depend on a language-specific SDK/lib for working with the formats you can fallback to exported WASM methods if none exist: > "Each self-describing F3 file includes both the data and meta-data, as well as WebAssembly (Wasm) binaries to decode the data. Embedding the decoders in each file requires minimal storage (kilobytes) and ensures compatibility on any platform in case native decoders are unavailable. "
- verdverm 4mo agoIs embedding executable code into a file a security risk? My assumption is a yes
- mirashii 4mo agoThat would be why it chose a VM that is explicitly designed for sandboxing rather than native executable code or similar, the risk can be minimized by reducing the surface area available to that executable code to almost nothing.
- jayd16 4mo agoYou still have the halting problem to solve to prevent denial of service.
- adwn 4mo agoSolving the halting problem is neither necessary nor sufficient to prevent DoS attacks. It isn't necessary, because settings timeouts or other resource restrictions works way better to prevent DoS. It isn't sufficient, because even if you can prove that a program will halt at some point, this alone doesn't tell you how long it will take. What good does it do to know that the program will run for 10 years before it halts? By that time, service will already have been denied. Even turning hash table lookups from O(1) to O(n) (still very much terminating!) can result in a DoS.
- 4mo ago
- deleted 4mo ago[deleted]
- ChrisArchitect 4mo agoA more descriptive title would be helpful OP: F3: Open-source data file format for the future Previous discussion: 2025 https://news.ycombinator.com/item?id=45437759 https://news.ycombinator.com/item?id=45437759
- meta-level 4mo agoDon't know why but I had to think of https://xkcd.com/2116/ https://xkcd.com/2116/
- s1mon 4mo agoSomehow I assumed that you were linking to this one: https://xkcd.com/927/ https://xkcd.com/927/
- MoonWalk 4mo agoIs what?
- drdexebtjl 4mo agoProbably not a good idea to name your project “future” anything, if you expect that future to become the present. Also, f3 is already “fight-flash-fraud”.
- GolDDranks 4mo agoI love the idea, and I developed something similar of myself in the past (https://github.com/golddranks/kobuta https://github.com/golddranks/kobuta), but... this reeks of slop. With Rust code, edition="2021" is a dead giveaway.
- corvad 4mo agohttps://xkcd.com/927/ https://xkcd.com/927/
- Groxx 4mo agoHm. I can kinda see it replacing self-extracting EXEs, but a lot of why you choose specific file formats is for specific features they offer - any self-describing system can fall into "there are too many competing features and nobody handles them all" exactly as easily as any other format. Like, can this file be efficiently mmap'd? Maybe if it emulates tar internally, but you don't know until you run it. Can it be seeked to specific bytes to only decompress part? It only supports a pre-release version of ISO-36898533 seeking, and your file library dropped support for it 6 years ago. If I rewrite 1MB in the middle, can it only change those pages on disk (and maybe an index), or do I have to rewrite the whole thing? Well the wasm blob supports 97 different APIs for it (there are 35 copies of one with different names), so it's larger than the data (but nobody paid attention to that), so you have 19 options that you recognize, but your CPU's native WASM accelerator only handles two or three so you've still got to specialize your code heavily. At least with "*.tar.gz" you have some idea of what's possible.
- deleted 4mo ago[deleted]
- coffeecoders 4mo agoIf I am archiving PBs of data for 10+ years, I don't want to rely on a WASM interpreter being available and performant in the future just to read a file. I want a dead-simple, heavily documented byte specification like Parquet. Additionally, putting the decoding logic inside an WASM binary introduces an active execution layer into what should be a cold storage.
- 0xbadcafebee 4mo agoYou don't want to run a custom 10-year-old data parsing function every time you read a single data record?
- bijowo1676 4mo agoWinRAR format does include RAR VM bytecode as part of the archive to achieve state of the art compression in media files. it was sandboxed and well accepted by everyone. the same sandboxing capability exists for WASM as well. it is actually better for long-term archival: you dont need to carry decompression program, since it will be a part of the archive file itself
- zerobees 4mo agoSome folks described it as genius. I guess it's my turn to play the role of an annoying HN skeptic: I find it somewhat silly. Data compression formats are secondary to what you're planning to do with the data once decoded. An audio file is completely different than an SVG image. An embedded VM that decompresses video to raw pixels doesn't magically let you play that video in a text editor, so there's no radically new kind of interoperability. Each new format still needs to be handled in a format-specific way. I guess one use case is that I come up with a video compression scheme that's better than H.265, but not all platforms support it, so I embed a decoder that would allow me to play it back on legacy hardware. But that also shows the weakness of the idea: it's unlikely that legacy hardware will perform well doing software-only decode for video formats from the future. If we rolled this idea out in the 1990s, it would not have allowed watching Netflix on an i386. In the same vein, I doubt this would have allowed me to open Word 2021 files in Word 97. There's no 1-to-1 mapping between the data structures. So if this kind of compat isn't slam-dunk, what's the goal? The downsides are clear. First, it's probably a maintenance nightmare: if your decoder has a bug that needs fixing, how do you patch all the files that already embed it? And then, there's size overhead and security risks. We're adding a considerable attack surface to every format parser. It's more opportunities for remote code execution, resource exhaustion attacks, and so on. Again, this is not always wrong, but what's the benefit?
- vouwfietsman 4mo agoI don't think you have encountered the problems that this class of formats solves. Try looking up columnar storage formats, the pros and cons are pretty well defined these days. It is not meant for video decoding, indeed.
- lowbloodsugar 4mo ago>via embedded Wasm decoders runs screaming
- amluto 4mo agoOne nice thing about some modern formats is that there are tools that read them at extraordinarily high effective speed. For example, DuckDB can do all manner of nifty optimizations while reading its own native format or Parquet. And I’m not sure that those optimizations can be effectively applied to a format that needs a WASM blob to be run to understand it. By the time you run a non-SIMD or even a SIMD-optimized pass over app the data, if that pass doesn’t understand your query, you may have already lost. I admit I only skimmed the beginning of the paper, and maybe the format is less general than it sounds.
- hahahacorn 4mo agoMy understanding is it’s a fallback mechanism
- ShinyLeftPad 4mo agoTo save a click it's a file format for columnar data specifically (like Parquet), which they very generically named Future-proof File Format. Most of this could fit in the title instead of just "F3"
- antisthenes 4mo agoThe description mentions shortcomings of the previous file types like parquet, but it isn't really evident to me what those shortcomings are, or if the use cases for parquet and F3 have really that much of an overlap to make this comparison valid in the first place.
- anygivnthursday 4mo agoMy concern is, if decode fails I need to debug WASM added by some other party maybe containing random bugs. Maybe a library of standard decoders maintained and tested by the project could help, but then not sure if it kills the advantage of the flexibility it provides.
- titzer 4mo agoBut Wasm has deterministic execution, so if decode fails for you, it should have failed for them. I.e. it's not a problem that your system has introduced; they should be able to reproduce the failure independent of any client.
- anygivnthursday 4mo agoYes, if it comes from some reliable partner I can report bugs to, or something built in-house. In such environment probably fine. And maybe thats the main audience and not some open data exchange format where your system may be brought down by someone's random decoder.
- ChadNauseam 4mo agoBut the WASM is integrated into the file. So what scenario would happen where the decode fails? They wrote a file, wrote a wasm decoder, and didn't test the decoder against the file they just wrote? I think it would be a natural step as part of the publish pipeline to run the decoder against the file and compare the output against the source to ensure correctness.
- anygivnthursday 4mo agoThats what a reliable party would do certainly, which was my point above. But there is no guarantee really. And there may be people testing their decoder on some test data then attaching it to whatever data they send without testing edge cases. Its like testing recovery from backups, people who care do it, many people dont.
- sph 4mo agoI don’t know what are people commenting on. I see a README with little to no information about what this is, what problems it solves, just links to its Flatbuffer description and a directory full of source code. What context am I missing?
- burkaman 4mo agoThere is a linked paper: https://dl.acm.org/doi/epdf/10.1145/3749163 https://dl.acm.org/doi/epdf/10.1145/3749163
- jauntywundrkind 4mo agoThe wasm decoder thing was also done in Anyblox. https://github.com/AnyBlox https://github.com/AnyBlox https://gienieczko.com/anyblox-paper https://gienieczko.com/anyblox-paper Has nimble/velox had any better luck lately? I forget what stories someone shared, but, it seemed to have such big intent, then real trouble actually getting released. I want to say someone was saying the lawyers ended up not letting a lot of the work get released. Nimble is the one competitor benchmarked against here that beats them, and is also extensible (to some degree?), so I'd love to know how things have gone for the past 6-12 months for nimble/velox. https://news.ycombinator.com/item?id=39995112 https://news.ycombinator.com/item?id=39995112 https://github.com/facebookincubator/nimble/ https://github.com/facebookincubator/nimble/ https://materializedview.io/p/nimble-and-lance-parquet-killers https://materializedview.io/p/nimble-and-lance-parquet-kille...
- vouwfietsman 4mo agoNot sure why this got so many upvotes, also the landing page is not great, its better to look at the paper (see link below). Seems to be a columnar storage format that addresses some shortcomings in parquet. Thing is, though, that of all these formats the real winning feature is compatibility, which is (obviously) very hard to improve on, as anything new immediately loses. Parquet is unfortunately very good just by virtue of being first, and so widely supported. The most widely used parquet version is the oldest version from 2013 (as per the paper itself), so parquet itself couldn't even supplant parquet. If you want to improve on it, you need to bring some serious results, which I don't think f3 does. Also, my main gripe with parquet (single table per file) is not even addressed, so, also the name is a bit hyped up. Also also, it seems to go out of its own way to include a compiled wasm binary for decoding, yet requires flatbuffers to parse that blob? Kind of defeats the purpose. Its main result seems to be improved random access which, although certainly welcome, is not the point of columnar storage, as columnar storage was invented to exchange random access for something else: fast analytics. F3 seems to sacrifice fast analytics for the wasm decoder. I don't get it. Maybe I'm being too cynical. Can someone help me out here? https://dl.acm.org/doi/epdf/10.1145/3749163 https://dl.acm.org/doi/epdf/10.1145/3749163
- aduffy 4mo ago> Also, my main gripe with parquet (single table per file) is not even addressed, so, also the name is a bit hyped up. This is really more of an expectation that has been put on file formats by the query engines. Spark/Datafusion/DuckDB wouldn't really know what to do with a multi-table file. > Parquet is unfortunately very good just by virtue of being first, and so widely supported IMO that is not how technology works. It is great that Parquet is so good at a lot of things, but that does not mean just because it came first that it deserves to be the only analytic file format forever. > Its main result seems to be improved random access which, although certainly welcome, is not the point of columnar storage, as columnar storage was invented to exchange random access for something else: fast analytics Fast analytics, as well as newer ML-shaped workloads, are inherently mix of batch scans and random access. Some of the authors of F3 previously authored another paper that goes into the details of the shortcomings of Parquet https://www.vldb.org/pvldb/vol17/p148-zeng.pdf https://www.vldb.org/pvldb/vol17/p148-zeng.pdf All of the newer formats that popped up recently (Vortex, Lance, F3 now) have been working on solving the problems outlined in that paper. Lance has some interesting ideas, Vortex focuses on extensibility and performance by replacing all of Parquet's black-box encoders with fully transparent encodings. This solves the tradeoff between bulk and element decoding, allowing you to have efficient full scans and really fast random access. E.g. Langchain recently rebuilt a system that used to be all Parquet files to use Vortex and saw a massive speedup, which they talk about more here: https://www.langchain.com/blog/introducing-smithdb https://www.langchain.com/blog/introducing-smithdb Disclaimer: I work on Vortex, so a lot of these questions about "what is the point of building a new format" are things that I have grappled with myself.
- mmaunder 4mo agoA Wasm decoder takes encoded bytes and returns an iterator of Arrow Buffers. In case you were wondering.
- Qerub 4mo agoThis reminds me of Alan Kay's OOPSLA 1997 presentation "The Computer Revolution Hasn’t Happened Yet" when he describes the Air Force / Burroughs 220 file format from 1961 where the file/tape contained both the data and the procedures to read/write/print them: https://youtu.be/oKg1hTOQXoY?t=1355 https://youtu.be/oKg1hTOQXoY?t=1355
- mmaunder 4mo agoNot quite vaporware, but few commits, PRs, history, actual examples etc. It's pretty thin.
- nine_k 4mo agoF3 seems to be a reasonable archival data format. I see many replies criticizing F3 as an operational data format, like Parquet. Of course it can't be made as fast in the general case, or as compatible to the existing infrastructure. OTOH F3 would be easy to decode into almost any of today's accepted formats, and likely to any of tomorrow's data formats. That's where being self-describing and self-unpacking would be important.
- stackskipton 4mo agoWhat's wrong with just archiving Parquet files? Worry about lack of support for file format 50 years in the future? If that was truly the concern, would it be best to archive it in more plain text format like CSV or JSON?
- chatmasta 4mo agoAs appealing as this is, it will never gain traction without some backwards compatibility with Parquet and wide adoption of query engines to implement that backwards compatible path.
- gruntled-worker 4mo agoAre we positively sure that WASM will prove to be more future-proof than 640K MS-DOS or WinXP, or SNES cartridge files for that matter? On 6/23/26 there are a lot of emulators that run these. Will WASM necessarily beat them on 6/23/2051? Might be a case of xkcd 927.
- kridsdale3 4mo agoI really enjoy the spirit that moved you to write this comment. I want my AI Agents to adopt this attitude.
- gruntled-worker 4mo agoI'll happily train them for you but I charge by the hour, not by the token. BTW, while we're on the topic. I don't do social media. I occasionally type up a text or post on a technical board. Maybe 98% of my textual interaction these days is with LLMs. I would not be surprised if my prose changes to resemble theirs over time. I suppose that's symbiosis for ya. It's possible that your AI-dar might get even more ineffective.
- dang 4mo agoOne past discussion: F3: Open-source data file format for the future [pdf] - https://news.ycombinator.com/item?id=45437759 https://news.ycombinator.com/item?id=45437759 - Oct 2025 (125 comments) plus this bit: An Open File Format for storing the information from a forge - https://news.ycombinator.com/item?id=44043253 https://news.ycombinator.com/item?id=44043253 - May 2025 (1 comment)
- alex7o 4mo agoSo you know what is a file format the we would be able to Reed 100 years from now. CSV, json even fits (that is 30 years old now). If you don't know the original way it was created you know what each field I supposed to mean (if done well). Otherwise you look at hex decoded data with no way if knowing how to decide it if you don't have tha spec on how and why this was encoded. Msgpack and cbor are cool but in 100 years there is no way to decide it.
- ignoreusernames 4mo ago[dead]
- jayd16 4mo agoSo a web page, basically?
- tbolt 4mo agoI think we know how this story ends https://en.wikipedia.org/wiki/OpenDoc https://en.wikipedia.org/wiki/OpenDoc
- JuniperMesos 4mo agoThis software project shares the same name as a series of late-1980s erotic manga: https://en.wikipedia.org/wiki/F3_(manga) https://en.wikipedia.org/wiki/F3_(manga)