5 ms·

bcachefs under workloads classic benchmarks skip

174 points by farlight 7d ago

farlight 7d ago
https://github.com/fenio/modern-fs-benchmark https://github.com/fenio/modern-fs-benchmark
Farmadupe 7d ago
@farlight assuming that you're the creator do you think you'd be able to rework the HTML/CSS? I'm sure you've got good data but speaking on behalf of my eyeballs, the results page is... hard to read!
- Farmadupe 7d ago
  > 2G of random garbage is written directly onto one member device (behind the filesystem's back, offset 1G — python injector; uutils dd mis-seeks on dm devices), caches dropped, then a full scrub: btrfs scrub -B, zpool scrub + wait, bcachefs scrub, md/lvm sync-action 'check' (which can only COUNT mismatches — no checksums to know which copy is right). I'm not sure that nuking 2G of the underlying block device is a recoverable error on any filesystem that I'm aware of? Can you confirm if any ofthe filesystems really came out of the other side in a usable state after scrubbing? ----- > Trivial-op p99, idle (ms) # A trivial operation — one 4k write + fsync every 200ms (like a shell appending history or an editor updating its swap file) — run alone for 10s. p99 of the fsync completion In fact, if it's OK for me to ask, are any of the metrics tht you used standard industry metrics? It looks like several of the tests are bypassing the kernel's page cache? -- which I worry may fall into the trap of "I modified the system to be unrepresentative of reality and then tested it". ---- > kernel 7.0.0-1012-azure Can you confirm if you tested on a bare metal machine? were you the only tenant?
  - vlovich123 7d ago
    1 device out of the replica set I’m assuming so all of them should recover.
  - hlieberman 7d ago
    The integrity check is only on the tests which are either RAID or the filesystem equivalent.
  - matja 7d ago
    > I'm not sure that nuking 2G of the underlying block device is a recoverable error on any filesystem that I'm aware of? ZFS and btrfs were designed from the start to handle this, by using checksums on every piece of (meta)data and redundancy to return the same data as was stored to the kernel, and rewrite the bad data. I've tested my own machines running ZFS by random writes out of band from the filesystem/kernel and it has always found and fixed them.
  - fenio 6d ago
    I answered in other thread how exactly integrity check looks like and why the name might be misleading. I'm also in the middle of updating descriptions to be more adequate. Thanks for pointing that out. Speaking about standard industry metrics. fio is an established tool, and throughput, IOPS, fsync latency, and percentiles are standard concepts. However, the exact job recipes and the composite “Overall Core” score are project-specific. The trivial-operation test is also custom: one 4 KiB write plus fsync every 200 ms. The idle window contains at most about 50 operations, so its p99 is effectively the slowest sample’s fio histogram bucket, not a statistically stable population percentile. It should be treated as a small-write durability-latency probe, not a universal application metric. But after all all tests are in the repository. If they need tweaks, changes I'm open to do so... I started from scratch and did whatever came to my mind. Some tests are added after my initial link here which went mostly unnoticed several weeks ago but I got some requests for more tests which I implemented. But to sum up. I want this test to be useful so feel free to open PRs with improvements. It's not like I've got some agenda. In fact I wrote here and there on the page that I'm counting on communities of various filesystems to provide improvements, changes etc to make their filesystem shining. This is personal project made when I realized that multiple-devices benchmarks were almost completely absent. Since I had not access to real hardware I decided to make at least initially everything based on GH runner with all the limitations that came with this approach. I tried to limit these limitations as far as I could. But feel free to submit bugreports, PRs, propositions for improvements.
- fenio 7d ago
  what exactly would you like to improve?
  - Farmadupe 7d ago
    I'd actually recommend some excellent books on the "philosophy" of data presentation: The first that comes to mind is "the visual display of quantitative information" by Edward Tufte seems to be freely available online, and the other one on my mind is "how charts lie" by Alberto Cairo (which doesn't seem to be freely accessible) But if it helps, just some "initial gut feel observations" from me: * It's definitely not possible to find issue with the the _sheer amount_ of results, but there's just far too much for a human to absorb, all presented at once * Overall text size is quite small, and difficult to read * The page doesn't make a strong statement of _what_ is under test: the first words are: "modern-fs-benchmark Multi-device CoW filesystems under workloads classic benchmarks skip" -- which defines the webpage in terms of what it is _not_, without stating what benchmarks are actually present. * The first line of teh page contains run statistics that probably eithre want to b at the bottom, or just don't need to be in the webpage at all: "latest run 2026-09-18 18:50:45 UTC, kernel 7.0.0-1012-azure, 593 runs recorded · 145 trend points shown" * A significant proportion of the free text is caveats. There's nothing wrong with being transparent about limitations, but they may be a sign that there might be alternative ways to present the data, or that the data may be flawed (depending on the caveat) * Theres several categories that I think have been invented for the purpose of collation, but I don't think are defined on the page. I think "Overall Core" and "Core I/O" aren't explained, which means by definition it's impossible for a reader to understand the score table. * And as we're all aware right now, current Claude models are currently struggling to write coherent English. There's several incoherent sentences on the page. It's a Claude issue.
    - koverstreet 7d ago
      Speaking as someone who consumes this, I appreciate how it's laid out. As a developer, we can often see at a glance where the bottleneck is if we have enough data laid out - IOW, data overload for you is me feeling like a kid in a candy store. Sometimes there are ways to make things easier without dumbing them down, but way too many people conflate the two; I get nervous when non engineers say "I've studied this, it should be easy".
skerit 7d ago
Oh, so bcachefs is doing pretty well.
- tarruda 7d ago
  Except for the fact that the developer has sabotaged the project into being removed from mainline?
  - AceJohnny2 7d ago
    that's not necessarily a sabotage.
    - eikenberry 7d ago
      Sabotage might not be the best word, but it hurt trust and adoption.
      - koverstreet 7d ago
        It's just been a lot less drama within the project since the split. I do have a lot more pull requests to merge than I did before. I don't know if you want to count "Kent isn't reviewing PRs fast enough" as drama :)
  - tombert 7d ago
    It's relatively easy to get it working as a kernel module at least. I got it set up on a NixOS box without too much trouble.
    - petre 7d ago
      Not something I'd want to do with my filesystem though. ZFS is different story it it isn't on the root fs because it might be packaged by the distribution.
      - tombert 7d ago
blop 7d ago
For peace of mind I'm still using zfs (since the last 15+ years) but I'm definitely not impressed by the performance...
- slyfox125 7d ago
  Different tools for different jobs; use ZFS for your data store and ext4 for your primary drive.
  - blop 7d ago
    yes indeed, zfs for my nas basically
- badgersnake 7d ago
  zfs-mirror-8k (whatever that is) looks alright.
  - deleted 6d ago
    [deleted]
  - magicalhippo 6d ago
    The script[1] shows it's a ZFS mirror (RAID-1) with 8k max record size, as opposed to the default 128k. I say max, as ZFS stores multiples of the physical block size up to the record size. So even with 128k record size, if your file fits in two physical blocks, it will write two physical blocks of data. The `ashift` parameter[2] controls the physical block size, typically 512 bytes or 4k for HDDs. Though higher can be useful on SSDs (but less tested, have seen some bug reports with >4k block sizes). Typically you'd only want to use such small records if you are tuning it for a specific workload, like if you have a database that writes 8k pages. Compression acts on records, so by limiting the record size, you limit the effectiveness of the compression: 1.2 and 1.8 blocks worth of data both gets written as 2 physical blocks. On the upside, record size is a dataset property, so you can have many different datasets with different record sizes on the same pool. [1]: https://github.com/fenio/modern-fs-benchmark/blob/599ec72fe335e2c043d83daa69c870fcf9fa8e72/scripts/fs/zfs.sh#L34 https://github.com/fenio/modern-fs-benchmark/blob/599ec72fe3... [2]: https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Workload%20Tuning.html#alignment-shift-ashift https://openzfs.github.io/openzfs-docs/Performance%20and%20T...
deleted 7d ago
[deleted]
loeg 7d ago
What is md-raid10 doing that is so much worse than lvm-raid10? In terms of "I/O" and "responsiveness." It's not really obvious to me from either the linked page or https://github.com/fenio/modern-fs-benchmark https://github.com/fenio/modern-fs-benchmark . In principle they should be similar?
- jmalicki 7d ago
  [flagged]
- deleted 7d ago
  [deleted]
Farmadupe 7d ago
> CI runs use loop devices on shared ephemeral VMs (one VM per filesystem): compare shapes and ratios, not absolute MB/s. Each job records a host-calibration anchor — see the table. I think if you're not using baremetal for such tests, it's likely that the results are simply not comparable at all? What if another tenant is also using the disk?
- walrus01 7d ago
  It's a fair point but it's also possible the person running the tests has a dedicated test hypervisor for this , so that different configurations of filesystems and VMs can be created and destroyed quickly in an automated manner. If it's something as simple as a KVM hypervisor that only runs 1 test VM at a time (with no other load from anything else other than the basic systemd daemons, ssh daemon etc running on the hypervisor), the results could be very close to bare metal. I can see it being very time consuming and annoying to do repeated manual bare metal OS installs and new partitioning/filesystem creation for such a large variety of tests. The author does also say that performance isn't really the main thing but rather, data integrity: https://github.com/fenio/modern-fs-benchmark https://github.com/fenio/modern-fs-benchmark
  - Farmadupe 7d ago
    > compare shapes and ratios, not absolute MB/s In this case, given that the author's own disclaimer (above) already disclaims the numeric readings, I'm not sure how it's possible to make any inference on "shapes and ratios" derived from the numeric readings.
  - toast0 7d ago
    > I can see it being very time consuming and annoying to do repeated manual bare metal OS installs. Well don't do that then. There's lots of other options. Probably the simplest is a single bare metal install on a simple filesystem on one device. run the filesystems under test on other storage dedicated to testing. You could also boot into a network install and use local storage exclusively for testing.
    - 7d ago
blop 7d ago
I think the reviews should also include the social aspect of these filesystems... There is and have been many promising and exciting FS to replace the old boring ones, but for storage you not only want to avoid technical issues but also maintainer(s) drama...
- koverstreet 7d ago
  Why do people keep bringing up drama? The community infighting has sucked, but that's a thing that matters primarily for maintainers. I think most users just want something that works.
  - Skunkleton 7d ago
    Related username? To answer the original question, most people who care about their filesystem at all care about its stability. Not just "does it work now" but also "will it work and improve over time". Infighting puts the future at risk.
    - koverstreet 7d ago
      It really does. But you might want to check out the bus factor on btrfs too; when a maintainer says "but we've saved Facebook billions and billions of dollars!", calls for the other filesystem maintainer to be ejected from the community, then quits to join Anthropic a month later - that's not a vote of confidence. I'd be very happy if people could just stop bringing up drama and us factors. We put it behind us a year ago, but it seems not everyone got the memo.
      - nolist_policy 6d ago
        Checking the bus factor: Btrfs regulars: - 1 from Meta - 1 from Oracle - 4 from SuSe - 2 from WDC bcachefs: - Kent Overstreet
        r0l1 6d ago
markhahn 7d ago
what does "integrity" fail mean in the first table? that the case didn't recover from the 2G corruption?
- fenio 7d ago
  The current “Integrity” label is broader than the test actually proves, and I’m changing it to “Corruption probe.” FAIL means that file changed or became unreadable. SURVIVED means only that the file remained readable and hash-identical. It does not prove the entire filesystem was healthy, that every overwritten byte was allocated, or that all 2 GiB were repaired. Thanks for pointing it.
sippingabonedry 7d ago
So two filesystems that are essentially shunned from the Linux kernel and permanent second-class citizens, and one that was removed from Red Hat and has a questionable history of reliability. Oh boy which do I choose? I'm saying ZFS on another OS.
- petre 7d ago
  xfs/lvm-raid10 or ext4/lvm-raid10, obviously.
  - sippingabonedry 7d ago
    If I'm forced to use Linux, sure. I could drop bricks on and cord pull those all day and they would not lose data. Which is a small ask for a filesystem IMO.
  - andriy_koval 6d ago
    would want to have compression..
    - simoncion 6d ago
      > would want to have compression.. I learned about VDO today. [0][1] I'd never heard of it before, but I'm using Gentoo Linux and both the dm-vdo kernel module and the 'vdo' software provided by [1] are distributed by Gentoo... so this isn't some weirdo Red Hat thing. It looks like you manage and used these just like any other thin-provisioned LVM volume [2], but -like I said- I've not used this before, and have only just skimmed the docs, so it's possible that I'm missing something important. [0] <https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/deduplicating_and_compressing_logical_volumes_on_rhel/introduction-to-vdo-on-lvm_deduplicating-and-compressing-logical-volumes-on-rhel https://docs.redhat.com/en/documentation/red_hat_enterprise_...> [1] <https://github.com/dm-vdo/vdo https://github.com/dm-vdo/vdo> [2] <https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/deduplicating_and_compressing_logical_volumes_on_rhel/creating-a-deduplicated-and-compressed-logical-volume_deduplicating-and-compressing-logical-volumes-on-rhel#creating-and-mounting-an-lvm-vdo-volume_creating-a-deduplicated-and-compressed-logical-volume https://docs.redhat.com/en/documentation/red_hat_enterprise_...>
magicalhippo 7d ago
> Every push/2-hourly cron builds each filesystem across 4 loop devices backed by sparse files, runs the suite, and publishes a results table in the job summary plus JSON artifacts. I get that real hardware costs (author mentions EUR 70 a month for a suitable server), but without at least a baseline snapshot comparison run between real hardware, both SSD and HDD, and the sparse file-backed loop devices, it's hard to take much away from this. Sadly the AI apocalypse isn't making stuff like this easy to do as a hobby.
fenio 7d ago
The author of the benchmark here. I went over some comments and I'll try to tackle them here. I'm pretty clear that GH runner based benchmark is far from perfect due to noisy neighbours etc. Thus every test first is running so called calibration... to reject completely unreliable VMs. I'm fully aware that this can't completely fix the issue. Can limit it but not fix. But as of now there are 593 runs recorded so average should still be quite meaningful. Having that said I'm desperately trying to get REAL hardware to run that benchmark. With some successes ;) Few months ago I got Hetzner machine from Kent Overstreet and I was able to finish 3 runs before machine died... Results: https://bartosz.fenski.pl/modern-fs-benchmark/real-hw/ https://bartosz.fenski.pl/modern-fs-benchmark/real-hw/ Currently I've got even more interesting machine with tons of disks and I'm running new set of benchmarks but it's really in its initial stage. https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/ https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/ 2nd run in progress... one run on REAL hardware takes much more time than on GH runner so it's slow. But this new hardware has also so many disks that the plan is to try also more complex, tiered cache topologies. I'm working on it. I'm happy to answer any other questions, sources of every piece of this benchmark are freely available and I'm not saying they are 100% correct. I'm open to improvements.
- koverstreet 7d ago
  I went back and forth with Hetzner a couple times, I think we just got a bad machine :) I've been saying it for months, but eventually I'm going to move the automated builds off the 48 core monster and we'll be able to use that for automated perf testing too. The machine we just got has spindles for EC perf testing, but the Hetzner monster has very high end enterprise ssdd. Also, just got done with the Rust for Linux conference, still not home but here's slides that still need reformatting: https://evilpiepirate.org/~kent/Kangrejos-2026-bcachefs.pdf https://evilpiepirate.org/~kent/Kangrejos-2026-bcachefs.pdf
- Joel_Mckay 7d ago
  Should include: 1. Dual Ext4 + external 32GB journal X4 pcie SSD (the prior winner of benchmark surveys) 2. Bare F2FS after a trim and SSD vendor software cache flush operation (it should be slower, but knowing how much slower on identical hardware could be interesting.) 3. DRBD across a 48U 100Gbps host rack (single X4 pcie data drive per host, OS on primary) 4. CephFS across a 48U 100Gbps host rack (single X4 pcie data drive per host, OS on primary) Best regards =3
irusensei 7d ago
BCacheFS is the best Linux filesystem now that storage costs a premium. You can mix devices of different sizes and types on bcachefs. You can have foreground and background devices to balance performance and also different compression settings for foreground and background transactions. You can set replicas=N to the individual file or directory on bcachefs. For example files you can just re-download or re-build. Likewise you can set a higher number of copies to important files.
- suncore 7d ago
  Sure, when it's mature...
  - mkeedlinger 7d ago
    Yeah, and I think Overstreet might be going hard on AI, so not sure how that might effect the project
    - koverstreet 7d ago
      If you have "refine until it's perfect and don't screw with things you don't understand" thoroughly ingrained, along with the dangers of overconfidence, you'll do fine with AI. Not everyone gets it though, that's for sure. And, if you want to know if it's mature, I'd trust the user reports over the one liners :)
- lukaslalinsky 6d ago
  I've been using ZFS on my Linux servers for years and have pretty good experience with it. I've not been following Linux development too much. Is bcachefs usable/stable/reliable enough to replace it?
  - khajdamowicz 6d ago
    It's not as battle-tested like ZFS with 15+ years of field usage, but it's good enough for me and other users. There's a NAS appliance called NASty and it shares publicly usage stats: https://nasty-telemetry.pages.dev/ https://nasty-telemetry.pages.dev/ Those numbers are NASty alone. There are more users on other systems.
azatom 7d ago
clicked, ctrl+f ssd, hdd, not found, closed
- azatom 7d ago
  In filesystem subject a proper critique is mentioning ssd vs hdd. I did not added value for karma increase, but karma lost? Who haven't I wished a happy birthday?
- fenio 6d ago
  The new machine - I mentioned in the other thread - has both HDDs and SSDs so tests with mixed topologies are planed.
bhaney 7d ago
Seeing great results from bcachefs just makes me more sad that Kent and the other kernel devs couldn't come to an understanding to keep bcachefs in-tree. I want to use it for my storage arrays so badly, but I'm still stuck with btrfs as the only available in-tree filesystem with modern features.
- koverstreet 7d ago
  NixOS. You can't go wrong.
  - Sha1rholder 6d ago
    What do u mean NixOS? It's an OS not a file system
    - koverstreet 6d ago
      NixOS thoroughly solves all the external module fragility. They also do distro level CI testing.
- khajdamowicz 6d ago
  You can use NASty as NAS appliance. It's based on NixOS, offers flexibility, atomic upgrades and all bells and whistles of bcachefs.
- locknitpicker 6d ago
  > Seeing great results from bcachefs just makes me more sad that Kent and the other kernel devs couldn't come to an understanding to keep bcachefs in-tree. I think you are sugarcoating the shit show that was bcachefs's history of involvement in the linux kernel. I mean, do I need to mention that the person was subjected to a code of conduct enforcement action due to his long history of abuse and unprofessional behavior? https://lwn.net/Articles/999197/ https://lwn.net/Articles/999197/
  - bhaney 6d ago
    No, you probably didn't need to mention that.
viraptor 7d ago
It could use some explanation for the exact setups. Like how many drives are there in each raid, what does the ec config actually look like, etc. As far as I can see it's not described.
- fenio 6d ago
  I extended info about setup. Should be visible in top part of benchmark.
nullc 7d ago
It'll be interesting when people start applying similar effort that is going into LLM search for security vulnerabilities to finding conventional reliability/performance bugs.
pzmarzly 7d ago
xfs is by far the slowest in "ftruncate empty file to 1G" but the fastest in "ftruncate 256M file to 512M" - does anyone know why?
- pzmarzly 7d ago
  Seems like a bug in benchmark code for that test, opened GH issue https://github.com/fenio/modern-fs-benchmark/issues/35 https://github.com/fenio/modern-fs-benchmark/issues/35
  - fenio 6d ago
    Should be fixed. Thanks.
deleted 6d ago
[deleted]
gdgghhhhh 6d ago
In case someone wonders why bcachefs is sadly no longer part of the mainline kernel. Start reading here: https://lwn.net/Articles/1027289/ https://lwn.net/Articles/1027289/
- lifty 6d ago
  Is there any chance bcachefs will be included again in the future?
  - gdgghhhhh 6d ago
    I'm optimistic.
    - doubletwoyou 6d ago
      Sir Overstreet still seems insistent on having done no wrong on account of fixing user bugs taking priority over literally everything, so clearly there’s a mindset clash that’s gonna prevent a re-merging unless that Torvalds decides to change the same procedure he’s been doing for the past 2 decades or Overstreet knocks his own ego down a peg. Edit: I think this makes for some good reading: https://lkml.org/lkml/2025/8/9/427 https://lkml.org/lkml/2025/8/9/427
      - gdgghhhhh 6d ago
        The key is forming a team of maintainers.
        koverstreet 5d ago
        If you want to be a filesystem maintainer, you start out by doing the work involved. I'm afraid not all opinions are created equal, this isn't design by committee :)
    - MBCook 6d ago
      I’ve been following kernel development for something like three years using LWN and other sources. I can’t remember anyone earning the level of hostile treatment he did and getting kicked out. I don’t see how it would ever happen.
vilas900420 6d ago
[flagged]
olavgg 6d ago
The numbers are quite interesting, does bcachefs run on Almalinux 10? I only see Fedora supported. All my servers run Almalinux today and I am super happy with it, no plans to use something else. How is root support for bcachefs? This is the one thing I really miss with ZFS today, its just too much work. I do love my FreeBSD systems with a root ZFS though.
- d12bb 6d ago
  Don’t know Almalinux, but can vouch for bcachefs root being just fine with NixOS.
cromka 6d ago
At this point with these results I wonder if there’s big corp backing for bcachefs yet? I’d imagine they’d want to migrate to it at some point for some of their uses.
- koverstreet 6d ago
  Yes, has been for awhile
  - cromka 6d ago
    Great to hear. Keep up the great work and I hope that with time and project maturity, I.e. when the code changes are more of the maintenance than “new feature” nature, it makes its way back to the mainline just fine. Also hope Proxmox decides to incorporate it somewhat soon in PVE.
fenio 6d ago
First hybrid tier run: https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/hybrid-tier/ https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/hybrid... It is FIRST run... I will probably start tuning it now. And I'm open for suggestions what and how to tune ;)
ttiurani 6d ago
I'd be interested to see what the upcoming Btrfs performance improvements in linux 7.3 do for these benchmarks. https://www.phoronix.com/review/linux-73-btrfs https://www.phoronix.com/review/linux-73-btrfs
lproven 6d ago
Interesting although I'd have liked more summaries: there's an awful lot there. But the reasons I choose filesystems are more about reliability, failure modes, surrounding tooling, and so on. Btrfs fails in several critical areas: 1. No way to accurately find free space 2. catastrophic failure on write if a volume fills up, the probability of which is greater because of #1 3. repair tools usually do not recover a corrupted volume and in my testing are most likely to render as damaged volume completely unreadable, which makes #2 worse Put these things together and I can never trust Btrfs again. In the 9 years since I encountered these, I see no effort to fix them, just fooling around witg unimportant side details like performance tweaks. Fix the critical issues first then make it faster.
- raegis 6d ago
  Does the report say any of this? I only see "FAIL" on a few tests with ext4 and one with xfs.
  - koverstreet 6d ago
    It's hard to show with any accuracy how likely a filesystem is to not break when the SHTF or something weird happens, or if they've handled all the weird corner cases, with any kind of automated test. For that you have to dig into the methodology, look at the code, look at user reports, etc. But you can get a pretty good approximation just from the philosophies and attitudes of the engineers and what they're talking about. The talk I just gave at the Rust for Linux conference was all about that - how do we make the system debugable, the community aspect of how we respond to bug reports and talk to users, the prep work for the Rust conversion and formal verification and how we're approaching all that. Reliability doesn't come out of nowhere, "all bugs are shallow with enough eyeballs" really doesn't apply to filesystems. You just have to plan for it, come up with a methodology, and do the work.
  - lproven 6d ago
    It's not just me and it's had serious problems for years: https://arstechnica.com/gadgets/2021/09/examining-btrfs-linuxs-perpetually-half-finished-filesystem/ https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu... <- 5Y ago. It's not materially better now. The devs are in denial about the problems because lots of big users are saying "works fine on my machine." Sure, if you have lots of backups, if you have huge volumes on huge disks and they never fill up... But it's the default in Fedora, Spiral Linux, Garuda Linux, siduction and others. Personal distros for people's own PCs and those are not well-supported enterprise kit.
hard_times 6d ago
Excuse my ignorance, but why didn't you benchmark on a RAM disk for more accurate results?
jmpman 6d ago
I'm interested in seeing these results on cloud instances. I'd expect rather extreme differences in latency.
tomsmart_ai 6d ago
[flagged]
bwl1289 6d ago
[dead]
ranomier 6d ago
[dead]

174 points by farlight 7d ago

https://github.com/fenio/modern-fs-benchmark https://github.com/fenio/modern-fs-benchmark

@farlight assuming that you're the creator do you think you'd be able to rework the HTML/CSS? I'm sure you've got good data but speaking on behalf of my eyeballs, the results page is... hard to read!

Farmadupe 7d ago
> 2G of random garbage is written directly onto one member device (behind the filesystem's back, offset 1G — python injector; uutils dd mis-seeks on dm devices), caches dropped, then a full scrub: btrfs scrub -B, zpool scrub + wait, bcachefs scrub, md/lvm sync-action 'check' (which can only COUNT mismatches — no checksums to know which copy is right). I'm not sure that nuking 2G of the underlying block device is a recoverable error on any filesystem that I'm aware of? Can you confirm if any ofthe filesystems really came out of the other side in a usable state after scrubbing? ----- > Trivial-op p99, idle (ms) # A trivial operation — one 4k write + fsync every 200ms (like a shell appending history or an editor updating its swap file) — run alone for 10s. p99 of the fsync completion In fact, if it's OK for me to ask, are any of the metrics tht you used standard industry metrics? It looks like several of the tests are bypassing the kernel's page cache? -- which I worry may fall into the trap of "I modified the system to be unrepresentative of reality and then tested it". ---- > kernel 7.0.0-1012-azure Can you confirm if you tested on a bare metal machine? were you the only tenant?
- vlovich123 7d ago
  1 device out of the replica set I’m assuming so all of them should recover.
- hlieberman 7d ago
  The integrity check is only on the tests which are either RAID or the filesystem equivalent.
- matja 7d ago
  > I'm not sure that nuking 2G of the underlying block device is a recoverable error on any filesystem that I'm aware of? ZFS and btrfs were designed from the start to handle this, by using checksums on every piece of (meta)data and redundancy to return the same data as was stored to the kernel, and rewrite the bad data. I've tested my own machines running ZFS by random writes out of band from the filesystem/kernel and it has always found and fixed them.
- fenio 6d ago
  I answered in other thread how exactly integrity check looks like and why the name might be misleading. I'm also in the middle of updating descriptions to be more adequate. Thanks for pointing that out. Speaking about standard industry metrics. fio is an established tool, and throughput, IOPS, fsync latency, and percentiles are standard concepts. However, the exact job recipes and the composite “Overall Core” score are project-specific. The trivial-operation test is also custom: one 4 KiB write plus fsync every 200 ms. The idle window contains at most about 50 operations, so its p99 is effectively the slowest sample’s fio histogram bucket, not a statistically stable population percentile. It should be treated as a small-write durability-latency probe, not a universal application metric. But after all all tests are in the repository. If they need tweaks, changes I'm open to do so... I started from scratch and did whatever came to my mind. Some tests are added after my initial link here which went mostly unnoticed several weeks ago but I got some requests for more tests which I implemented. But to sum up. I want this test to be useful so feel free to open PRs with improvements. It's not like I've got some agenda. In fact I wrote here and there on the page that I'm counting on communities of various filesystems to provide improvements, changes etc to make their filesystem shining. This is personal project made when I realized that multiple-devices benchmarks were almost completely absent. Since I had not access to real hardware I decided to make at least initially everything based on GH runner with all the limitations that came with this approach. I tried to limit these limitations as far as I could. But feel free to submit bugreports, PRs, propositions for improvements.
fenio 7d ago
what exactly would you like to improve?
- Farmadupe 7d ago
  I'd actually recommend some excellent books on the "philosophy" of data presentation: The first that comes to mind is "the visual display of quantitative information" by Edward Tufte seems to be freely available online, and the other one on my mind is "how charts lie" by Alberto Cairo (which doesn't seem to be freely accessible) But if it helps, just some "initial gut feel observations" from me: * It's definitely not possible to find issue with the the _sheer amount_ of results, but there's just far too much for a human to absorb, all presented at once * Overall text size is quite small, and difficult to read * The page doesn't make a strong statement of _what_ is under test: the first words are: "modern-fs-benchmark Multi-device CoW filesystems under workloads classic benchmarks skip" -- which defines the webpage in terms of what it is _not_, without stating what benchmarks are actually present. * The first line of teh page contains run statistics that probably eithre want to b at the bottom, or just don't need to be in the webpage at all: "latest run 2026-09-18 18:50:45 UTC, kernel 7.0.0-1012-azure, 593 runs recorded · 145 trend points shown" * A significant proportion of the free text is caveats. There's nothing wrong with being transparent about limitations, but they may be a sign that there might be alternative ways to present the data, or that the data may be flawed (depending on the caveat) * Theres several categories that I think have been invented for the purpose of collation, but I don't think are defined on the page. I think "Overall Core" and "Core I/O" aren't explained, which means by definition it's impossible for a reader to understand the score table. * And as we're all aware right now, current Claude models are currently struggling to write coherent English. There's several incoherent sentences on the page. It's a Claude issue.
  - koverstreet 7d ago
    Speaking as someone who consumes this, I appreciate how it's laid out. As a developer, we can often see at a glance where the bottleneck is if we have enough data laid out - IOW, data overload for you is me feeling like a kid in a candy store. Sometimes there are ways to make things easier without dumbing them down, but way too many people conflate the two; I get nervous when non engineers say "I've studied this, it should be easy".

skerit 7d ago

Oh, so bcachefs is doing pretty well.

tarruda 7d ago
Except for the fact that the developer has sabotaged the project into being removed from mainline?
- AceJohnny2 7d ago
  that's not necessarily a sabotage.
  - eikenberry 7d ago
    Sabotage might not be the best word, but it hurt trust and adoption.
    - koverstreet 7d ago
      It's just been a lot less drama within the project since the split. I do have a lot more pull requests to merge than I did before. I don't know if you want to count "Kent isn't reviewing PRs fast enough" as drama :)
- tombert 7d ago
  It's relatively easy to get it working as a kernel module at least. I got it set up on a NixOS box without too much trouble.
  - petre 7d ago
    Not something I'd want to do with my filesystem though. ZFS is different story it it isn't on the root fs because it might be packaged by the distribution.
    - tombert 7d ago

blop 7d ago

For peace of mind I'm still using zfs (since the last 15+ years) but I'm definitely not impressed by the performance...

slyfox125 7d ago
Different tools for different jobs; use ZFS for your data store and ext4 for your primary drive.
- blop 7d ago
  yes indeed, zfs for my nas basically
badgersnake 7d ago
zfs-mirror-8k (whatever that is) looks alright.
- deleted 6d ago
  [deleted]
- magicalhippo 6d ago
  The script[1] shows it's a ZFS mirror (RAID-1) with 8k max record size, as opposed to the default 128k. I say max, as ZFS stores multiples of the physical block size up to the record size. So even with 128k record size, if your file fits in two physical blocks, it will write two physical blocks of data. The `ashift` parameter[2] controls the physical block size, typically 512 bytes or 4k for HDDs. Though higher can be useful on SSDs (but less tested, have seen some bug reports with >4k block sizes). Typically you'd only want to use such small records if you are tuning it for a specific workload, like if you have a database that writes 8k pages. Compression acts on records, so by limiting the record size, you limit the effectiveness of the compression: 1.2 and 1.8 blocks worth of data both gets written as 2 physical blocks. On the upside, record size is a dataset property, so you can have many different datasets with different record sizes on the same pool. [1]: https://github.com/fenio/modern-fs-benchmark/blob/599ec72fe335e2c043d83daa69c870fcf9fa8e72/scripts/fs/zfs.sh#L34 https://github.com/fenio/modern-fs-benchmark/blob/599ec72fe3... [2]: https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Workload%20Tuning.html#alignment-shift-ashift https://openzfs.github.io/openzfs-docs/Performance%20and%20T...

deleted 7d ago

[deleted]

loeg 7d ago

What is md-raid10 doing that is so much worse than lvm-raid10? In terms of "I/O" and "responsiveness." It's not really obvious to me from either the linked page or https://github.com/fenio/modern-fs-benchmark https://github.com/fenio/modern-fs-benchmark . In principle they should be similar?

jmalicki 7d ago
[flagged]
deleted 7d ago
[deleted]

Farmadupe 7d ago

> CI runs use loop devices on shared ephemeral VMs (one VM per filesystem): compare shapes and ratios, not absolute MB/s. Each job records a host-calibration anchor — see the table. I think if you're not using baremetal for such tests, it's likely that the results are simply not comparable at all? What if another tenant is also using the disk?

walrus01 7d ago
It's a fair point but it's also possible the person running the tests has a dedicated test hypervisor for this , so that different configurations of filesystems and VMs can be created and destroyed quickly in an automated manner. If it's something as simple as a KVM hypervisor that only runs 1 test VM at a time (with no other load from anything else other than the basic systemd daemons, ssh daemon etc running on the hypervisor), the results could be very close to bare metal. I can see it being very time consuming and annoying to do repeated manual bare metal OS installs and new partitioning/filesystem creation for such a large variety of tests. The author does also say that performance isn't really the main thing but rather, data integrity: https://github.com/fenio/modern-fs-benchmark https://github.com/fenio/modern-fs-benchmark
- Farmadupe 7d ago
  > compare shapes and ratios, not absolute MB/s In this case, given that the author's own disclaimer (above) already disclaims the numeric readings, I'm not sure how it's possible to make any inference on "shapes and ratios" derived from the numeric readings.
- toast0 7d ago
  > I can see it being very time consuming and annoying to do repeated manual bare metal OS installs. Well don't do that then. There's lots of other options. Probably the simplest is a single bare metal install on a simple filesystem on one device. run the filesystems under test on other storage dedicated to testing. You could also boot into a network install and use local storage exclusively for testing.
  - 7d ago

blop 7d ago

I think the reviews should also include the social aspect of these filesystems... There is and have been many promising and exciting FS to replace the old boring ones, but for storage you not only want to avoid technical issues but also maintainer(s) drama...

koverstreet 7d ago
Why do people keep bringing up drama? The community infighting has sucked, but that's a thing that matters primarily for maintainers. I think most users just want something that works.
- Skunkleton 7d ago
  Related username? To answer the original question, most people who care about their filesystem at all care about its stability. Not just "does it work now" but also "will it work and improve over time". Infighting puts the future at risk.
  - koverstreet 7d ago
    It really does. But you might want to check out the bus factor on btrfs too; when a maintainer says "but we've saved Facebook billions and billions of dollars!", calls for the other filesystem maintainer to be ejected from the community, then quits to join Anthropic a month later - that's not a vote of confidence. I'd be very happy if people could just stop bringing up drama and us factors. We put it behind us a year ago, but it seems not everyone got the memo.
    - nolist_policy 6d ago
      Checking the bus factor: Btrfs regulars: - 1 from Meta - 1 from Oracle - 4 from SuSe - 2 from WDC bcachefs: - Kent Overstreet
      - r0l1 6d ago

markhahn 7d ago

what does "integrity" fail mean in the first table? that the case didn't recover from the 2G corruption?

fenio 7d ago
The current “Integrity” label is broader than the test actually proves, and I’m changing it to “Corruption probe.” FAIL means that file changed or became unreadable. SURVIVED means only that the file remained readable and hash-identical. It does not prove the entire filesystem was healthy, that every overwritten byte was allocated, or that all 2 GiB were repaired. Thanks for pointing it.

sippingabonedry 7d ago

So two filesystems that are essentially shunned from the Linux kernel and permanent second-class citizens, and one that was removed from Red Hat and has a questionable history of reliability. Oh boy which do I choose? I'm saying ZFS on another OS.

petre 7d ago
xfs/lvm-raid10 or ext4/lvm-raid10, obviously.
- sippingabonedry 7d ago
  If I'm forced to use Linux, sure. I could drop bricks on and cord pull those all day and they would not lose data. Which is a small ask for a filesystem IMO.
- andriy_koval 6d ago
  would want to have compression..
  - simoncion 6d ago
    > would want to have compression.. I learned about VDO today. [0][1] I'd never heard of it before, but I'm using Gentoo Linux and both the dm-vdo kernel module and the 'vdo' software provided by [1] are distributed by Gentoo... so this isn't some weirdo Red Hat thing. It looks like you manage and used these just like any other thin-provisioned LVM volume [2], but -like I said- I've not used this before, and have only just skimmed the docs, so it's possible that I'm missing something important. [0] <https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/deduplicating_and_compressing_logical_volumes_on_rhel/introduction-to-vdo-on-lvm_deduplicating-and-compressing-logical-volumes-on-rhel https://docs.redhat.com/en/documentation/red_hat_enterprise_...> [1] <https://github.com/dm-vdo/vdo https://github.com/dm-vdo/vdo> [2] <https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/deduplicating_and_compressing_logical_volumes_on_rhel/creating-a-deduplicated-and-compressed-logical-volume_deduplicating-and-compressing-logical-volumes-on-rhel#creating-and-mounting-an-lvm-vdo-volume_creating-a-deduplicated-and-compressed-logical-volume https://docs.redhat.com/en/documentation/red_hat_enterprise_...>

magicalhippo 7d ago

> Every push/2-hourly cron builds each filesystem across 4 loop devices backed by sparse files, runs the suite, and publishes a results table in the job summary plus JSON artifacts. I get that real hardware costs (author mentions EUR 70 a month for a suitable server), but without at least a baseline snapshot comparison run between real hardware, both SSD and HDD, and the sparse file-backed loop devices, it's hard to take much away from this. Sadly the AI apocalypse isn't making stuff like this easy to do as a hobby.

fenio 7d ago

The author of the benchmark here. I went over some comments and I'll try to tackle them here. I'm pretty clear that GH runner based benchmark is far from perfect due to noisy neighbours etc. Thus every test first is running so called calibration... to reject completely unreliable VMs. I'm fully aware that this can't completely fix the issue. Can limit it but not fix. But as of now there are 593 runs recorded so average should still be quite meaningful. Having that said I'm desperately trying to get REAL hardware to run that benchmark. With some successes ;) Few months ago I got Hetzner machine from Kent Overstreet and I was able to finish 3 runs before machine died... Results: https://bartosz.fenski.pl/modern-fs-benchmark/real-hw/ https://bartosz.fenski.pl/modern-fs-benchmark/real-hw/ Currently I've got even more interesting machine with tons of disks and I'm running new set of benchmarks but it's really in its initial stage. https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/ https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/ 2nd run in progress... one run on REAL hardware takes much more time than on GH runner so it's slow. But this new hardware has also so many disks that the plan is to try also more complex, tiered cache topologies. I'm working on it. I'm happy to answer any other questions, sources of every piece of this benchmark are freely available and I'm not saying they are 100% correct. I'm open to improvements.

koverstreet 7d ago
I went back and forth with Hetzner a couple times, I think we just got a bad machine :) I've been saying it for months, but eventually I'm going to move the automated builds off the 48 core monster and we'll be able to use that for automated perf testing too. The machine we just got has spindles for EC perf testing, but the Hetzner monster has very high end enterprise ssdd. Also, just got done with the Rust for Linux conference, still not home but here's slides that still need reformatting: https://evilpiepirate.org/~kent/Kangrejos-2026-bcachefs.pdf https://evilpiepirate.org/~kent/Kangrejos-2026-bcachefs.pdf
Joel_Mckay 7d ago
Should include: 1. Dual Ext4 + external 32GB journal X4 pcie SSD (the prior winner of benchmark surveys) 2. Bare F2FS after a trim and SSD vendor software cache flush operation (it should be slower, but knowing how much slower on identical hardware could be interesting.) 3. DRBD across a 48U 100Gbps host rack (single X4 pcie data drive per host, OS on primary) 4. CephFS across a 48U 100Gbps host rack (single X4 pcie data drive per host, OS on primary) Best regards =3

irusensei 7d ago

BCacheFS is the best Linux filesystem now that storage costs a premium. You can mix devices of different sizes and types on bcachefs. You can have foreground and background devices to balance performance and also different compression settings for foreground and background transactions. You can set replicas=N to the individual file or directory on bcachefs. For example files you can just re-download or re-build. Likewise you can set a higher number of copies to important files.

suncore 7d ago
Sure, when it's mature...
- mkeedlinger 7d ago
  Yeah, and I think Overstreet might be going hard on AI, so not sure how that might effect the project
  - koverstreet 7d ago
    If you have "refine until it's perfect and don't screw with things you don't understand" thoroughly ingrained, along with the dangers of overconfidence, you'll do fine with AI. Not everyone gets it though, that's for sure. And, if you want to know if it's mature, I'd trust the user reports over the one liners :)
lukaslalinsky 6d ago
I've been using ZFS on my Linux servers for years and have pretty good experience with it. I've not been following Linux development too much. Is bcachefs usable/stable/reliable enough to replace it?
- khajdamowicz 6d ago
  It's not as battle-tested like ZFS with 15+ years of field usage, but it's good enough for me and other users. There's a NAS appliance called NASty and it shares publicly usage stats: https://nasty-telemetry.pages.dev/ https://nasty-telemetry.pages.dev/ Those numbers are NASty alone. There are more users on other systems.

azatom 7d ago

clicked, ctrl+f ssd, hdd, not found, closed

azatom 7d ago
In filesystem subject a proper critique is mentioning ssd vs hdd. I did not added value for karma increase, but karma lost? Who haven't I wished a happy birthday?
fenio 6d ago
The new machine - I mentioned in the other thread - has both HDDs and SSDs so tests with mixed topologies are planed.

bhaney 7d ago

Seeing great results from bcachefs just makes me more sad that Kent and the other kernel devs couldn't come to an understanding to keep bcachefs in-tree. I want to use it for my storage arrays so badly, but I'm still stuck with btrfs as the only available in-tree filesystem with modern features.

koverstreet 7d ago
NixOS. You can't go wrong.
- Sha1rholder 6d ago
  What do u mean NixOS? It's an OS not a file system
  - koverstreet 6d ago
    NixOS thoroughly solves all the external module fragility. They also do distro level CI testing.
khajdamowicz 6d ago
You can use NASty as NAS appliance. It's based on NixOS, offers flexibility, atomic upgrades and all bells and whistles of bcachefs.
locknitpicker 6d ago
> Seeing great results from bcachefs just makes me more sad that Kent and the other kernel devs couldn't come to an understanding to keep bcachefs in-tree. I think you are sugarcoating the shit show that was bcachefs's history of involvement in the linux kernel. I mean, do I need to mention that the person was subjected to a code of conduct enforcement action due to his long history of abuse and unprofessional behavior? https://lwn.net/Articles/999197/ https://lwn.net/Articles/999197/
- bhaney 6d ago
  No, you probably didn't need to mention that.

viraptor 7d ago

It could use some explanation for the exact setups. Like how many drives are there in each raid, what does the ec config actually look like, etc. As far as I can see it's not described.

fenio 6d ago
I extended info about setup. Should be visible in top part of benchmark.

nullc 7d ago

It'll be interesting when people start applying similar effort that is going into LLM search for security vulnerabilities to finding conventional reliability/performance bugs.

pzmarzly 7d ago

xfs is by far the slowest in "ftruncate empty file to 1G" but the fastest in "ftruncate 256M file to 512M" - does anyone know why?

pzmarzly 7d ago
Seems like a bug in benchmark code for that test, opened GH issue https://github.com/fenio/modern-fs-benchmark/issues/35 https://github.com/fenio/modern-fs-benchmark/issues/35
- fenio 6d ago
  Should be fixed. Thanks.

deleted 6d ago

[deleted]

gdgghhhhh 6d ago

In case someone wonders why bcachefs is sadly no longer part of the mainline kernel. Start reading here: https://lwn.net/Articles/1027289/ https://lwn.net/Articles/1027289/

lifty 6d ago
Is there any chance bcachefs will be included again in the future?
- gdgghhhhh 6d ago
  I'm optimistic.
  - doubletwoyou 6d ago
    Sir Overstreet still seems insistent on having done no wrong on account of fixing user bugs taking priority over literally everything, so clearly there’s a mindset clash that’s gonna prevent a re-merging unless that Torvalds decides to change the same procedure he’s been doing for the past 2 decades or Overstreet knocks his own ego down a peg. Edit: I think this makes for some good reading: https://lkml.org/lkml/2025/8/9/427 https://lkml.org/lkml/2025/8/9/427
    - gdgghhhhh 6d ago
      The key is forming a team of maintainers.
      - koverstreet 5d ago
        If you want to be a filesystem maintainer, you start out by doing the work involved. I'm afraid not all opinions are created equal, this isn't design by committee :)
  - MBCook 6d ago
    I’ve been following kernel development for something like three years using LWN and other sources. I can’t remember anyone earning the level of hostile treatment he did and getting kicked out. I don’t see how it would ever happen.

vilas900420 6d ago

[flagged]

olavgg 6d ago

The numbers are quite interesting, does bcachefs run on Almalinux 10? I only see Fedora supported. All my servers run Almalinux today and I am super happy with it, no plans to use something else. How is root support for bcachefs? This is the one thing I really miss with ZFS today, its just too much work. I do love my FreeBSD systems with a root ZFS though.

d12bb 6d ago
Don’t know Almalinux, but can vouch for bcachefs root being just fine with NixOS.

cromka 6d ago

At this point with these results I wonder if there’s big corp backing for bcachefs yet? I’d imagine they’d want to migrate to it at some point for some of their uses.

koverstreet 6d ago
Yes, has been for awhile
- cromka 6d ago
  Great to hear. Keep up the great work and I hope that with time and project maturity, I.e. when the code changes are more of the maintenance than “new feature” nature, it makes its way back to the mainline just fine. Also hope Proxmox decides to incorporate it somewhat soon in PVE.

fenio 6d ago

First hybrid tier run: https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/hybrid-tier/ https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/hybrid... It is FIRST run... I will probably start tuning it now. And I'm open for suggestions what and how to tune ;)

ttiurani 6d ago

I'd be interested to see what the upcoming Btrfs performance improvements in linux 7.3 do for these benchmarks. https://www.phoronix.com/review/linux-73-btrfs https://www.phoronix.com/review/linux-73-btrfs

lproven 6d ago

Interesting although I'd have liked more summaries: there's an awful lot there. But the reasons I choose filesystems are more about reliability, failure modes, surrounding tooling, and so on. Btrfs fails in several critical areas: 1. No way to accurately find free space 2. catastrophic failure on write if a volume fills up, the probability of which is greater because of #1 3. repair tools usually do not recover a corrupted volume and in my testing are most likely to render as damaged volume completely unreadable, which makes #2 worse Put these things together and I can never trust Btrfs again. In the 9 years since I encountered these, I see no effort to fix them, just fooling around witg unimportant side details like performance tweaks. Fix the critical issues first then make it faster.

raegis 6d ago
Does the report say any of this? I only see "FAIL" on a few tests with ext4 and one with xfs.
- koverstreet 6d ago
  It's hard to show with any accuracy how likely a filesystem is to not break when the SHTF or something weird happens, or if they've handled all the weird corner cases, with any kind of automated test. For that you have to dig into the methodology, look at the code, look at user reports, etc. But you can get a pretty good approximation just from the philosophies and attitudes of the engineers and what they're talking about. The talk I just gave at the Rust for Linux conference was all about that - how do we make the system debugable, the community aspect of how we respond to bug reports and talk to users, the prep work for the Rust conversion and formal verification and how we're approaching all that. Reliability doesn't come out of nowhere, "all bugs are shallow with enough eyeballs" really doesn't apply to filesystems. You just have to plan for it, come up with a methodology, and do the work.
- lproven 6d ago
  It's not just me and it's had serious problems for years: https://arstechnica.com/gadgets/2021/09/examining-btrfs-linuxs-perpetually-half-finished-filesystem/ https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu... <- 5Y ago. It's not materially better now. The devs are in denial about the problems because lots of big users are saying "works fine on my machine." Sure, if you have lots of backups, if you have huge volumes on huge disks and they never fill up... But it's the default in Fedora, Spiral Linux, Garuda Linux, siduction and others. Personal distros for people's own PCs and those are not well-supported enterprise kit.

hard_times 6d ago

Excuse my ignorance, but why didn't you benchmark on a RAM disk for more accurate results?

jmpman 6d ago

I'm interested in seeing these results on cloud instances. I'd expect rather extreme differences in latency.

tomsmart_ai 6d ago

[flagged]

bwl1289 6d ago

[dead]

ranomier 6d ago

[dead]