8 ms·
Jepsen: NATS 2.12.1
- selectodude 10mo agoDefinitely thought this was about aviation for a moment.
- sam_lowry_ 10mo ago[flagged]
- Infiltrator 10mo agoLikewise. It took me a moment to realise Jepsen!== Jeppesen
- selectodude 10mo agoAnd NATS being the North Atlantic tracks.
- crote 10mo agoIt's named after Carly Rae Jepsen, of 2012 hit single "Call Me Maybe" fame.
- loeg 10mo agoI think Aphyr will insist it isn't actually named after Carly Rae for legal reasons, just a striking coincidence.
- the__alchemist 10mo agoYea! I did a double-take, as in addition to Jeppesen, NATS is something I worked with in the past as a UK NOTAM service.
- vrnvu 10mo agoSort of related. Jepsen and Antithesis recently released a glossary of common terms which is a fantastic reference. https://jepsen.io/blog/2025-10-20-distsys-glossary https://jepsen.io/blog/2025-10-20-distsys-glossary
- merb 10mo ago> 3.4 Lazy fsync by Default Why? Why do some databases do that? To have better performance in benchmarks? It’s not like that it’s ok to do that if you have a better default or at least write a lot about it. But especially when you run stuff in a small cluster you get bitten by stuff like that.
- dilyevsky 10mo agoMassively improves benchmark performance. Like 5-10x
- speedgoose 10mo ago/dev/null is even faster.
- formerly_proven 10mo ago/dev/null tends to lose a lot more data.
- onionisafruit 10mo agoJust wait until the jepsen report on /dev/null. It's going to be brutal.
- orthoxerox 10mo ago/dev/null works according to spec, can't accuse it of not doing something it has never promised
- thinkharderdev 10mo ago> To have better performance in benchmarks Yes, exactly.
- millipede 10mo agoI always wondered why the fsync has to be lazy. It seems like the fsync's can be bundled up together, and the notification messages held for a few millis while the write completes. Similar to TCP corking. There doesn't need to be one fsync per consensus.
- clemlesne 10mo agoNATS is a fantastic piece of software. But doc’s unpractical and half backed. That’s a shame to be required to retro engineer the software from GitHub to know the auth schemes.
- belter 10mo ago[flagged]
- hurturue 10mo agodo you have a better solution? as they would say, NATS is a terrible message bus system, but all the others are worse
- adhamsalama 10mo agoAre RabbitMQ's durable queues worse?
- johncolanduoni 10mo agoPulsar can do most of what NATS can, but at a much higher cost in both compute and operations (though I haven’t seen a head-to-head of each with durability turned on), along with some simply different characteristics (like NATS being suitable for sidecar deployment). NATS is fantastic for ephemeral messaging, but some of this report is really concerning when JetStream has been shipping for years.
- Thaxll 10mo ago"PostgreSQL used fsync incorrectly for 20 years" https://archive.fosdem.org/2019/schedule/event/postgresql_fsync/ https://archive.fosdem.org/2019/schedule/event/postgresql_fs... It did not prevent people from using it. You won't find a database that has the perfect durability, ease of use, performance ect.. It's all about tradeoffs.
- dijit 10mo agoRealistically speaking, postgresql wasn’t handling a failed call to fsync, which is wrong: but materially different from a bad design or errors in logic stemming from many areas. Postgresql was able to fix their bug in 3 lines of code, how many for the parent system? I understand your core thesis (sometimes durability guarantees aren’t as needed as we think) but in postgresql’s case, the edge was incredibly thin. It would have had to have been: a failed call to fsync and a system level failure of the host before another call to fsync (which are reasonably common). It’s far too apples to oranges to be meaningful to bring up I am afraid.
- gostsamo 10mo agoThanks, those reports are always a quiet pleasure to read even if one is a bit far from the domain.
- rdtsc 10mo ago> By default, NATS only flushes data to disk every two minutes, but acknowledges operations immediately. This approach can lead to the loss of committed writes when several nodes experience a power failure, kernel crash, or hardware fault concurrently—or in rapid succession (#7564). I am getting strong early MongoDB vibes. "Look how fast it is, it's web-scale!". Well, if you don't fsync, you'll go fast, but you'll go even faster piping customer data to /dev/null, too. Coordinated failures shouldn't be a novelty or a surprise any longer these days. I wouldn't trust a product that doesn't default to safest options. It's fine to provide relaxed modes of consistency and durability but just don't make them default. Let the user configure those themselves.
- CuriouslyC 10mo agoNATS data is ephemeral in many cases anyhow, so it makes a bit more sense here. If you wanted something fully durable with a stronger persistence story you'd probably use Kafka anyhow.
- nchmy 10mo agoCore nats is ephemeral. Jetstream is meant to be persisted, and presented as a replacement for kafka
- petre 10mo agoSo is MQTT, why bother with NATS then?
- KaiserPro 10mo agoMQTT doesn't have the same semantics. https://docs.nats.io/nats-concepts/core-nats/reqreply https://docs.nats.io/nats-concepts/core-nats/reqreply request reply is really useful if you need low latency, but reasonably efficient queuing. (making sure to mark your workers as busy when processing otherwise you get latency spikes. )
- RedShift1 10mo ago
- maxmcd 10mo ago> > You can force an fsync after each messsage [sic] with always, this will slow down the throughput to a few hundred msg/s. Is the performance warning in the NATS possible to improve on? Couldn't you still run fsync on an interval and queue up a certain number of writes to be flushed at once? I could imagine latency suffering, but batches throughput could be preserved to some extent?
- scottlamb 10mo ago> Is the performance warning in the NATS possible to improve on? Couldn't you still run fsync on an interval and queue up a certain number of writes to be flushed at once? I could imagine latency suffering, but batches throughput could be preserved to some extent? Yes, and you shouldn't even need a fixed interval. Just queue up any writes while an `fsync` is pending; then do all those in the next batch. This is the same approach you'd use for rounds of Paxos, particularly between availability zones or regions where latency is expected to be high. You wouldn't say "oh, I'll ack and then put it in the next round of Paxos", or "I'll wait until the next round in 2 seconds then ack"; you'd start the next batch as soon as the current one is done.
- ADefenestrator 10mo agoYes, this is a reasonably common strategy. It's how Cassandra's batch and group commit modes work, and Postgres has a similar option. Hopefully NATS will implement something similar eventually.
- stmw 10mo agoEvery time someone builds one of these things and skips over "overcomplicated theory", aphyr destroys them. At this point, I wonder if we could train an AI to look over a project's documentation, and predict whether it's likely to lose commmitted writes just based on the marketing / technical claims. We probably can.
- awesome_dude 10mo ago/me strokes my long grey beard and nods People always think "theory is overrated" or "hacking is better than having a school education" And then proceed to shoot themselves in the foot with "workarounds" that break well known, well documented, well traversed problem spaces
- whimsicalism 10mo agocertainly a narrative that is popular among the grey beard crowd, yes. in pretty much every field i've worked on, the opposite problem has been much much more common.
- _zoltan_ 10mo agowhat's the opposite problem statement?
- MrDarcy 10mo agoThe ivory tower standing in the way of delivering value I think.
- colechristensen 10mo agoTo be more specific, goals of perfection where perfection does not at all matter.
- johncolanduoni 10mo ago
- dzonga 10mo agonats jetstream vs say redis streams - which one have people found easier to work with ?
- ViewTrick1002 10mo agoWhen I worked with bounded Redis streams a couple of years ago we had to implement our own backpressure mechanism which was quite tricky to get right. To implement backpressure without relying on out of band signals (distributed systems beware) you need to have a deep understanding of the entire redis streams architecture and how the the pending entries list, consumers groups, consumers etc. works and interacts to not lose data by overwriting yourself. Unbounded would have been fine if we could spill to disk and periodically clean up the data, but this is redis. Not sure if that has improved.
- ubercore 10mo agoI don't have a direct comment to add, but after working on the fringes of streams a bit, they've worked as advertised, but the API surface area for them is full of cases where, as you say, you have to kind of internalize the full architecture to really understand what's going on. It can be a bit overwhelming.
- johncolanduoni 10mo agoWow. I’ve used NATS for best-effort in-memory pub/sub, which it has been great for, including getting subtle scaling details right. I never touched their persistence and would have investigated more before I did, but I wouldn’t have expected it to be this bad. Vulnerability to simple single-bit file corruption is embarrassing.
- rishabhaiover 10mo agoNATS be trippin, no CAP.
- veverkap 10mo agoUnderrated
- deleted 10mo ago[deleted]
- williamstein 10mo agohttps://github.com/williamstein/nats-bugs https://github.com/williamstein/nats-bugs
- williamstein 10mo agoFor example, https://github.com/williamstein/nats-bugs/issues/5 https://github.com/williamstein/nats-bugs/issues/5 links to a discussion I have with them about data loss, where they fundamentally don't understand that their incorrect defaults lead to data loss on the application side. It's weird. I got very deep into using NATS last year, and then realized the choices it makes for persistence are really surprising. Another horrible example if that server startup time is O(number of streams), with a big constant; this is extremely painful to hit in production. I ended up implementing from scratch something with the same functionality (for me as NATS server + Jetstream), but based on socket.io and sqlite. It works vastly better for my use cases, since socketio and sqlite are so mature.
- PaoloBarbolini 10mo agoThere are many things they don't seem to understand about their own product. https://github.com/nats-io/nats.rs/issues/1253#issuecomment-2063047468 https://github.com/nats-io/nats.rs/issues/1253#issuecomment-...
- shikhar 10mo agoIf you are looking for a serverless alternative to JetStream, check out https://s2.dev https://s2.dev Pros: unlimited streams with the durability of object storage – JetStream can only do a few K topics Cons: no consumer groups yet, it's on the agenda
- embedding-shape 10mo agoHave you tried running Jepsen against it?
- shikhar 10mo agoWe do deterministic simulation testing https://s2.dev/blog/dst https://s2.dev/blog/dst https://s2.dev/blog/linearizability https://s2.dev/blog/linearizability We have also adopted Antithesis for a more thorough DST environment, and plan to do more with it. One day we will engage Kyle to Jepsen, too. I'm not sure when though.
- Kinrany 10mo agoNATS claims to use Antithesis as well, so that's nothing comparatively speaking
- embedding-shape 10mo agoI guess that's better than nothing. But now I'm unsure what your original comment was about, if your project doesn't use Jepsen for testing to "prove" it works fine, how is your project relevant to bring up on a submission about a Jepsen test of some other software? If everyone who was making a database/message queue/whatever distributed system shared their projects on every Jepsen submission, we'd never have any discussions about the actual software in question.
- shikhar 10mo agoIt seemed like the kind of Jepsen outcome where folks would be considering alternatives, but yeah maybe it was not appropriate to plug here.
- mysfi 10mo agoCurious about the differences between content on aphyr.com/tags/jepsen and jepsen.io/analyses. I recently discovered aphyr.com and was excited about the potential insights!
- aphyr 10mo agoJepsen started as a personal blog series in nights and weekends; jepsen.io is when I started doing it professionally, about ten years ago.
- bsaul 10mo agoCurious : do you have a team of people working with you, or is it mostly solo work ? your work is so valuable, i would be scared for our industry if it had a bus factor of 1.
- andersmurphy 10mo agoHighly recommend you check out the interview series they are a lot of fun. > They will refuse, of course, and ever so ashamed, cite a lack of culture fit. Alight upon your cloud-pine, and exit through the window. This place could never contain you. https://aphyr.com/posts/340-reversing-the-technical-interview https://aphyr.com/posts/340-reversing-the-technical-intervie...
- dangoodmanUT 10mo agoHalf-expected tbh, but didn’t expect to be this bad. Just use redpanda.
- t0i7a1r1a 10mo agohttps://news.ycombinator.com/news https://news.ycombinator.com/news
- sreekanth850 10mo agothis is absolutely shocking!Does kafka do fsync on every write?
- akshayshah 10mo agoNo. Redpanda has made a lot of noise about this over the years [0], and Confluent's Jack Vanlightly has responded in a fair bit of detail [1]. [0]: https://www.redpanda.com/blog/why-fsync-is-needed-for-data-safety-in-kafka-or-non-byzantine-protocols https://www.redpanda.com/blog/why-fsync-is-needed-for-data-s... [1]: https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-doesnt-need-fsync-to-be-safe https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-...
- sreekanth850 10mo agoI think all modern system even scylla db do commit batch no fsync on every write, you either need throughput or durability both cannot exist together. Only thing what redpanda claim is you have to do replication before fsync so your data is not lost if the written node is dead due to a power failure. this is how scylla and cassandra works, if iam not wrong, so even if a node dead before the batch fsync, replication will be done before fsync from memtable,so other nodes will bring the durability and data loss is no longer true in a replicated setup. single node? obviously 100% data loss. but this is the trade off for a high tps system vs durable single ndoe system brings. its how you want to operate.
- menaerus 10mo agoSimilarly in regular SQL systems, the same is achieved by fsyncing to WAL.
- lionkor 10mo agothe article says no )
- jessekv 10mo agoA tiny bit more context here: https://github.com/nats-io/nats-server/discussions/3312#discussioncomment-15203259 https://github.com/nats-io/nats-server/discussions/3312#disc... (I opened this discussion 2.5 years ago and get an email from github every once in a while ever since. I had given up hope TBH)
- jwr 10mo agoFor anyone dealing with databases, and especially distributed databases, I highly recommend reading the Jepsen page on consistency models: https://jepsen.io/consistency/models https://jepsen.io/consistency/models It provides a dictionary of terms that we can use to have educated discussions, rather than throwing around terms like "ACID".
- plandis 10mo agoI love that resource and reference it fairly frequently. There is also this [1] which Aphyr collabed on which you might find interesting if you haven’t seen it yet. [1] https://antithesis.com/resources/reliability_glossary/ https://antithesis.com/resources/reliability_glossary/
- deleted 10mo ago[deleted]