22 ms·

Jepsen: MongoDB 4.2.6

604 points by aphyr 6y ago

sorokod 6y ago
I suppose there are reasons why the defaults are the way they are. Can anyone comment on the implications, performance or otherwise, of bumping up the read/write concerns?
- aphyr 6y ago
  Latency is a big one--you've got to wait an extra round-trip for secondaries to acknowledge primary writes, and primaries (assuming you don't have reliable clocks) need to check in with secondaries to confirm they have the most recent picture of things if you want to do a linearizable read. Snapshot isolated reads shouldn't require that, at least in theory--it's legal to read state from the past under SI, so there's no need to establish present leadership. That's why I'm surprised that MongoDB requires snapshot reads to go through write concern majority--it doesn't seem like it'd be necessary. Might have something to do with sharding--maybe establishing a consistent cut across shards requires a round of coordination. Even then I feel like that's a cost you should be able to pay only at write time, making reads fast, but... apparently not! I'm sure the MongoDB engineers who designed this system have good reasons; they're smart folks and understand the replication protocol much better than I do. MongoDB's also published a writeup (which is cited a few times in the Jepsen report!) talking about the impact of stronger safety settings and why they choose weak defaults: http://www.vldb.org/pvldb/vol12/p2071-schultz.pdf http://www.vldb.org/pvldb/vol12/p2071-schultz.pdf
- goatinaboat 6y ago
  In general, MongoDB’s defaults fall into two categories. The first could possibly be justified as making it easy for inexperienced devs to get started, but it means that people rely on those defaults and then try to promote to production, and unless there is an experienced traditional DBA with the power to veto it, it will go ahead. This is how they “backdoor” their way into companies. The second category is whatever will look good on a benchmark, regardless of any corners cut. Compare and contrast with the highly ethical Postgres team, who encourage good practices from the start and who get a feature right first before worrying about performance. That may harm their adoption in the short term but over the long term, that's why they're the gold standard. And with their JSONB datatype they have a better MongoDB than MongoDB anyway! And have a million other features besides!
  - logicchains 6y ago
    >And have a million other features besides! Yeah, but in spite of that their performance still sucks compared to writing directly to /dev/null, and that's where Mongo steals their thunder.
  - threeseed 6y ago
    > Compare and contrast with the highly ethical Postgres team You do know that PostgreSQL had issues with not fsyncing data as well ? It's technology. Bugs will be made. Design decisions will be wrong. I think it's really disappointing and inappropriate to be labelling MongoDB engineers as unethical for simply having incorrect defaults. Which in their history they often change after they are made aware of them.
    - goatinaboat 6y ago
      You do know that PostgreSQL had issues with not fsyncing data as well ? See, you can name just one Postgres bug, and they held their hands up to it straight away. Whereas the MongoDB "bugs" are countless and by sheer coincidence, they mostly skew to improving performance in benchmarks and demos. That's a pattern.
nevi-me 6y ago
Friendly question: did you update anything on the findings since https://news.ycombinator.com/item?id=23191439 https://news.ycombinator.com/item?id=23191439 ?
- aphyr 6y ago
  Nope! Something weird happened to that post; it got a lot of upvotes and some comments, but never made it to frontpage. After the InfoQ article took off yesterday, an HN mod got in touch and asked if I'd like to resubmit it.
tester756 6y ago
Why is this being here everyday for last 3 days?
aphyr 6y ago
Hi folks! Author of the report here. If anyone has questions about detecting transactional anomalies, what those anomalies are in the first place, snapshot isolation, etc., I'm happy to answer as best I can.
- eternalban 6y ago
  "3.4 Duplicate Effects" This section seems to be the most worrying results in your report, Kyle, with no work around. Did I read that correctly?
  - aphyr 6y ago
    Yeah, there's no workaround that I can find for 3.4 (duplicate effects), 3.5 (read skew), 3.6 (cyclic information flow), or 3.7 (read own future writes). I've arranged those in "increasingly worrying order"--duplicating writes doesn't feel as bad as allowing transactions to mutually observe each other's effects, for example. The fact that you can't even rely on a single transactions' operations taking place (or, more precisely, appearing to take place) in the order they're written is especially worrying. All of these behaviors occurred with read and write concerns set to snapshot/majority. That's not to say that workarounds don't exist, just that I didn't find any in the documentation or by twiddling config flags in the ~2 weeks I was working on this report. :)
- danpalmer 6y ago
  Not a question necessarily about the technical side, but I'm interested in your opinion as to the root cause – is it desire to achieve certain results for marketing purposes, lack of understanding/training in the team about distributed systems, just bugs and a lack of testing...? Alternatively does most of this come down to one specific technical choice, and why might they have made that choice? Very happy for (informed) speculation here, I recognise we'll probably never know for certain, but I'm interested to avoid making similar mistakes myself.
  - aphyr 6y ago
    There's a few things at play here. One is talking only about the positive results from the previous Jepsen analysis, while not discussing the negative ones. Vendors often try to represent findings in the most positive light, but this was a particularly extreme case. Not discussing default behavior is a significant oversight, and it's especially important given ~80% of people run with default write concern, and 99% run with default read concern. The middle part of the report talks about unexpected but (almost all) documented behavior around read and write concern for transactions. I don't want to conjecture too much about motivations here, but based on my professional experience with a few dozen databases, and surveys of colleagues, I termed it "surprising". The fact that there's explicit documentation for what I'd consider Counterintuitive API Design suggests that this is something MongoDB engineers considered, and possibly debated, internally. The final part of the report talks about what I'm pretty sure are bugs. I'm strongly suspicious of the retry mechanism: it's possible that an idempotency token doesn't exist, isn't properly used, or that MongoDB's client or server layers are improperly interpreting an indeterminate failure as a determinate one. It seems possible that all 4 phenomena we observed stem from the retry mechanism, but as discussed in the report, it's not entirely clear that's the case.
deleted 6y ago
[deleted]
mtrycz2 6y ago
aphyr, you are of great inspiration as an engineer and as a human. Your attitude of "a tool I need doesn't exists, so I'll just go ahead and create it" blew my mind and changed me for the better. I'm dedicating my next test framework to you. Thank you for everything.
- aphyr 6y ago
  Aw shucks, thank you! <3
- afarrell 6y ago
  > I'm dedicating my next test framework to you. As an engineer for whom automated testing tools are crucial to my mental health, let me know if you want a UX tester or just someone to provide feedback on the documentation.
sam1r 6y ago
Extremely well written! I learned a lot. I wonder if someone can type up a well-manicured post-Morten of the recent triple byte incident?
bithavoc 6y ago
> Clients observed a monotonically growing list of elements until [1 2 3 5 4 6 7], at which point the list reset to [], and started afresh with [8]. This could be an example of MongoDB rollbacks, which is a fancy way of saying “data loss”. I hope they learned the lesson, don't fuck with aphyr.
- amenod 6y ago
  That's... not the lesson they need to learn. Databases are app foundations. Make sure you do them right and don't overpromise.
  - baq 6y ago
    I agree but maybe it’s the only lesson they are able to understand at this time. Their attitude was asking for somebody to call them, which aphyr is maybe the best positioned to do. I’d love to read a roasting like that authored by Leslie Lamport for a different perspective but aphyr’s works absolutely stand on their own. Any ideas how to get Jepsen and TLA to work together? :)
egeozcan 6y ago
The general mood I observed about MongoDB was that it used to be inconsistent and unreliable but they fixed most, if not all of those problems and they now have a stable product but bad word of mouth among developers. Personally, I've treated it as "legacy" and migrated everything that I had to touch since 2013 [0], and luckily (just read the article so hindsight 20/20 -- transaction running twice and seeing its own updates? holy...) never gave it another try. [0]: https://news.ycombinator.com/item?id=6801970 https://news.ycombinator.com/item?id=6801970 (BTW: no, my dream of simple migration never materialized, but exporting and dumping data to Postgres JSONB columns and rewriting queries turned out to be neither buggy nor hard).
- cyphar 6y ago
  > MongoDB was that it used to be inconsistent and unreliable but they fixed most, if not all of those problems and they now have a stable product but bad word of mouth among developers. This report is 9 days old, and tests the latest stable release of MongoDB. The problems it discusses are present on modern MongoDB.
  - egeozcan 6y ago
    If it wasn't clear, I said "mood" (what you conveniently ignored), referring to chit-chat I heard recently, and was underlining the fact how wrong it has been. I totally understand what the report says and know what version it tests.
    - cyphar 6y ago
      In my defense, it wasn't clear that's what you were saying in your original comment. "Mood" has become a filler word at this point -- hence why I omitted it from the quote -- and can mean anything from the traditional meaning of "mood in the room" to "incredibly relatable/factual statement". How I originally understood your comment was that you were saying that you felt that most of the issues are in the past, but you still decided to migrate away from it.
      - 6y ago
koishikomeiji 6y ago
Fuck me gently with a chainsaw
lllr_finger 6y ago
Mongo has been related to "perpetual irritation" up to "major production issue" at all three of my last companies. For as easy as it is to use jsonb in Postgres, or Redis, or RocksDB/SQLite, or whatever else depending on your use case - I can't find any reason to advocate its use these days. In my anecdotal experience, the success stories never happen, and nearly developer I know has an unpleasant experience they can share. Big thanks to aphyr and the Jepsen suite (and unrelated blog posts like Hexing the Interview) for inspiring me to do thorough engineering.
- StavrosK 6y ago
  I find that using JSON for things you don't need to query/validate (like big blobs you just want to store) and breaking the rest out to columns works well enough. Plus, you can always migrate the data out to a field anyway.
  - emerongi 6y ago
    Postgres 12 has generated columns, so you can throw your data in a jsonb column and have Postgres pull data out of it into separate columns for indexing for example.
    - magnushiie 6y ago
      Generated columns are not necessary for indexing in Postgres, you can create an index on any expression based on the record (supported by many versions now).
- mtrycz2 6y ago
  > I can't find any reason to advocate its use these days. Don't you know? It's web-scale.
  - rmdashrfstar 6y ago
    If I was a moderator on HN, I would instantly ban commenters who continue to make these asinine posts. Is this Reddit, or is HN striving to be Reddit?
judofyr 6y ago
This is not directly related to this report or Jepsen, but since you're here I've got to ask: Aphyr, are there any recent papers/research in the realm of distributed databases which you're excited about?
- aphyr 6y ago
  Calvin and CRDTs aren't new, but I still think they're dramatically underappreciated! Heidi Howard's recent work on generalizing Paxos quorums is super intriguing, and from some discussion with her, I think there are open possibilities in making leaderless single-round-trip consensus systems for log-oriented FSMs, which is what pretty much everyone WANTS. I'm also excited about my own research with Elle, but we're still working on getting that through peer review, haha. ;-)
  - thramp 6y ago
    > I think there are open possibilities in making leaderless single-round-trip consensus systems for log-oriented FSMs, which is what pretty much everyone WANTS. Woah, that's wild. Are there any pre-prints/papers/talks that you can link to on this subject? I'd _love_ to read this. > I'm also excited about my own research with Elle, but we're still working on getting that through peer review, haha. ;-) I read over bits of Elle; the documentation in it is absolutely top-notch. You and Peter Alvaro knocked it out of the park!
    - aphyr 6y ago
      I think there are open possibilities in making leaderless single-round-trip consensus systems for log-oriented FSMs, which is what pretty much everyone WANTS. This is based on her presentation and some dinner conversation at HPTS 2019, so I don't know if there's actually a paper I can point to. The gist of is that Paxos normally involves an arbitration phase where there are conflicting proposals, which adds a second pair of message delays. But if you relax the consensus problem to agreement on a set of proposals, rather than a single proposal, you don't need the arbitration phase. Instead of "who won", it becomes "everyone wins". Then you can impose an order on that set via, say, sorting, and iterate to get a replicated log. I read over bits of Elle; the documentation in it is absolutely top-notch. You and Peter Alvaro knocked it out of the park! Thank you! Could I... hang on, just let me grab reviewer #1 quickly, I'd like them to hear this. ;-)
lmilcin 6y ago
I am tech lead for a project that revolves around multiple terabytes of trading data for one of top ten largest banks in the world. My team has three, 3-node, 3TB per node MongoDB clusters where we keep huge amount of documents (mostly immutable 1kB to 10kB in size). Majority write/read concern is exactly so that you don't loose data and don't observe stuff that is going to be rolled back. It is important to understand this fact when you evaluate MongoDB for your solution. That it comes with additional downsides is hardly a surprise, otherwise there would be no reason to specify anything else than majority. You just can't test lower levels of guarantees and then complain you did not get what higher levels of guarantees were designed to provide. It is also obvious, when you use majority concern, that some of the nodes may accept the write but then have to roll back when the majority cannot acknowledge the write. It is obvious this may cause some of the writes to fail that would succeed should the write concern be configured to not require majority acknowledgment. The article simply misses the mark by trying to create sensation where there is none to be found. The MongoDB documentation explains the architecture and guarantees provided by MongoDB enough so that you should be able to understand various read/write concerns and that anything below majority does not guarantee much. This is a tradeoff which you are allowed to make provided you understand the consequences.
- aphyr 6y ago
  You just can't test lower levels of guarantees and then complain you did not get what higher levels of guarantees were designed to provide. Gently, may I suggest that you read the report, or at least the abstract? This is addressed in the second sentence. :-)
  - lmilcin 6y ago
    To quote from the report: "Moreover, the snapshot read concern did not guarantee snapshot unless paired with write concern majority—even for read-only transactions." Of course, it doesn't work when you don't pair it with majority read/write concern. You can't expect to get a snapshot of data that wasn't yet acknowledged by majority of the cluster. As to the quote you probably are referring to: "Jepsen evaluated MongoDB version 4.2.6, and found that even at the strongest levels of read and write concern, it failed to preserve snapshot isolation." I did not find any proof of this in the rest of the report. It seems this is mostly complaint of what happens when you mix different read and write concerns. I would also suggest to think a little bit on the concept of snapshot in the context of distributed system. It is not possible to have the same kind of snapshot that you would get with a single-node application with the architecture of MongoDB. MongoDB is a distributed system where you will get different results depending on which node you are asking. The only way you could get close to having a global snapshot is if all nodes agreed on a single truth (for example single log file, block chain, etc.) which would preclude read/write with concern level less than majority.
inglor 6y ago
Without going into details die to NDAs, the experience in the OP matches the ones of several fortune 500 companies I had gigs with.
inglor 6y ago
I also want to point their Node.js transactions API is wrong and looks like they have no idea how promises or async code work in JS. In mongo, you have a `withTransaction(fn)` helper that passes a session parameter. Mongo can call this function mutliple times with the same session object. This means that if you have an async function with reference to a session and a transaction gets retried - you very often get "part of one attempt + some parts of another" committed. We had to write a ton of logic around their poor implementation and I was shocked to see the code underneath. It was just such a stark contrast to products that I worked with before that generally "just worked" like postgres, elasticsearch or redis. Even tools people joke about a lot like mysql never gave me this sort of data corruption. Edit: I was kind of angry when writing this so I didn't provide a source and I'm a bit surprised this go so many upvotes without a source (I guess this community is more trusting than I assumed :] ). Anyway for good measure and to behave the way I'd like others to when making such accusations here is where they pass the same session object to the transacton https://github.com/mongodb/node-mongodb-native/blob/e5b762c6d53afa967f24c26a1d1b6c921757c9c9/lib/sessions.js#L376 https://github.com/mongodb/node-mongodb-native/blob/e5b762c6... (follow from withTransaction in that file) - I can add examples of code easily introducing the above mentioned bug if people are interested.
- inglor 6y ago
  If you work for Mongo and are reading this. Please just fix it. I don't need to win and I don't care about being "right". I just don't want to be called to the office on a weekend anymore for this sort of BS. Production incidents with MongoDB last year: 15 Production instances with redis, elasticsearch and mysql combined last year: 2 (and with much less severity) Edit: just to add: I didn't pick Mongo, I was just the engineer called to clean that mess. I created enough of my own messes to not resent the person who made that shot for it. We are constantly on the verge of rewriting the MongoDB stuff since a database that small (~250GB) should really not have these many issues (In previous workplaces I ran ~10TB PostgreSQL deployments with much more complicated schemas and queries with far fewer issues). It's also expensive and support at Mongo Atlas hasn't been great (we should probably self host but I am not used to small databases being so problematic)
  - 6y ago
junon 6y ago
I wanted to incorporate MongoDB into a C++ server at one point. Their C/C++ client is literally unusable. I went to look into writing my own that actually worked and their network protocols are almost impossible to understand. BSON is a wreck and basically the whole thing discouraged me from ever trying to interact with that project again.
loeg 6y ago
Aphyr is such a competent professional. What a relatively thorough and polite response to Mongo's inaccurate claims. "We also wish to thank MongoDB’s Maxime Beugnet for inspiration." is a nice touch.
jtdev 6y ago
Now do DynamoDB.
- aphyr 6y ago
  I'd like to, but I don't have any way to do fault injection on a system someone else owns. :(
  - jtdev 6y ago
    Would love to see AWS agree to facilitating this. Appreciate your work very much!
  - petrikapu 6y ago
    They have downloadable version of it https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
    - TheDong 6y ago
      The downloadable version of dynamodb is only intended for testing and is not a distributed system by any definition, nor does it behavior match the production system exactly. There's no reason for Jepson to be applied to a single-node in-memory kv store.
rmdashrfstar 6y ago
The main argument for using a documented-oriented database: https://martinfowler.com/bliki/AggregateOrientedDatabase.html https://martinfowler.com/bliki/AggregateOrientedDatabase.htm...
zzzeek 6y ago
How many more years do we have to keep evaluating, studying, and reading about MongoDB's ongoing failures? It would appear this product has been a great burden on the community for many years.
- aphyr 6y ago
  I like to keep in mind that MongoDB's existing feature set is maturing--occasional regressions may happen, but by and large they're making progress. The problems in this analysis were in a transaction system that's only been around for a couple years, so it's had less time to have rough edges sanded off.
  - zzzeek 6y ago
    there are so many great databases out there. There's no need for one that has been mediocre for years and continues to make false claims. This is an issue of years of super aggressive marketing of an inferior product making it hard on engineers.
    - trashcan 6y ago
      I think if you compared it to other databases that are designed to scale horizontally like Cassandra and DynamoDB, you might have a more favorable opinion. IMHO, most products at this scale are terrible in different ways, because it is a difficult problem to solve generally. I have been responsible for <100 clustered Cassandra instances, and <500 clustered MongoDB instances, and I would choose the latter every time.
      - runT1ME 6y ago
        Can mongo really support the kind of large scale ETL or time series use cases Cassandra can?
  - bborud 6y ago
    It has been a decade since MongoDB was initially released. It still isn't a database in any meaningful sense of the word. Please don't downplay the amount of trouble one can get into by assuming it is a database.
fastball 6y ago
At this point I think we might be going a bit overboard with title changes. Now that it's just "MongoDB 4.2.6", the title makes me think that this is a release announcement, not an analysis of the software. The first title (that specifically referenced a finding of the analysis) was best, imo. Mildly opinionated or whatever, but at least it quickly communicated the gist of the post. On the other hand: "Jepsen: MongoDB 4.2.6" – not super helpful if you're not already familiar with the Jepsen body of work. "MongoDB 4.2.6" – as stated above, sounds like a release announcement. If you want a suggestion, maybe something like "Jepsen evaluation of MongoDB 4.2.6"? Not overly specific (/ negative) like the first title, but at least provides some slight amount of context. @dang
- Ecco 6y ago
  It's the article's title...
  - Fiveplus 6y ago
    No context doesn't help.
  - petepete 6y ago
    And taken out of context it makes little sense.
  - fiddlerwoaroof 6y ago
    A generic “Mongo 4.2.6” title doesn’t help me decide whether to click on the link (especially with how light the domain is). I thought it was a release announcement and only clicked through to the comments because of yesterday’s discusssion.
    - dang 6y ago
      An HN title needs to be read along with the site name to the right of it.
      - fiddlerwoaroof 6y ago
bbulkow 6y ago
mongodb's business model, forever, has been to get developers to write code, be damned the fact that you can't support it reliably on a cloudy day.
deleted 6y ago
[deleted]
chousuke 6y ago
This article reinforces my stance that bad defaults are a bug. Defaults should be set up with the least number of pitfalls and safety tradeoffs possible so that the system is as robust as it can be for the majority of its users, since the vast majority of them aren't going to change the defaults. Sometimes you end up with bad defaults simply by accident but I feel like for MongoDB the morally correct choice would be to own up to past mistakes and change the defaults rather than maintain a dangerous status quo for "backwards compatibility", even if you end up looking worse in benchmarks as a result.
- aphyr 6y ago
  I think this is a good way to look at things, and there are vendors who do this! VoltDB, for instance, changed their defaults to be strict serializable even though it imposed a performance hit, following their Jepsen analysis. https://www.voltdb.com/blog/2016/07/voltdb-6-4-passes-official-jepsen-testing/ https://www.voltdb.com/blog/2016/07/voltdb-6-4-passes-offici...
azernik 6y ago
Ouch. This is what you get when you order up a third-party review and then misrepresent it in advertising.
- taywrobel 6y ago
  I’m still waiting for Jepsen to put Confluent’s “Kafka provides exactly once delivery semantics” claim to the test. Since they’re claiming something provably false, it’d be nice to have some empirical evidence as such.
  - aphyr 6y ago
    I'm not convinced it is false--IIRC their claim is specifically w.r.t other Kafka side effects, and those they can control.
  - staticassertion 6y ago
    I thought they only guaranteed exactly once processing ? That could just be idempotency.
dang 6y ago
All: there was a big thread about this yesterday (https://news.ycombinator.com/item?id=23285249 https://news.ycombinator.com/item?id=23285249) but because it didn't focus on the technical content, and because there were glitches with a previous submission of this report (described at https://news.ycombinator.com/item?id=23288120 https://news.ycombinator.com/item?id=23288120 and https://news.ycombinator.com/item?id=23287763 https://news.ycombinator.com/item?id=23287763 if anyone cares), we invited aphyr to repost this. Normally we downweight follow-up posts that have such close overlap with a recent discussion (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20follow-up&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...), so the exception is probably worth explaining.
depr 6y ago
>Sometimes, Programs That Use Transactions… Are Worse I understood that reference
pier25 6y ago
> Normally we downweight follow-up posts So you manually moderate the content?
- VonGuard 6y ago
  I mean, this was kind of an exception case, where there is a big old technical war of words back and forth. Almost a "He said She said" except here, He is an absolute expert, and She is just some marketing dorks at Mongo. I, for one, welcome this by-hand moderation because it keeps this issue alive, and allows Kyle to keep the discussion going. As I commented in a previous post, Kyle is the Chef Ramsey of database testing, and here, he's in a position where some idiot has just served him an undercooked hamburger. Bits will fly, marketing people will be flayed alive, and Kyle will be the only one left standing at the end. Without this by-hand moderation, we'd be missing out on the second act of this intense thriller!
  - pier25 6y ago
    I'm totally ok with the moderation/curation/whatever!
- deleted 6y ago
  [deleted]
- DoreenMichele 6y ago
  They use a combination of algorithms and human intervention, to generally good effect. No clue if this "downweighting" in this case is an algorithm or a manual thing. I would assume algorithm for the downweighting and human intervention for reversing it, but that's sort of a guess or inference.
- baq 6y ago
  of course they do? given the quality of discussion here it's a hard requirement to preserve high snr.
- dang 6y ago
  Oh yes. HN has always been moderated/curated/whatever term you prefer. Many past explanations can be found through these links: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20downweight&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20penalt&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20moderate&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20curate&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... (I've detached this subthread from https://news.ycombinator.com/item?id=23294048 https://news.ycombinator.com/item?id=23294048 to prevent the top comment from being too distracting.)
Laszlotejfel86 6y ago
SSID: Tejfel Lazalo Protocol: Wi-Fi 5 (802.11ac) Security type: WPA2-Personal Network band: 5 GHz Network channel: 157 Link-local IPv6 address: fe80::1d0:5810:36ab:219d%8 IPv4 address: 10.0.0.148 IPv4 DNS servers: 64.71.255.204 64.71.255.198 Manufacturer: Intel Corporation Description: Intel(R) Wireless-AC 9462 Driver version: 20.50.0.5 Physical address (MAC): 7C-2A-31-7B-02-46

604 points by aphyr 6y ago

I suppose there are reasons why the defaults are the way they are. Can anyone comment on the implications, performance or otherwise, of bumping up the read/write concerns?

aphyr 6y ago
Latency is a big one--you've got to wait an extra round-trip for secondaries to acknowledge primary writes, and primaries (assuming you don't have reliable clocks) need to check in with secondaries to confirm they have the most recent picture of things if you want to do a linearizable read. Snapshot isolated reads shouldn't require that, at least in theory--it's legal to read state from the past under SI, so there's no need to establish present leadership. That's why I'm surprised that MongoDB requires snapshot reads to go through write concern majority--it doesn't seem like it'd be necessary. Might have something to do with sharding--maybe establishing a consistent cut across shards requires a round of coordination. Even then I feel like that's a cost you should be able to pay only at write time, making reads fast, but... apparently not! I'm sure the MongoDB engineers who designed this system have good reasons; they're smart folks and understand the replication protocol much better than I do. MongoDB's also published a writeup (which is cited a few times in the Jepsen report!) talking about the impact of stronger safety settings and why they choose weak defaults: http://www.vldb.org/pvldb/vol12/p2071-schultz.pdf http://www.vldb.org/pvldb/vol12/p2071-schultz.pdf
goatinaboat 6y ago
In general, MongoDB’s defaults fall into two categories. The first could possibly be justified as making it easy for inexperienced devs to get started, but it means that people rely on those defaults and then try to promote to production, and unless there is an experienced traditional DBA with the power to veto it, it will go ahead. This is how they “backdoor” their way into companies. The second category is whatever will look good on a benchmark, regardless of any corners cut. Compare and contrast with the highly ethical Postgres team, who encourage good practices from the start and who get a feature right first before worrying about performance. That may harm their adoption in the short term but over the long term, that's why they're the gold standard. And with their JSONB datatype they have a better MongoDB than MongoDB anyway! And have a million other features besides!
- logicchains 6y ago
  >And have a million other features besides! Yeah, but in spite of that their performance still sucks compared to writing directly to /dev/null, and that's where Mongo steals their thunder.
- threeseed 6y ago
  > Compare and contrast with the highly ethical Postgres team You do know that PostgreSQL had issues with not fsyncing data as well ? It's technology. Bugs will be made. Design decisions will be wrong. I think it's really disappointing and inappropriate to be labelling MongoDB engineers as unethical for simply having incorrect defaults. Which in their history they often change after they are made aware of them.
  - goatinaboat 6y ago
    You do know that PostgreSQL had issues with not fsyncing data as well ? See, you can name just one Postgres bug, and they held their hands up to it straight away. Whereas the MongoDB "bugs" are countless and by sheer coincidence, they mostly skew to improving performance in benchmarks and demos. That's a pattern.

nevi-me 6y ago

Friendly question: did you update anything on the findings since https://news.ycombinator.com/item?id=23191439 https://news.ycombinator.com/item?id=23191439 ?

aphyr 6y ago
Nope! Something weird happened to that post; it got a lot of upvotes and some comments, but never made it to frontpage. After the InfoQ article took off yesterday, an HN mod got in touch and asked if I'd like to resubmit it.

tester756 6y ago

Why is this being here everyday for last 3 days?

aphyr 6y ago

Hi folks! Author of the report here. If anyone has questions about detecting transactional anomalies, what those anomalies are in the first place, snapshot isolation, etc., I'm happy to answer as best I can.

eternalban 6y ago
"3.4 Duplicate Effects" This section seems to be the most worrying results in your report, Kyle, with no work around. Did I read that correctly?
- aphyr 6y ago
  Yeah, there's no workaround that I can find for 3.4 (duplicate effects), 3.5 (read skew), 3.6 (cyclic information flow), or 3.7 (read own future writes). I've arranged those in "increasingly worrying order"--duplicating writes doesn't feel as bad as allowing transactions to mutually observe each other's effects, for example. The fact that you can't even rely on a single transactions' operations taking place (or, more precisely, appearing to take place) in the order they're written is especially worrying. All of these behaviors occurred with read and write concerns set to snapshot/majority. That's not to say that workarounds don't exist, just that I didn't find any in the documentation or by twiddling config flags in the ~2 weeks I was working on this report. :)
danpalmer 6y ago
Not a question necessarily about the technical side, but I'm interested in your opinion as to the root cause – is it desire to achieve certain results for marketing purposes, lack of understanding/training in the team about distributed systems, just bugs and a lack of testing...? Alternatively does most of this come down to one specific technical choice, and why might they have made that choice? Very happy for (informed) speculation here, I recognise we'll probably never know for certain, but I'm interested to avoid making similar mistakes myself.
- aphyr 6y ago
  There's a few things at play here. One is talking only about the positive results from the previous Jepsen analysis, while not discussing the negative ones. Vendors often try to represent findings in the most positive light, but this was a particularly extreme case. Not discussing default behavior is a significant oversight, and it's especially important given ~80% of people run with default write concern, and 99% run with default read concern. The middle part of the report talks about unexpected but (almost all) documented behavior around read and write concern for transactions. I don't want to conjecture too much about motivations here, but based on my professional experience with a few dozen databases, and surveys of colleagues, I termed it "surprising". The fact that there's explicit documentation for what I'd consider Counterintuitive API Design suggests that this is something MongoDB engineers considered, and possibly debated, internally. The final part of the report talks about what I'm pretty sure are bugs. I'm strongly suspicious of the retry mechanism: it's possible that an idempotency token doesn't exist, isn't properly used, or that MongoDB's client or server layers are improperly interpreting an indeterminate failure as a determinate one. It seems possible that all 4 phenomena we observed stem from the retry mechanism, but as discussed in the report, it's not entirely clear that's the case.

deleted 6y ago

[deleted]

mtrycz2 6y ago

aphyr, you are of great inspiration as an engineer and as a human. Your attitude of "a tool I need doesn't exists, so I'll just go ahead and create it" blew my mind and changed me for the better. I'm dedicating my next test framework to you. Thank you for everything.

aphyr 6y ago
Aw shucks, thank you! <3
afarrell 6y ago
> I'm dedicating my next test framework to you. As an engineer for whom automated testing tools are crucial to my mental health, let me know if you want a UX tester or just someone to provide feedback on the documentation.

sam1r 6y ago

Extremely well written! I learned a lot. I wonder if someone can type up a well-manicured post-Morten of the recent triple byte incident?

bithavoc 6y ago

> Clients observed a monotonically growing list of elements until [1 2 3 5 4 6 7], at which point the list reset to [], and started afresh with [8]. This could be an example of MongoDB rollbacks, which is a fancy way of saying “data loss”. I hope they learned the lesson, don't fuck with aphyr.

amenod 6y ago
That's... not the lesson they need to learn. Databases are app foundations. Make sure you do them right and don't overpromise.
- baq 6y ago
  I agree but maybe it’s the only lesson they are able to understand at this time. Their attitude was asking for somebody to call them, which aphyr is maybe the best positioned to do. I’d love to read a roasting like that authored by Leslie Lamport for a different perspective but aphyr’s works absolutely stand on their own. Any ideas how to get Jepsen and TLA to work together? :)

egeozcan 6y ago

The general mood I observed about MongoDB was that it used to be inconsistent and unreliable but they fixed most, if not all of those problems and they now have a stable product but bad word of mouth among developers. Personally, I've treated it as "legacy" and migrated everything that I had to touch since 2013 [0], and luckily (just read the article so hindsight 20/20 -- transaction running twice and seeing its own updates? holy...) never gave it another try. [0]: https://news.ycombinator.com/item?id=6801970 https://news.ycombinator.com/item?id=6801970 (BTW: no, my dream of simple migration never materialized, but exporting and dumping data to Postgres JSONB columns and rewriting queries turned out to be neither buggy nor hard).

cyphar 6y ago
> MongoDB was that it used to be inconsistent and unreliable but they fixed most, if not all of those problems and they now have a stable product but bad word of mouth among developers. This report is 9 days old, and tests the latest stable release of MongoDB. The problems it discusses are present on modern MongoDB.
- egeozcan 6y ago
  If it wasn't clear, I said "mood" (what you conveniently ignored), referring to chit-chat I heard recently, and was underlining the fact how wrong it has been. I totally understand what the report says and know what version it tests.
  - cyphar 6y ago
    In my defense, it wasn't clear that's what you were saying in your original comment. "Mood" has become a filler word at this point -- hence why I omitted it from the quote -- and can mean anything from the traditional meaning of "mood in the room" to "incredibly relatable/factual statement". How I originally understood your comment was that you were saying that you felt that most of the issues are in the past, but you still decided to migrate away from it.
    - 6y ago

koishikomeiji 6y ago

Fuck me gently with a chainsaw

lllr_finger 6y ago

Mongo has been related to "perpetual irritation" up to "major production issue" at all three of my last companies. For as easy as it is to use jsonb in Postgres, or Redis, or RocksDB/SQLite, or whatever else depending on your use case - I can't find any reason to advocate its use these days. In my anecdotal experience, the success stories never happen, and nearly developer I know has an unpleasant experience they can share. Big thanks to aphyr and the Jepsen suite (and unrelated blog posts like Hexing the Interview) for inspiring me to do thorough engineering.

StavrosK 6y ago
I find that using JSON for things you don't need to query/validate (like big blobs you just want to store) and breaking the rest out to columns works well enough. Plus, you can always migrate the data out to a field anyway.
- emerongi 6y ago
  Postgres 12 has generated columns, so you can throw your data in a jsonb column and have Postgres pull data out of it into separate columns for indexing for example.
  - magnushiie 6y ago
    Generated columns are not necessary for indexing in Postgres, you can create an index on any expression based on the record (supported by many versions now).
mtrycz2 6y ago
> I can't find any reason to advocate its use these days. Don't you know? It's web-scale.
- rmdashrfstar 6y ago
  If I was a moderator on HN, I would instantly ban commenters who continue to make these asinine posts. Is this Reddit, or is HN striving to be Reddit?

judofyr 6y ago

This is not directly related to this report or Jepsen, but since you're here I've got to ask: Aphyr, are there any recent papers/research in the realm of distributed databases which you're excited about?

aphyr 6y ago
Calvin and CRDTs aren't new, but I still think they're dramatically underappreciated! Heidi Howard's recent work on generalizing Paxos quorums is super intriguing, and from some discussion with her, I think there are open possibilities in making leaderless single-round-trip consensus systems for log-oriented FSMs, which is what pretty much everyone WANTS. I'm also excited about my own research with Elle, but we're still working on getting that through peer review, haha. ;-)
- thramp 6y ago
  > I think there are open possibilities in making leaderless single-round-trip consensus systems for log-oriented FSMs, which is what pretty much everyone WANTS. Woah, that's wild. Are there any pre-prints/papers/talks that you can link to on this subject? I'd _love_ to read this. > I'm also excited about my own research with Elle, but we're still working on getting that through peer review, haha. ;-) I read over bits of Elle; the documentation in it is absolutely top-notch. You and Peter Alvaro knocked it out of the park!
  - aphyr 6y ago
    I think there are open possibilities in making leaderless single-round-trip consensus systems for log-oriented FSMs, which is what pretty much everyone WANTS. This is based on her presentation and some dinner conversation at HPTS 2019, so I don't know if there's actually a paper I can point to. The gist of is that Paxos normally involves an arbitration phase where there are conflicting proposals, which adds a second pair of message delays. But if you relax the consensus problem to agreement on a set of proposals, rather than a single proposal, you don't need the arbitration phase. Instead of "who won", it becomes "everyone wins". Then you can impose an order on that set via, say, sorting, and iterate to get a replicated log. I read over bits of Elle; the documentation in it is absolutely top-notch. You and Peter Alvaro knocked it out of the park! Thank you! Could I... hang on, just let me grab reviewer #1 quickly, I'd like them to hear this. ;-)

lmilcin 6y ago

I am tech lead for a project that revolves around multiple terabytes of trading data for one of top ten largest banks in the world. My team has three, 3-node, 3TB per node MongoDB clusters where we keep huge amount of documents (mostly immutable 1kB to 10kB in size). Majority write/read concern is exactly so that you don't loose data and don't observe stuff that is going to be rolled back. It is important to understand this fact when you evaluate MongoDB for your solution. That it comes with additional downsides is hardly a surprise, otherwise there would be no reason to specify anything else than majority. You just can't test lower levels of guarantees and then complain you did not get what higher levels of guarantees were designed to provide. It is also obvious, when you use majority concern, that some of the nodes may accept the write but then have to roll back when the majority cannot acknowledge the write. It is obvious this may cause some of the writes to fail that would succeed should the write concern be configured to not require majority acknowledgment. The article simply misses the mark by trying to create sensation where there is none to be found. The MongoDB documentation explains the architecture and guarantees provided by MongoDB enough so that you should be able to understand various read/write concerns and that anything below majority does not guarantee much. This is a tradeoff which you are allowed to make provided you understand the consequences.

aphyr 6y ago
You just can't test lower levels of guarantees and then complain you did not get what higher levels of guarantees were designed to provide. Gently, may I suggest that you read the report, or at least the abstract? This is addressed in the second sentence. :-)
- lmilcin 6y ago
  To quote from the report: "Moreover, the snapshot read concern did not guarantee snapshot unless paired with write concern majority—even for read-only transactions." Of course, it doesn't work when you don't pair it with majority read/write concern. You can't expect to get a snapshot of data that wasn't yet acknowledged by majority of the cluster. As to the quote you probably are referring to: "Jepsen evaluated MongoDB version 4.2.6, and found that even at the strongest levels of read and write concern, it failed to preserve snapshot isolation." I did not find any proof of this in the rest of the report. It seems this is mostly complaint of what happens when you mix different read and write concerns. I would also suggest to think a little bit on the concept of snapshot in the context of distributed system. It is not possible to have the same kind of snapshot that you would get with a single-node application with the architecture of MongoDB. MongoDB is a distributed system where you will get different results depending on which node you are asking. The only way you could get close to having a global snapshot is if all nodes agreed on a single truth (for example single log file, block chain, etc.) which would preclude read/write with concern level less than majority.

inglor 6y ago

Without going into details die to NDAs, the experience in the OP matches the ones of several fortune 500 companies I had gigs with.

inglor 6y ago

I also want to point their Node.js transactions API is wrong and looks like they have no idea how promises or async code work in JS. In mongo, you have a `withTransaction(fn)` helper that passes a session parameter. Mongo can call this function mutliple times with the same session object. This means that if you have an async function with reference to a session and a transaction gets retried - you very often get "part of one attempt + some parts of another" committed. We had to write a ton of logic around their poor implementation and I was shocked to see the code underneath. It was just such a stark contrast to products that I worked with before that generally "just worked" like postgres, elasticsearch or redis. Even tools people joke about a lot like mysql never gave me this sort of data corruption. Edit: I was kind of angry when writing this so I didn't provide a source and I'm a bit surprised this go so many upvotes without a source (I guess this community is more trusting than I assumed :] ). Anyway for good measure and to behave the way I'd like others to when making such accusations here is where they pass the same session object to the transacton https://github.com/mongodb/node-mongodb-native/blob/e5b762c6d53afa967f24c26a1d1b6c921757c9c9/lib/sessions.js#L376 https://github.com/mongodb/node-mongodb-native/blob/e5b762c6... (follow from withTransaction in that file) - I can add examples of code easily introducing the above mentioned bug if people are interested.

inglor 6y ago
If you work for Mongo and are reading this. Please just fix it. I don't need to win and I don't care about being "right". I just don't want to be called to the office on a weekend anymore for this sort of BS. Production incidents with MongoDB last year: 15 Production instances with redis, elasticsearch and mysql combined last year: 2 (and with much less severity) Edit: just to add: I didn't pick Mongo, I was just the engineer called to clean that mess. I created enough of my own messes to not resent the person who made that shot for it. We are constantly on the verge of rewriting the MongoDB stuff since a database that small (~250GB) should really not have these many issues (In previous workplaces I ran ~10TB PostgreSQL deployments with much more complicated schemas and queries with far fewer issues). It's also expensive and support at Mongo Atlas hasn't been great (we should probably self host but I am not used to small databases being so problematic)
- 6y ago

junon 6y ago

I wanted to incorporate MongoDB into a C++ server at one point. Their C/C++ client is literally unusable. I went to look into writing my own that actually worked and their network protocols are almost impossible to understand. BSON is a wreck and basically the whole thing discouraged me from ever trying to interact with that project again.

loeg 6y ago

Aphyr is such a competent professional. What a relatively thorough and polite response to Mongo's inaccurate claims. "We also wish to thank MongoDB’s Maxime Beugnet for inspiration." is a nice touch.

jtdev 6y ago

Now do DynamoDB.

aphyr 6y ago
I'd like to, but I don't have any way to do fault injection on a system someone else owns. :(
- jtdev 6y ago
  Would love to see AWS agree to facilitating this. Appreciate your work very much!
- petrikapu 6y ago
  They have downloadable version of it https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
  - TheDong 6y ago
    The downloadable version of dynamodb is only intended for testing and is not a distributed system by any definition, nor does it behavior match the production system exactly. There's no reason for Jepson to be applied to a single-node in-memory kv store.

rmdashrfstar 6y ago

The main argument for using a documented-oriented database: https://martinfowler.com/bliki/AggregateOrientedDatabase.html https://martinfowler.com/bliki/AggregateOrientedDatabase.htm...

zzzeek 6y ago

How many more years do we have to keep evaluating, studying, and reading about MongoDB's ongoing failures? It would appear this product has been a great burden on the community for many years.

aphyr 6y ago
I like to keep in mind that MongoDB's existing feature set is maturing--occasional regressions may happen, but by and large they're making progress. The problems in this analysis were in a transaction system that's only been around for a couple years, so it's had less time to have rough edges sanded off.
- zzzeek 6y ago
  there are so many great databases out there. There's no need for one that has been mediocre for years and continues to make false claims. This is an issue of years of super aggressive marketing of an inferior product making it hard on engineers.
  - trashcan 6y ago
    I think if you compared it to other databases that are designed to scale horizontally like Cassandra and DynamoDB, you might have a more favorable opinion. IMHO, most products at this scale are terrible in different ways, because it is a difficult problem to solve generally. I have been responsible for <100 clustered Cassandra instances, and <500 clustered MongoDB instances, and I would choose the latter every time.
    - runT1ME 6y ago
      Can mongo really support the kind of large scale ETL or time series use cases Cassandra can?
- bborud 6y ago
  It has been a decade since MongoDB was initially released. It still isn't a database in any meaningful sense of the word. Please don't downplay the amount of trouble one can get into by assuming it is a database.

fastball 6y ago

At this point I think we might be going a bit overboard with title changes. Now that it's just "MongoDB 4.2.6", the title makes me think that this is a release announcement, not an analysis of the software. The first title (that specifically referenced a finding of the analysis) was best, imo. Mildly opinionated or whatever, but at least it quickly communicated the gist of the post. On the other hand: "Jepsen: MongoDB 4.2.6" – not super helpful if you're not already familiar with the Jepsen body of work. "MongoDB 4.2.6" – as stated above, sounds like a release announcement. If you want a suggestion, maybe something like "Jepsen evaluation of MongoDB 4.2.6"? Not overly specific (/ negative) like the first title, but at least provides some slight amount of context. @dang

Ecco 6y ago
It's the article's title...
- Fiveplus 6y ago
  No context doesn't help.
- petepete 6y ago
  And taken out of context it makes little sense.
- fiddlerwoaroof 6y ago
  A generic “Mongo 4.2.6” title doesn’t help me decide whether to click on the link (especially with how light the domain is). I thought it was a release announcement and only clicked through to the comments because of yesterday’s discusssion.
  - dang 6y ago
    An HN title needs to be read along with the site name to the right of it.
    - fiddlerwoaroof 6y ago

bbulkow 6y ago

mongodb's business model, forever, has been to get developers to write code, be damned the fact that you can't support it reliably on a cloudy day.

deleted 6y ago

[deleted]

chousuke 6y ago

This article reinforces my stance that bad defaults are a bug. Defaults should be set up with the least number of pitfalls and safety tradeoffs possible so that the system is as robust as it can be for the majority of its users, since the vast majority of them aren't going to change the defaults. Sometimes you end up with bad defaults simply by accident but I feel like for MongoDB the morally correct choice would be to own up to past mistakes and change the defaults rather than maintain a dangerous status quo for "backwards compatibility", even if you end up looking worse in benchmarks as a result.

aphyr 6y ago
I think this is a good way to look at things, and there are vendors who do this! VoltDB, for instance, changed their defaults to be strict serializable even though it imposed a performance hit, following their Jepsen analysis. https://www.voltdb.com/blog/2016/07/voltdb-6-4-passes-official-jepsen-testing/ https://www.voltdb.com/blog/2016/07/voltdb-6-4-passes-offici...

azernik 6y ago

Ouch. This is what you get when you order up a third-party review and then misrepresent it in advertising.

taywrobel 6y ago
I’m still waiting for Jepsen to put Confluent’s “Kafka provides exactly once delivery semantics” claim to the test. Since they’re claiming something provably false, it’d be nice to have some empirical evidence as such.
- aphyr 6y ago
  I'm not convinced it is false--IIRC their claim is specifically w.r.t other Kafka side effects, and those they can control.
- staticassertion 6y ago
  I thought they only guaranteed exactly once processing ? That could just be idempotency.

dang 6y ago

All: there was a big thread about this yesterday (https://news.ycombinator.com/item?id=23285249 https://news.ycombinator.com/item?id=23285249) but because it didn't focus on the technical content, and because there were glitches with a previous submission of this report (described at https://news.ycombinator.com/item?id=23288120 https://news.ycombinator.com/item?id=23288120 and https://news.ycombinator.com/item?id=23287763 https://news.ycombinator.com/item?id=23287763 if anyone cares), we invited aphyr to repost this. Normally we downweight follow-up posts that have such close overlap with a recent discussion (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20follow-up&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...), so the exception is probably worth explaining.

depr 6y ago

>Sometimes, Programs That Use Transactions… Are Worse I understood that reference

pier25 6y ago

> Normally we downweight follow-up posts So you manually moderate the content?

VonGuard 6y ago
I mean, this was kind of an exception case, where there is a big old technical war of words back and forth. Almost a "He said She said" except here, He is an absolute expert, and She is just some marketing dorks at Mongo. I, for one, welcome this by-hand moderation because it keeps this issue alive, and allows Kyle to keep the discussion going. As I commented in a previous post, Kyle is the Chef Ramsey of database testing, and here, he's in a position where some idiot has just served him an undercooked hamburger. Bits will fly, marketing people will be flayed alive, and Kyle will be the only one left standing at the end. Without this by-hand moderation, we'd be missing out on the second act of this intense thriller!
- pier25 6y ago
  I'm totally ok with the moderation/curation/whatever!
deleted 6y ago
[deleted]
DoreenMichele 6y ago
They use a combination of algorithms and human intervention, to generally good effect. No clue if this "downweighting" in this case is an algorithm or a manual thing. I would assume algorithm for the downweighting and human intervention for reversing it, but that's sort of a guess or inference.
baq 6y ago
of course they do? given the quality of discussion here it's a hard requirement to preserve high snr.
dang 6y ago
Oh yes. HN has always been moderated/curated/whatever term you prefer. Many past explanations can be found through these links: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20downweight&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20penalt&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20moderate&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20curate&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... (I've detached this subthread from https://news.ycombinator.com/item?id=23294048 https://news.ycombinator.com/item?id=23294048 to prevent the top comment from being too distracting.)

Laszlotejfel86 6y ago

SSID: Tejfel Lazalo Protocol: Wi-Fi 5 (802.11ac) Security type: WPA2-Personal Network band: 5 GHz Network channel: 157 Link-local IPv6 address: fe80::1d0:5810:36ab:219d%8 IPv4 address: 10.0.0.148 IPv4 DNS servers: 64.71.255.204 64.71.255.198 Manufacturer: Intel Corporation Description: Intel(R) Wireless-AC 9462 Driver version: 20.50.0.5 Physical address (MAC): 7C-2A-31-7B-02-46