15 ms·

So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that? Apparently, we don't hav

So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that? Apparently, we don't have enough wasted cycles in modern software, just add a bunch more in a fundamental IPC infrastructure.

nbbnbb 2y ago
Lets take it a level up. I'm still not sure why we even need a message bus in there in the first place. The whole Linux boot, init, systemd pile is a Rube Goldberg machine that is very difficult to understand at the moment. The only reason I suspect most people aren't complaining is that our abstraction level is currently a lot higher (docker / kubernetes etc) and machines are mostly ephemeral cattle and few people go near it. As for JSON, please don't even start me on that. It is one of the worst serialisation decisions we ever made as a society. Poorly defined and unreliable primitive types, terrible schema support and expensive to parse. In a kernel, it does not belong! Give it a few months and varlink will be YAML over the top of that.
- otabdeveloper4 2y ago
  > I'm still not sure why we even need a message bus in there in the first place. Because traditional POSIX IPC mechanisms are absolute unworkable dogshit. > It is one of the worst serialisation decisions we ever made as a society. There isn't really any alternative. It's either JSON or "JSON but in binary". (Like CBOR.) Anything else is not interoperable.
  - nbbnbb 2y ago
    This is a quite frankly ridiculous point. Most of that garb came from the HPC people who built loads of stuff on top of it in the first place. It's absolutely fine for this sort of stuff. It's sending the odd little thing here and there, not on a complex HPC cluster. As for JSON, are you really that short sighted that it's the only method of encoding something? Is "oh well it doesn't fit the primitive types, so just shove it in a string and add another layer of parsing" acceptable? Hell no.
    - otabdeveloper4 2y ago
      > ...it's the only method of encoding something? If you want something on the system level parsable by anything? Yes it is.
      - nbbnbb 2y ago
        protobufs / asn.1 / structs ... Edit: hell even XML is better than this!
        baq 2y ago
        Thank goodness they didn’t pick YAML though.
        nbbnbb 2y ago
        Yet!
        RandomThoughts3 2y ago
        Structs are a part of C semantic. They are not an ipc format. You can somewhat use them like one if you take a lot of precaution about how they are laid out in memory including padding and packing but it’s very brittle. Asn.1 is both quite complicated and not very efficient. They could certainly have gone with protobufs or another binary serialisation format but would it really be better than the current choice? I don’t think the issue they are trying to solve is related to serialisation anyway. Seems to me they are unhappy about the bus part not the message format part.
        dfox 2y ago
        ASN.1 BER/DER is more or less the same thing as CBOR. The perceived complexity of ASN.1 comes from the schema language and specifications written in the convoluted telco/ITU-T style (and well, the 80's type system that has ~8 times two different types for “human readable string”).
  - jchw 2y ago
    There are a world of serialization formats that can offer a similar interoperability story to JSON or JSON-but-binary formats. And sure, implementing them in every language that someone might be interested in using them in might require complication, but: - Whatever: people in more niche languages are pretty used to needing to do FFI for things like this anyhow. - Many of them already have a better ecosystem than D-Bus. e.g. interoperability between Protobuf and Cap'n'proto implementations is good. Protobuf in most (all?) runtimes supports dynamically reading a schema and parsing binary wire format with it, as well as code generation. You can also maintain backwards compatibility in these formats by following relatively simple rules that can be statically-enforced. - JSON and JSON-but-binary have some annoying downsides. I really don't think field names of composite types belong as part of the ABI. JSON-like formats also often have to try to deal with the fact that JSON doesn't strictly define all semantics. Some of them differ from JSON is subtle ways, so supporting both JSON and sorta-JSON can lead to nasty side-effects. Maybe most importantly, since we're not writing software that's speaking to web browsers, JSON isn't even particularly convenient to begin with. A lot of the software will be in C and Rust most likely. It helps a bit for scripting languages like Python, but I'm not convinced it's worth the downsides.
    - otabdeveloper4 2y ago
      Sorry, but bash isn't a "niche language" and it doesn't have an FFI story.
      - jchw 2y ago
        I don't know how to tell you this, but, you don't need to implement an RPC protocol in bash, nor do you need FFI. You can use CLI tools like `dbus-send`. I pray to God nothing meaningful is actually doing what you are insinuating in any serious environment.
        otabdeveloper4 2y ago
        I'm trying to tell you that something that isn't straceable and greappable shouldn't belong in your system services stack.
- jchw 2y ago
  I think it doesn't have to be a message bus per-se, that design decision is mostly just because it's convenient. Even D-Bus can actually be used without the bus, if you really want to. D-Bus is just a bus so it can solve a bunch of different IPC problems at the same time. e.g. D-Bus can handle ACLs/permissions on RPC methods, D-Bus can handle multiple producers and multiple consumers, and so on. I think ultimately all that's really needed is IPC, and having a unified bus is just to allow for some additional use cases that are harder if you're only using UNIX domain sockets directly. If there are going to be systemd components that have IPC, then I'd argue they should probably use something standard rather than something bespoke. It's good to not re-invent the wheel. Not that I think Varlink is any better. It seems like at best it's probably a lateral move. I hope this does not go forward.
  - nbbnbb 2y ago
    > If there are going to be systemd components that have IPC, then I'd argue they should probably use something standard rather than something bespoke. It's good to not re-invent the wheel. This is my point. My favourite DBus situation a number of years ago was a CentOS 7 box that reboot stopped working on with a cryptic DBus error that no one has ever seen before. I had to sync it, power cycle the node from the ILO card and cross my fingers. I really don't give a shit about this. I just wanted to run my jobs on the node, not turn into a sysadmin due to someone else's dubious architectural decisions.
    - jchw 2y ago
      Yes but the systemd developers don't want to implement their own protocols with e.g. ACL checking, and given some of their track record I kind of think you don't want them to, either. I'm pretty sure the error conditions would be even more bespoke if they "just" used UNIX domain sockets directly. Don't get me wrong, there's nothing particularly wrong with UNIX domain sockets, but there's no "standard" protocols for communicating over UDS.
      - amluto 2y ago
        This is systemd we’re talking about. A service manager that already mucks with mount namespaces. It would be quite straightforward to map a capability-like UNIX socket into each service’s filesystem and give it a private view of the world. But instead… > Public varlink interfaces are registered system-wide by their well-known address, by default /run/org.varlink.resolver. The resolver translates a given varlink interface to the service address which provides this interface. …we have well known names, and sandboxing, or replacing a service for just one client, remains a mess. Sigh.
- therein 2y ago
  Yeah, how will number/float serialization go? Are we going to serialize them as strings and parse them? That abstraction isn't handled the same way across multiple languages.
- poettering 2y ago
  Varlink is not a message bus. Hence you should be happy?
jchw 2y ago
Unfortunately I concur. There are a lot of problems with D-Bus, but that it wasn't using JSON wasn't one of them. The tooling for D-Bus is terrible. It has composite types but they are very primitive, e.g. you can't name fields in a structure without extensions (there's like three different extensions for this.) A lot of the code generators can't actually handle real-world schemas. Now I understand that D-Bus itself is also a problem (as in, the protocol design beyond serialization) but varlink looks like a very solid step backwards. In any single individual use case JSON serialization is unlikely to be a huge bottleneck, but in practice if this will be used for all kinds of things it will add up. I really wish the Linux desktop would coalesce around something like Cap'n'proto for serialization. P.S.: It'd probably be possible to shoehorn in multiple formats but I think that's a mistake too. Something like capnp will have a strong impedance mismatch with JSON; mixing formats with different degrees of self-describability is unwise. But then formats like MsgPack and BSON which are direct JSON analogues miss out on the best benefits of using binary serialization formats, which makes them almost the worst of both worlds, and you'd probably be better off with using a highly-optimized JSON library like simdjson.
- arianvanp 2y ago
  CBOR is the obvious choice here. I've suggested it before and the systemd folks didn't seem completely opposed to it. Also because cbor parser is already in the dependencies due to FIDO2 disk unlocking
  - theamk 2y ago
    What is the data rate (messages, bytes) that you expect for this "D-Bus replacement protocol"? Which fraction of CPU will JSON ser/des will take to justify using CBOR?
    - XorNot 2y ago
      I mean conversely every second week on HN someone complains about "inefficient modern development practices" and this is the sort of minor thing which would be trotted out as an example. So I'd argue it's not an unreasonable question (although I lean closer in: better types then JSON is the problem, since a decent text serialization format is always going to be needed for debugging and development).
deivid 2y ago
There was a Varlink talk[0] a few days ago at All Systems Go, in that talk, Lennart mentioned that JSON is unfortunate (primarily due to no 64 bit ints) but it has the surprising benefit of being able to understand the bus messages when using `strace` for debugging. [0]: https://www.youtube.com/watch?v=emaVH2oWs2Y&list=PLWYdJViL9EipIImmvuoGFAeS-lKeHH2DD&index=34 https://www.youtube.com/watch?v=emaVH2oWs2Y&list=PLWYdJViL9E...
- gmokki 2y ago
  I do not understand where the 64bit integers not working with JSON comes from. JSON the format had no limit on integer size. And all Java JSON libraries I know can handle arbitrary prevsion integers (BigInt) and 32/64bit int/long types when serializing and deserializing. Quick googling shows that also JavaScript has proper support for 64bit integers with their BigInt type, and it can be used to deserialize incoming data correctly with the default parser, albeit requiring a bit more work to annotate the fields. Personally I often explicitly make sure that the integers I return in trust environments as identifiers in REST APIs are by default longer than 52bits so that buggy parser libraries are caught early.
  - GrayShade 2y ago
    Qt refused for almost a decade to support deserializing 64-bit integers from JSON because of compatibility concerns.
    - Spivak 2y ago
      jq couldn't handle them until relatively recently. This isn't a few bad parsers. You can't assume a json parser will handle bigints correctly and when you're angling to be low-level plumbing and work with every language and software that hasn't been recompiled in years you have to be pretty conservative in what you send.
      - capitainenemo 2y ago
        Yeah, and it isn't like jq is being incorrect in this, the JSON RFCs recommend not using values that can't be represented as a 64 bit float (pasted links to a few RFCs in another response). So if you want to represent a large number safely, better to put it in a string, where special handling can be done after the initial JSON parse without loss of information.
jsnell 2y ago
I've done similar things in the past (json over udp-to-localhost, with a schema description used to generate the code for both parsing and generating on each end). It's a totally reasonable point in the design space, and I literally never saw a reason to revisit the decision to use json for that system. You'd do that because everything under the sun knows how to generate and parse json. Performance looks like a total non-issue for this use case. The messages are tiny, local and infrequent. You wouldn't want to do this on some kind of a high performance application on the data plane, but on the control plane it's totally fine. Even if you expect that all production use cases will use some kind of higher level library that does schema validation, it can still be quite nice during debugging to a) be able to inspect the messages on the wire when debugging, b) to be able to hand-write and inject messages.
- ptx 2y ago
  > Performance looks like a total non-issue for this use case. The presentation brings up performance as one of the problems with the D-Bus though, so it seems like it is an issue.
  - jsnell 2y ago
    Fair. What I meant is that the performance delta between JSON and any other interchange data format should be a non-issue in this context. I would be quite surprised to hear that any performance problems systemd has with D-Bus was with serialization/de-serialization.
    - IshKebab 2y ago
      Yeah... it's easy to say JSON won't be an issue. It's difficult to switch from JSON when it turns out you were wrong. https://raphlinus.github.io/xi/2020/06/27/xi-retrospective.html https://raphlinus.github.io/xi/2020/06/27/xi-retrospective.h... Given how central this is going to be why would you skip the tiny bit of extra work needed to use a binary format?
    - dathinab 2y ago
      > performance delta between JSON and any other interchange data format should be a non-issue it's hard to say, for most D-Bus taks perf. is an absolute non issues, but still you have bus-1 and all kind of tries to make D-Bus perform faster, reason is there are edge cases where it matters and even if the marshaling speed might not matter the size can, like if something goes really wrong and tens of thousand of system events get spammed the difference between the memory usage of in-flight messages of a compact format vs. json can make the difference between memory pressure effectively killing your server and you recovering reliable though what is an open question is how relevant that is in pracive and that is quite hard to say without inside information for companies running systems where such a thing could happen on a scale where such a thing does happen (e.g. google)
Asooka 2y ago
To be honest I don't see why they don't use the existing IPC mechanism created for Android - binder and ashmem. It has been tested and developed in real world scenarios for over a decade by now.
- RandomThoughts3 2y ago
  Varlink was specifically designed to be available as early as possible when the kernel is booting and requires virtually nothing apart from good old sockets. But yes, definitely getting yet another one vibe from this. Then again, Dbus is not really what I would call an incredible piece of software - we are very much in the adequate territory if even - so anything shaking up the status quo can’t be too bad.
  - deleted 2y ago
    [deleted]
- AshamedCaptain 2y ago
  Same reason Android (or PalmOS) didn't decide to use any of the existing IPC mechanisms, and why then Luna/webOS also didn't decide to use Binder even though they had experience with it and reinvent something else. That urge that forces every developer to reinvent RPC every other week.
  - altfredd 2y ago
    Not really. Binder solves a real-world problem — priority inversion. It is also the only IPC on Linux that allows to perform a blocking call from process A to process B (and have process B synchronously call back to process A if needed). D-Bus, Varlink and other "IPC solutions" on top of sockets can not do those things. Android uses synchronous Binder calls for 90% of it's application APIs, so there is clearly a real use case for it. Incidentally, Binder itself does not specify message format. A message is a simple memory buffer. Android runtime uses a custom binary serialization format, but you can use JSON or whatever. Binder is an alternative to sockets/pipes, not D-Bus or other wrappers on top of them.
hi-v-rocknroll 2y ago
Sigh. Cap'n Proto already exists. Reinventing the wheel yet again because NIH.
- NekkoDroid 2y ago
  (Varlink also isn't something new)
- IshKebab 2y ago
  CapnProto is kind of awful though. I had to use it for several years and the API is extremely unfriendly.
  - kentonv 2y ago
    Which implementation were you using? (I.e. which programming language?)
    - IshKebab 2y ago
      C++
theamk 2y ago
Sigh. Varlink is designed for IPC, so the expected data rate is maybe hundreds of messages per second in the worst case. JSON can be parsed at 3 Gigabytes/second [0]. Even unoptimized stdlib parsers in scripting languages get 50 MB/sec. That's more than enough, standard library support is much more important for a project like this. And if there is ever a reason to send tar archive or video data, there is always "upgrade" option to switch to raw TCP sockets, with no JSON overhead at all. [0] https://news.ycombinator.com/item?id=19214387 https://news.ycombinator.com/item?id=19214387 [1] https://github.com/TeskaLabs/cysimdjson https://github.com/TeskaLabs/cysimdjson
- fanf2 2y ago
  Hundreds of messages per second sounds like the performance I would expect from a 1980s computer. Why so incredibly slow?
  - orf 2y ago
    Read the comment you’re replying to. In it OP says it’s not slow, and the “hundreds of messages per second” you are referring to is not regarding the performance but the expected rate. It’s possible for something to be fast, but only happen infrequently.
    - ptx 2y ago
      It just seems strange that the message rate is assumed to be in some very specific range where the performance of D-Bus is a problem (mentioned in the slides) yet the performance overhead of JSON is not a problem.
- userbinator 2y ago
  It doesn't matter what the rate is. Of all the things they could do, they decided to choose one of the worst possible formats for IPC data[1], maybe to pander to the web crowd. [1] Maybe XML would be a close competitor.
  - 2y ago
deleted 2y ago
[deleted]
m463 2y ago
As a little bit of perspective, I thought along the same lines when I saw the first web browser. Text was good, but why did they have all these inline images. It was so wasteful of bandwidth. (also, unix uses files for everything, including /proc talking to drivers!)
surajrmal 2y ago
Lennart pointed out the fact you can see readable messages via strace to be a benefit of json. If their average message size is small and they don't expect to ever need high volume or large messages, then it's not really likely to be a problem in practice. He also pointed out that they waste far more cycles today on context switches (he suggested something on the order of 6 per IPC message). Even if they eventually come to the conclusion they need something more performant in the future, it's probably still a net win to make the switch today. It's also possible it turns out fine for their use case. I personally would think they would want to benefit from an IPC system that includes provisions for sending file descriptors.
- markasoftware 2y ago
  > Lennart pointed out the fact you can see readable messages via strace to be a benefit of json. from the guy who brought you binary logfiles!
  - solarengineer 2y ago
    One can view the binary log files using journalctl. Per https://systemd.io/JOURNAL_FILE_FORMAT/ https://systemd.io/JOURNAL_FILE_FORMAT/, the benefits of the binary format are: The systemd journal stores log data in a binary format with several features: Fully indexed by all fields Can store binary data, up to 2^64-1 in size Seekable Primarily append-based, hence robust to corruption Support for in-line compression Support for in-line Forward Secure Sealing As a user system-administrator, I see the cryptographic checksum as a benefit of being able to show tampering evidence of on-system log files.
    - Dunedan 2y ago
      > Primarily append-based, hence robust to corruption It's so robust, it doesn't even let you modify the journal if you want to (e.g. https://github.com/systemd/systemd/issues/20673 https://github.com/systemd/systemd/issues/20673). > Support for in-line compression Mind that journald only supports compressing single lines, but not the whole journal (https://github.com/systemd/systemd/issues/31358 https://github.com/systemd/systemd/issues/31358), which is pretty limiting.
poettering 2y ago
The marshalling cost for JSON is negligible. Yes, it might be a bit slower than GVariant for example, but only by some fractional linear factor. And on small messages (which D-Bus currently always is, due to message size constraints enforced by broker) the difference is impossible to measure. To a point it really doesn't matter, in particular as JSON parsers have been ridiculously well optimized in this world. What does matter though are roundtrips. In Varlink there are much fewer required for typical ops than there are in D-Bus. That's because D-Bus implies a broker (which doubles the number of roundtrips), but also because D-Bus forces you into a model of sending smaller "summary" messages when enumerating plus querying "details" for each listed objects, because it enforces transfer rate limits on everything (if you hit them, you are kicked off the bus), which means you have to refrain from streaming too large data. Or in other words: marshalling is quite an irrelevant minor detail when it comes to performance, you must look at roundtrips instead and the context switches it effects, instead. Using JSON for this has two major benefits: the whole world speaks JSON, and modern programming languages typically pretty natively. And it's directly readable in tools such as strace. With a simple "strace" I can now reasonably trace my programs, which a binary serialization will never allow you. And if you tell me that that doesn't matter, then you apparently live in an entirely different world than I do, because in mine debuggability does matter. A lot. Probably more than most other things. Lennart
- steventhedev 2y ago
  The danger I see is that JSON has lots of edge behavior around deserialization, and some languages will deserialize a 100 digit number differently. If the main benefit is removing the broker and the need for rate limiting - it could have been accomplished without using JSON.
  - poettering 2y ago
    You are writing this as if JSON was a newly invented thing, and not a language that has become the lingua franca of the Internet when it comes to encoding structured data. Well understood, and universally handled, since 1997. A 100 digit number cannot be encoded losslessly in D-Bus btw, nor in the far majority of IPC marshallings on this word. Having done systems-level OS development since 25y or so I never felt the burning urge to send a 100 digit number over local IPC. Not that 100 digit numbers aren't useful, even in IPC, but typically, that's a cryptography thing, and they generally use their own serializations anyway.
pjmlp 2y ago
This looks like a joke, naturally D-BUS without JSON is what is preventing Linux on Desktop to take off.
- ChoHag 2y ago
  [dead]
- guappa 2y ago
  You surely are aware that systemd and dbus aren't only used on desktop, right?
  - pjmlp 2y ago
    A side effect of its adoption, D-BUS was created as replacement for Bonobo and DCOP.
    - guappa 2y ago
      That was 15 years ago? I was talking about the usage today.
      - pjmlp 2y ago
        And I was being sarcastic, as there are things much more relevant to spend brain cells on.
        guappa 2y ago
        Wasn't sarcasm frowned upon here?
thayne 2y ago
I think it is primarily for compatibility. Practically every language has a readily available library, often part of the standard library, for json. That is less true for something like Msgpack or cbor, and even less so if make a new binary format. Also, there are benefits to the format being human readable.
jeroenhd 2y ago
Not a fan of JSON per se for this type of messaging, but I don't think binary parser+validator combinations are that much faster. Plus, this forces C(++) developers to actually validate the data they're exchanging rather than casting raw structs from data buffers, which solves a huge vulnerability. Then again, the lack of 64 bit integers makes the protocol design comically unsuited. I don't think the serialization/deserialisation overhead matters much when also taking into account the bus connection overhead. I would've preferred a better data exchange format but out of all problems I have with Linux IPC, the performance of JSON is the least of my worries.
- poettering 2y ago
  The 64bit issue is certainly an issue, but very much overblown. First of all, in systemd, which is a heavy D-Bus user, we effectively IRL only send integers > 2^53 when we use UINT64_MAX as a special "niche" marker meaning "unlimited"/"unset"/"undefined". But the thing is that JSON has a proper concept for this: the "null" value (or simply not specifying a specific JSON field). Hence, in reasonably clean APIs this is mostly a non-issue. That said, sd-varlink/sd-json (i.e. systemd's implementation of it), of course is 64bit signed and unsigned integer clean when it processes JSON. More-over it automatically handles if you do what the various specs on the internet suggest you do if you have an integer > 2^53: you encode it as decimal value as a string. Would it be better if JSON would have been more precise on this, yes. Is it a big issue? No, not at all.
  - Netch 2y ago
    > if you do what the various specs on the internet suggest you do if you have an integer > 2^53: you encode it as decimal value as a string. Hexadecimal is better - more compact, much less encoding/decoding effort. Alternatively, array of two integers does as well. Text forms for any error, not NaN, are also useful. But issues with non-UTF-8 encodings - as in CJK world - may spoil the picture pretty more.
  - accelbred 2y ago
    Are strings used for timestamps? Those are usually why I'm sending 64-bit ints over IPC.
foul 2y ago
It's probably the first time in 14 years of systemd I appreciate a technical choice coming from the systemd team. It's a small format, it has limited pitfalls, it's fast to marshal and unmarshal (and with simd gets even more fast), it's deflate-friendly, would performance be an issue they can pack that through CBOR or msgpack or what-have-you.
bitwize 2y ago
> So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that? Someone took a look at D-Bus and decided "You know what? It just isn't shitty enough. It's 2024 and D-Bus runs halfway decently on the average laptop. Let's fix that." I wouldn't be surprised if the Scooby Gang pulled the mask off the head dev, revealing ThePrimeagen, in a scheme of long-form comedy that's a cross between one of his "let's write a network protocol" streams and his new enterprise development satire series.
deleted 2y ago
[deleted]
eternityforest 2y ago
How many messages is this actually gonna handle? If it's tens per second, should you maybe be using something else? JSON is so prevalent I'm pretty sure it's a lot of people's mental model of what non relational structured data looks like, if it's not just a file. It's nice from a dev perspective to always work directly with the same high level concepts that you use when thinking about this stuff. Other data formats are often advertised in terms of JSON, either subsets or supersets, or equivalent representations. I'd use something else for high performance stuff.... But for everything else it's pretty awesome.

nbbnbb 2y ago

Lets take it a level up. I'm still not sure why we even need a message bus in there in the first place. The whole Linux boot, init, systemd pile is a Rube Goldberg machine that is very difficult to understand at the moment. The only reason I suspect most people aren't complaining is that our abstraction level is currently a lot higher (docker / kubernetes etc) and machines are mostly ephemeral cattle and few people go near it. As for JSON, please don't even start me on that. It is one of the worst serialisation decisions we ever made as a society. Poorly defined and unreliable primitive types, terrible schema support and expensive to parse. In a kernel, it does not belong! Give it a few months and varlink will be YAML over the top of that.

otabdeveloper4 2y ago
> I'm still not sure why we even need a message bus in there in the first place. Because traditional POSIX IPC mechanisms are absolute unworkable dogshit. > It is one of the worst serialisation decisions we ever made as a society. There isn't really any alternative. It's either JSON or "JSON but in binary". (Like CBOR.) Anything else is not interoperable.
- nbbnbb 2y ago
  This is a quite frankly ridiculous point. Most of that garb came from the HPC people who built loads of stuff on top of it in the first place. It's absolutely fine for this sort of stuff. It's sending the odd little thing here and there, not on a complex HPC cluster. As for JSON, are you really that short sighted that it's the only method of encoding something? Is "oh well it doesn't fit the primitive types, so just shove it in a string and add another layer of parsing" acceptable? Hell no.
  - otabdeveloper4 2y ago
    > ...it's the only method of encoding something? If you want something on the system level parsable by anything? Yes it is.
    - nbbnbb 2y ago
      protobufs / asn.1 / structs ... Edit: hell even XML is better than this!
      - baq 2y ago
        Thank goodness they didn’t pick YAML though.
        nbbnbb 2y ago
        Yet!
      - RandomThoughts3 2y ago
        Structs are a part of C semantic. They are not an ipc format. You can somewhat use them like one if you take a lot of precaution about how they are laid out in memory including padding and packing but it’s very brittle. Asn.1 is both quite complicated and not very efficient. They could certainly have gone with protobufs or another binary serialisation format but would it really be better than the current choice? I don’t think the issue they are trying to solve is related to serialisation anyway. Seems to me they are unhappy about the bus part not the message format part.
        dfox 2y ago
        ASN.1 BER/DER is more or less the same thing as CBOR. The perceived complexity of ASN.1 comes from the schema language and specifications written in the convoluted telco/ITU-T style (and well, the 80's type system that has ~8 times two different types for “human readable string”).
- jchw 2y ago
  There are a world of serialization formats that can offer a similar interoperability story to JSON or JSON-but-binary formats. And sure, implementing them in every language that someone might be interested in using them in might require complication, but: - Whatever: people in more niche languages are pretty used to needing to do FFI for things like this anyhow. - Many of them already have a better ecosystem than D-Bus. e.g. interoperability between Protobuf and Cap'n'proto implementations is good. Protobuf in most (all?) runtimes supports dynamically reading a schema and parsing binary wire format with it, as well as code generation. You can also maintain backwards compatibility in these formats by following relatively simple rules that can be statically-enforced. - JSON and JSON-but-binary have some annoying downsides. I really don't think field names of composite types belong as part of the ABI. JSON-like formats also often have to try to deal with the fact that JSON doesn't strictly define all semantics. Some of them differ from JSON is subtle ways, so supporting both JSON and sorta-JSON can lead to nasty side-effects. Maybe most importantly, since we're not writing software that's speaking to web browsers, JSON isn't even particularly convenient to begin with. A lot of the software will be in C and Rust most likely. It helps a bit for scripting languages like Python, but I'm not convinced it's worth the downsides.
  - otabdeveloper4 2y ago
    Sorry, but bash isn't a "niche language" and it doesn't have an FFI story.
    - jchw 2y ago
      I don't know how to tell you this, but, you don't need to implement an RPC protocol in bash, nor do you need FFI. You can use CLI tools like `dbus-send`. I pray to God nothing meaningful is actually doing what you are insinuating in any serious environment.
      - otabdeveloper4 2y ago
        I'm trying to tell you that something that isn't straceable and greappable shouldn't belong in your system services stack.
jchw 2y ago
I think it doesn't have to be a message bus per-se, that design decision is mostly just because it's convenient. Even D-Bus can actually be used without the bus, if you really want to. D-Bus is just a bus so it can solve a bunch of different IPC problems at the same time. e.g. D-Bus can handle ACLs/permissions on RPC methods, D-Bus can handle multiple producers and multiple consumers, and so on. I think ultimately all that's really needed is IPC, and having a unified bus is just to allow for some additional use cases that are harder if you're only using UNIX domain sockets directly. If there are going to be systemd components that have IPC, then I'd argue they should probably use something standard rather than something bespoke. It's good to not re-invent the wheel. Not that I think Varlink is any better. It seems like at best it's probably a lateral move. I hope this does not go forward.
- nbbnbb 2y ago
  > If there are going to be systemd components that have IPC, then I'd argue they should probably use something standard rather than something bespoke. It's good to not re-invent the wheel. This is my point. My favourite DBus situation a number of years ago was a CentOS 7 box that reboot stopped working on with a cryptic DBus error that no one has ever seen before. I had to sync it, power cycle the node from the ILO card and cross my fingers. I really don't give a shit about this. I just wanted to run my jobs on the node, not turn into a sysadmin due to someone else's dubious architectural decisions.
  - jchw 2y ago
    Yes but the systemd developers don't want to implement their own protocols with e.g. ACL checking, and given some of their track record I kind of think you don't want them to, either. I'm pretty sure the error conditions would be even more bespoke if they "just" used UNIX domain sockets directly. Don't get me wrong, there's nothing particularly wrong with UNIX domain sockets, but there's no "standard" protocols for communicating over UDS.
    - amluto 2y ago
      This is systemd we’re talking about. A service manager that already mucks with mount namespaces. It would be quite straightforward to map a capability-like UNIX socket into each service’s filesystem and give it a private view of the world. But instead… > Public varlink interfaces are registered system-wide by their well-known address, by default /run/org.varlink.resolver. The resolver translates a given varlink interface to the service address which provides this interface. …we have well known names, and sandboxing, or replacing a service for just one client, remains a mess. Sigh.
therein 2y ago
Yeah, how will number/float serialization go? Are we going to serialize them as strings and parse them? That abstraction isn't handled the same way across multiple languages.
poettering 2y ago
Varlink is not a message bus. Hence you should be happy?

jchw 2y ago

Unfortunately I concur. There are a lot of problems with D-Bus, but that it wasn't using JSON wasn't one of them. The tooling for D-Bus is terrible. It has composite types but they are very primitive, e.g. you can't name fields in a structure without extensions (there's like three different extensions for this.) A lot of the code generators can't actually handle real-world schemas. Now I understand that D-Bus itself is also a problem (as in, the protocol design beyond serialization) but varlink looks like a very solid step backwards. In any single individual use case JSON serialization is unlikely to be a huge bottleneck, but in practice if this will be used for all kinds of things it will add up. I really wish the Linux desktop would coalesce around something like Cap'n'proto for serialization. P.S.: It'd probably be possible to shoehorn in multiple formats but I think that's a mistake too. Something like capnp will have a strong impedance mismatch with JSON; mixing formats with different degrees of self-describability is unwise. But then formats like MsgPack and BSON which are direct JSON analogues miss out on the best benefits of using binary serialization formats, which makes them almost the worst of both worlds, and you'd probably be better off with using a highly-optimized JSON library like simdjson.

arianvanp 2y ago
CBOR is the obvious choice here. I've suggested it before and the systemd folks didn't seem completely opposed to it. Also because cbor parser is already in the dependencies due to FIDO2 disk unlocking
- theamk 2y ago
  What is the data rate (messages, bytes) that you expect for this "D-Bus replacement protocol"? Which fraction of CPU will JSON ser/des will take to justify using CBOR?
  - XorNot 2y ago
    I mean conversely every second week on HN someone complains about "inefficient modern development practices" and this is the sort of minor thing which would be trotted out as an example. So I'd argue it's not an unreasonable question (although I lean closer in: better types then JSON is the problem, since a decent text serialization format is always going to be needed for debugging and development).

deivid 2y ago

There was a Varlink talk[0] a few days ago at All Systems Go, in that talk, Lennart mentioned that JSON is unfortunate (primarily due to no 64 bit ints) but it has the surprising benefit of being able to understand the bus messages when using `strace` for debugging. [0]: https://www.youtube.com/watch?v=emaVH2oWs2Y&list=PLWYdJViL9EipIImmvuoGFAeS-lKeHH2DD&index=34 https://www.youtube.com/watch?v=emaVH2oWs2Y&list=PLWYdJViL9E...

gmokki 2y ago
I do not understand where the 64bit integers not working with JSON comes from. JSON the format had no limit on integer size. And all Java JSON libraries I know can handle arbitrary prevsion integers (BigInt) and 32/64bit int/long types when serializing and deserializing. Quick googling shows that also JavaScript has proper support for 64bit integers with their BigInt type, and it can be used to deserialize incoming data correctly with the default parser, albeit requiring a bit more work to annotate the fields. Personally I often explicitly make sure that the integers I return in trust environments as identifiers in REST APIs are by default longer than 52bits so that buggy parser libraries are caught early.
- GrayShade 2y ago
  Qt refused for almost a decade to support deserializing 64-bit integers from JSON because of compatibility concerns.
  - Spivak 2y ago
    jq couldn't handle them until relatively recently. This isn't a few bad parsers. You can't assume a json parser will handle bigints correctly and when you're angling to be low-level plumbing and work with every language and software that hasn't been recompiled in years you have to be pretty conservative in what you send.
    - capitainenemo 2y ago
      Yeah, and it isn't like jq is being incorrect in this, the JSON RFCs recommend not using values that can't be represented as a 64 bit float (pasted links to a few RFCs in another response). So if you want to represent a large number safely, better to put it in a string, where special handling can be done after the initial JSON parse without loss of information.

jsnell 2y ago

I've done similar things in the past (json over udp-to-localhost, with a schema description used to generate the code for both parsing and generating on each end). It's a totally reasonable point in the design space, and I literally never saw a reason to revisit the decision to use json for that system. You'd do that because everything under the sun knows how to generate and parse json. Performance looks like a total non-issue for this use case. The messages are tiny, local and infrequent. You wouldn't want to do this on some kind of a high performance application on the data plane, but on the control plane it's totally fine. Even if you expect that all production use cases will use some kind of higher level library that does schema validation, it can still be quite nice during debugging to a) be able to inspect the messages on the wire when debugging, b) to be able to hand-write and inject messages.

ptx 2y ago
> Performance looks like a total non-issue for this use case. The presentation brings up performance as one of the problems with the D-Bus though, so it seems like it is an issue.
- jsnell 2y ago
  Fair. What I meant is that the performance delta between JSON and any other interchange data format should be a non-issue in this context. I would be quite surprised to hear that any performance problems systemd has with D-Bus was with serialization/de-serialization.
  - IshKebab 2y ago
    Yeah... it's easy to say JSON won't be an issue. It's difficult to switch from JSON when it turns out you were wrong. https://raphlinus.github.io/xi/2020/06/27/xi-retrospective.html https://raphlinus.github.io/xi/2020/06/27/xi-retrospective.h... Given how central this is going to be why would you skip the tiny bit of extra work needed to use a binary format?
  - dathinab 2y ago
    > performance delta between JSON and any other interchange data format should be a non-issue it's hard to say, for most D-Bus taks perf. is an absolute non issues, but still you have bus-1 and all kind of tries to make D-Bus perform faster, reason is there are edge cases where it matters and even if the marshaling speed might not matter the size can, like if something goes really wrong and tens of thousand of system events get spammed the difference between the memory usage of in-flight messages of a compact format vs. json can make the difference between memory pressure effectively killing your server and you recovering reliable though what is an open question is how relevant that is in pracive and that is quite hard to say without inside information for companies running systems where such a thing could happen on a scale where such a thing does happen (e.g. google)

Asooka 2y ago

To be honest I don't see why they don't use the existing IPC mechanism created for Android - binder and ashmem. It has been tested and developed in real world scenarios for over a decade by now.

RandomThoughts3 2y ago
Varlink was specifically designed to be available as early as possible when the kernel is booting and requires virtually nothing apart from good old sockets. But yes, definitely getting yet another one vibe from this. Then again, Dbus is not really what I would call an incredible piece of software - we are very much in the adequate territory if even - so anything shaking up the status quo can’t be too bad.
- deleted 2y ago
  [deleted]
AshamedCaptain 2y ago
Same reason Android (or PalmOS) didn't decide to use any of the existing IPC mechanisms, and why then Luna/webOS also didn't decide to use Binder even though they had experience with it and reinvent something else. That urge that forces every developer to reinvent RPC every other week.
- altfredd 2y ago
  Not really. Binder solves a real-world problem — priority inversion. It is also the only IPC on Linux that allows to perform a blocking call from process A to process B (and have process B synchronously call back to process A if needed). D-Bus, Varlink and other "IPC solutions" on top of sockets can not do those things. Android uses synchronous Binder calls for 90% of it's application APIs, so there is clearly a real use case for it. Incidentally, Binder itself does not specify message format. A message is a simple memory buffer. Android runtime uses a custom binary serialization format, but you can use JSON or whatever. Binder is an alternative to sockets/pipes, not D-Bus or other wrappers on top of them.

hi-v-rocknroll 2y ago

Sigh. Cap'n Proto already exists. Reinventing the wheel yet again because NIH.

NekkoDroid 2y ago
(Varlink also isn't something new)
IshKebab 2y ago
CapnProto is kind of awful though. I had to use it for several years and the API is extremely unfriendly.
- kentonv 2y ago
  Which implementation were you using? (I.e. which programming language?)
  - IshKebab 2y ago
    C++

theamk 2y ago

Sigh. Varlink is designed for IPC, so the expected data rate is maybe hundreds of messages per second in the worst case. JSON can be parsed at 3 Gigabytes/second [0]. Even unoptimized stdlib parsers in scripting languages get 50 MB/sec. That's more than enough, standard library support is much more important for a project like this. And if there is ever a reason to send tar archive or video data, there is always "upgrade" option to switch to raw TCP sockets, with no JSON overhead at all. [0] https://news.ycombinator.com/item?id=19214387 https://news.ycombinator.com/item?id=19214387 [1] https://github.com/TeskaLabs/cysimdjson https://github.com/TeskaLabs/cysimdjson

fanf2 2y ago
Hundreds of messages per second sounds like the performance I would expect from a 1980s computer. Why so incredibly slow?
- orf 2y ago
  Read the comment you’re replying to. In it OP says it’s not slow, and the “hundreds of messages per second” you are referring to is not regarding the performance but the expected rate. It’s possible for something to be fast, but only happen infrequently.
  - ptx 2y ago
    It just seems strange that the message rate is assumed to be in some very specific range where the performance of D-Bus is a problem (mentioned in the slides) yet the performance overhead of JSON is not a problem.
userbinator 2y ago
It doesn't matter what the rate is. Of all the things they could do, they decided to choose one of the worst possible formats for IPC data[1], maybe to pander to the web crowd. [1] Maybe XML would be a close competitor.
- 2y ago

deleted 2y ago

[deleted]

m463 2y ago

As a little bit of perspective, I thought along the same lines when I saw the first web browser. Text was good, but why did they have all these inline images. It was so wasteful of bandwidth. (also, unix uses files for everything, including /proc talking to drivers!)

surajrmal 2y ago

Lennart pointed out the fact you can see readable messages via strace to be a benefit of json. If their average message size is small and they don't expect to ever need high volume or large messages, then it's not really likely to be a problem in practice. He also pointed out that they waste far more cycles today on context switches (he suggested something on the order of 6 per IPC message). Even if they eventually come to the conclusion they need something more performant in the future, it's probably still a net win to make the switch today. It's also possible it turns out fine for their use case. I personally would think they would want to benefit from an IPC system that includes provisions for sending file descriptors.

markasoftware 2y ago
> Lennart pointed out the fact you can see readable messages via strace to be a benefit of json. from the guy who brought you binary logfiles!
- solarengineer 2y ago
  One can view the binary log files using journalctl. Per https://systemd.io/JOURNAL_FILE_FORMAT/ https://systemd.io/JOURNAL_FILE_FORMAT/, the benefits of the binary format are: The systemd journal stores log data in a binary format with several features: Fully indexed by all fields Can store binary data, up to 2^64-1 in size Seekable Primarily append-based, hence robust to corruption Support for in-line compression Support for in-line Forward Secure Sealing As a user system-administrator, I see the cryptographic checksum as a benefit of being able to show tampering evidence of on-system log files.
  - Dunedan 2y ago
    > Primarily append-based, hence robust to corruption It's so robust, it doesn't even let you modify the journal if you want to (e.g. https://github.com/systemd/systemd/issues/20673 https://github.com/systemd/systemd/issues/20673). > Support for in-line compression Mind that journald only supports compressing single lines, but not the whole journal (https://github.com/systemd/systemd/issues/31358 https://github.com/systemd/systemd/issues/31358), which is pretty limiting.

poettering 2y ago

The marshalling cost for JSON is negligible. Yes, it might be a bit slower than GVariant for example, but only by some fractional linear factor. And on small messages (which D-Bus currently always is, due to message size constraints enforced by broker) the difference is impossible to measure. To a point it really doesn't matter, in particular as JSON parsers have been ridiculously well optimized in this world. What does matter though are roundtrips. In Varlink there are much fewer required for typical ops than there are in D-Bus. That's because D-Bus implies a broker (which doubles the number of roundtrips), but also because D-Bus forces you into a model of sending smaller "summary" messages when enumerating plus querying "details" for each listed objects, because it enforces transfer rate limits on everything (if you hit them, you are kicked off the bus), which means you have to refrain from streaming too large data. Or in other words: marshalling is quite an irrelevant minor detail when it comes to performance, you must look at roundtrips instead and the context switches it effects, instead. Using JSON for this has two major benefits: the whole world speaks JSON, and modern programming languages typically pretty natively. And it's directly readable in tools such as strace. With a simple "strace" I can now reasonably trace my programs, which a binary serialization will never allow you. And if you tell me that that doesn't matter, then you apparently live in an entirely different world than I do, because in mine debuggability does matter. A lot. Probably more than most other things. Lennart

steventhedev 2y ago
The danger I see is that JSON has lots of edge behavior around deserialization, and some languages will deserialize a 100 digit number differently. If the main benefit is removing the broker and the need for rate limiting - it could have been accomplished without using JSON.
- poettering 2y ago
  You are writing this as if JSON was a newly invented thing, and not a language that has become the lingua franca of the Internet when it comes to encoding structured data. Well understood, and universally handled, since 1997. A 100 digit number cannot be encoded losslessly in D-Bus btw, nor in the far majority of IPC marshallings on this word. Having done systems-level OS development since 25y or so I never felt the burning urge to send a 100 digit number over local IPC. Not that 100 digit numbers aren't useful, even in IPC, but typically, that's a cryptography thing, and they generally use their own serializations anyway.

pjmlp 2y ago

This looks like a joke, naturally D-BUS without JSON is what is preventing Linux on Desktop to take off.

ChoHag 2y ago
[dead]
guappa 2y ago
You surely are aware that systemd and dbus aren't only used on desktop, right?
- pjmlp 2y ago
  A side effect of its adoption, D-BUS was created as replacement for Bonobo and DCOP.
  - guappa 2y ago
    That was 15 years ago? I was talking about the usage today.
    - pjmlp 2y ago
      And I was being sarcastic, as there are things much more relevant to spend brain cells on.
      - guappa 2y ago
        Wasn't sarcasm frowned upon here?

thayne 2y ago

I think it is primarily for compatibility. Practically every language has a readily available library, often part of the standard library, for json. That is less true for something like Msgpack or cbor, and even less so if make a new binary format. Also, there are benefits to the format being human readable.

jeroenhd 2y ago

Not a fan of JSON per se for this type of messaging, but I don't think binary parser+validator combinations are that much faster. Plus, this forces C(++) developers to actually validate the data they're exchanging rather than casting raw structs from data buffers, which solves a huge vulnerability. Then again, the lack of 64 bit integers makes the protocol design comically unsuited. I don't think the serialization/deserialisation overhead matters much when also taking into account the bus connection overhead. I would've preferred a better data exchange format but out of all problems I have with Linux IPC, the performance of JSON is the least of my worries.

poettering 2y ago
The 64bit issue is certainly an issue, but very much overblown. First of all, in systemd, which is a heavy D-Bus user, we effectively IRL only send integers > 2^53 when we use UINT64_MAX as a special "niche" marker meaning "unlimited"/"unset"/"undefined". But the thing is that JSON has a proper concept for this: the "null" value (or simply not specifying a specific JSON field). Hence, in reasonably clean APIs this is mostly a non-issue. That said, sd-varlink/sd-json (i.e. systemd's implementation of it), of course is 64bit signed and unsigned integer clean when it processes JSON. More-over it automatically handles if you do what the various specs on the internet suggest you do if you have an integer > 2^53: you encode it as decimal value as a string. Would it be better if JSON would have been more precise on this, yes. Is it a big issue? No, not at all.
- Netch 2y ago
  > if you do what the various specs on the internet suggest you do if you have an integer > 2^53: you encode it as decimal value as a string. Hexadecimal is better - more compact, much less encoding/decoding effort. Alternatively, array of two integers does as well. Text forms for any error, not NaN, are also useful. But issues with non-UTF-8 encodings - as in CJK world - may spoil the picture pretty more.
- accelbred 2y ago
  Are strings used for timestamps? Those are usually why I'm sending 64-bit ints over IPC.

foul 2y ago

It's probably the first time in 14 years of systemd I appreciate a technical choice coming from the systemd team. It's a small format, it has limited pitfalls, it's fast to marshal and unmarshal (and with simd gets even more fast), it's deflate-friendly, would performance be an issue they can pack that through CBOR or msgpack or what-have-you.

bitwize 2y ago

> So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that? Someone took a look at D-Bus and decided "You know what? It just isn't shitty enough. It's 2024 and D-Bus runs halfway decently on the average laptop. Let's fix that." I wouldn't be surprised if the Scooby Gang pulled the mask off the head dev, revealing ThePrimeagen, in a scheme of long-form comedy that's a cross between one of his "let's write a network protocol" streams and his new enterprise development satire series.

deleted 2y ago

[deleted]

eternityforest 2y ago

How many messages is this actually gonna handle? If it's tens per second, should you maybe be using something else? JSON is so prevalent I'm pretty sure it's a lot of people's mental model of what non relational structured data looks like, if it's not just a file. It's nice from a dev perspective to always work directly with the same high level concepts that you use when thinking about this stuff. Other data formats are often advertised in terms of JSON, either subsets or supersets, or equivalent representations. I'd use something else for high performance stuff.... But for everything else it's pretty awesome.