4 ms·
Linux has already half-solved this, eg with netlink. If you're on a system with huge numbers of connections, reading from /proc/net/tcp get extremely slow. Mod
by throwaway09223 4y ago
Linux has already half-solved this, eg with netlink.
If you're on a system with huge numbers of connections, reading from /proc/net/tcp get extremely slow. Modern tools query connection state using netlink instead (ss vs netstat). This was done by necessity: /proc/net/tcp actually doesn't work at scale.
I agree with you, serializing and deserializing files with records is a terrible idea - it cannot be performant. JSON fixes parsing ambiguity but at a cost of being even slower. We already know it won't work.
We have already solved this for specific parts of /proc and it works great. All we have to do is finish the work and provide the rest of proc via netlink as well (or whatever else similar non-text based system for querying structured records)
- dale_glass 4y agoJSON can be parsed very quickly: https://github.com/simdjson/simdjson https://github.com/simdjson/simdjson CBOR could be another option: https://en.wikipedia.org/wiki/CBOR https://en.wikipedia.org/wiki/CBOR
- the8472 4y agoJSON cannot be efficient for binary data or even utf8 strings because it needs escaping which means you can't have a zero-copy implementation. CBOR is better in that regard, yeah.
- ilyt 4y agoAll it needs to beat is stringifying and formatting that /proc|/sys already does
- throwaway09223 4y agoIt can't (simdjson is fast compared to other text parsing - it's still extremely slow compared to native types), but that fact actually doesn't matter because the problem isn't the parsing. Even the construction of these text files in proc is too slow. You can't read /proc/net/tcp within a reasonable amount of time on a system with hundreds of thousands of connections. Even allocating/churning memory to store a textual representation becomes a problematic overhead.