4 ms·
Netstrings (1997)
- gnabgib 3mo ago(1997) -DJB
- ocrow 3mo agoSeems like a coherent, sensible proposal, as one might expect from djb. Any notable protocols use them?
- Scaevolus 3mo agoBitTorrent's bencoding format, used in .torrent files, effectively uses netstrings-- but without the trailing commas, so it uses "5:hello" to represent filenames and similar.
- deleted 3mo ago[deleted]
- toast0 3mo agoPhp serialized uses s:size:value; For strings, which is pretty similar. Size is in bytes.
- asalahli 3mo agoNot sure if it counts as notable, but SCGI uses it too: https://python.ca/scgi/protocol.txt https://python.ca/scgi/protocol.txt
- Asmod4n 3mo agozurl and mongrel2 are using it.
- regularfry 3mo agoTagged Netstrings (tnetstrings) was a related proposal from 15 years ago or so. It replaces the comma with a single-character type definition so you can do JSON-like objects with a couple of recursive types: you had ',', '#', '^', '!', and '~' for strings, integers, floats, booleans, and nulls, then ']' and '}' for lists and dictionaries. Most of the links have bitrotted and I don't think it ever got much traction, but I did always like how simple it was. There's a copy someone grabbed of the original spec here: https://raw.githubusercontent.com/ged/tnetstrings.info/refs/heads/live/index.html https://raw.githubusercontent.com/ged/tnetstrings.info/refs/...
- bmacho 3mo agoHere's a HTML viewer (not mine; use it at your own risk): <https://html-preview.github.io/?url=https://raw.githubusercontent.com/ged/tnetstrings.info/refs/heads/live/index.html https://html-preview.github.io/?url=https://raw.githubuserco...>
- Joker_vD 3mo agoif (scanf("%9lu",&len) < 1) barf(); /* >999999999 bytes is bad */ if (getchar() != ':') barf(); buf = malloc(len + 1); /* malloc(0) is not portable */ if (!buf) barf(); if (fread(buf,1,len,stdin) < len) barf(); if (getchar() != ',') barf(); Ah, the wonders of error-handling in C. Also, I wonder what's wrong with buf = malloc(len ? len : 1);
- deleted 3mo ago[deleted]
- kazinator 3mo agoOr even: if (len == 0) return null_buffer_singleton; /* special shared representation for 0: */
- repiret 3mo agoBecause then the caller needs to treat it differently from non-empty buffers.
- kazinator 3mo agoNo, it doesn't. It calls: void buffer_free(void *buf) { if (buf != null_buffer_singleton) free(buf); }
- Joker_vD 3mo ago...which treats it differently from non-empty buffers :) But frankly, having a 1-byte buffer, pointer to which can serve as a sentinel value à la NULL (but dereferenceable!), and which you can pass to free() without it being deallocated is indeed rather useful.
- weinzierl 3mo agoMaking the thing that describes the bounds of an arbitrary length thing itself arbitrary length sound like an unnecessarily risky complication to me. Especially since it only grows with the log of the thing it bounds. So, we could easily have s fixed length length field that covers all ever possible length values.
- kazinator 3mo agoOr you could just have the standard have a "Conformance Limits" section where it says something similar to "Every implementation shall support a length field value of at least 9223372036854775807, with an unlimited number of leading zeros". In other words, values beyond that are not forbidden by the spec, but are outside of the conformance requirements. It is a quality of implementation issue whether a given implementation handles more than required.
- weinzierl 3mo agoThat the spec is lenient and the conformance test strict makes ot only worse.
- repiret 3mo agoIt's mathematically impossible to describe an arbitrary length in a fixed length.
- weinzierl 3mo agoLuckily in math we do not have to allocate memory and worry about buffer overflows.
- account42 3mo agoI don't like formats that look like text buy may actually contain binary data - that's only going to tempt implementations that will choke when the string actually contains arbitrary data. Would be safer to encode the length and/or separator as something more obviously binary, which will also make the thing easier to parse in low-level implementations.
- GoblinSlayer 3mo agoFor this I made a tlv format, where string is encoded in byte prefixed chunks with length between 0 and 222 for final chunk and between 223 and 255 for intermediate chunks.
- dayjah 3mo agoI’ve near infinite respect for DJB, so I’m assuming I’ve missed something obvious here. Why is ‘,’ being used as a termination byte. Is it just a backstop? If buf[len+1] != ‘,’ then there’s a line error? I’ve done plenty of wire protocol work, and length prefixed strings are great to work with. I’ve also been lucky that those strings were typically contained within a broader payload. To that end, I’ve not had to think about the case of many strings one after another.
- kazinator 3mo agoThis is very similar to a notation occuring in Ronald Rivest's S-Expression proposal from 1996. https://people.csail.mit.edu/rivest/pubs/RL96.ver-1.0.pdf https://people.csail.mit.edu/rivest/pubs/RL96.ver-1.0.pdf He has it as a hexadecimal length preceded by a pound sign (#), a colon, and the raw octet data.
- kazinator 3mo agoRight, in later drafts he changed to decimal with no # prefix: https://www.ietf.org/archive/id/draft-rivest-sexp-04.html https://www.ietf.org/archive/id/draft-rivest-sexp-04.html
- burnt-resistor 3mo agoAnother string length O(1) encoding format like Pascal strings that were len (unsigned byte) + data. Limited to 255 characters however. Dollar, NUL, and other terminated strings, by contrast, are string length O(N).
- cowboylowrez 3mo agofast cgi has a good one, length then binary follows, if the length is 127 or less, the length is contained within one byte, if not the length is contained in 4 bytes, then the data follows. Midi has a similar scheme for representing some numbers, the high order bit denotes if there's another byte of the number following or something like that lol
- repiret 3mo agostrlen for netstring is O(log n).
- johnea 3mo agoSo much of the terse, utilitarian, DJB Way was overpowered by bloated corporate crap 8-/ One still functional example exists in voidlinux's init system. There was never any need for systemd 8-/ Except leveraging RedHat/IBM's domination of the linux user space ecosystem...