8 ms·

The internet was designed with a narrow waist

109 points by crawshaw 5y ago

skybrian 5y ago
This works well for transport and storage, but not so well if you need to understand and transform the data. For the problems with ad-hoc extension mechanisms, see: On Variance and Extensibility <https://acko.net/blog/on-variance-and-extensibility/ https://acko.net/blog/on-variance-and-extensibility/> In particular, the section on "The Mirage of Extensibility", which uses ImageMagick as an example. > One solution is to do both. You declare a standard schema upfront, with common conventions that can be relied on by anyone. But you also provide the ability to extend it with custom data, so that specific pairs of Input/Process/Output can coordinate. HTTP and e-mail headers are X-able. > The problem is that there is no universal reason why something should be standard or not. Standard is the common set of functionality "we" are aware of today. Non-standard is what's unanticipated. This is entirely haphazard. [...] > X- fields and headers are the norm and have a habit of becoming defacto standards. When they do, you find it's too late to clean it up into a new standard. People try anyway, like with X-Forwarded-For vs Forwarded. Or -webkit-transform vs transform. New software must continue to accept old input. It must also produce output compatible with old software. This means old software never needs to be updated, which means new software can never ditch its legacy code. Also see: IPv6. IP packets being so universal makes upgrading very difficult.
- chubot 5y ago
  (author here) Hm very interesting, I hadn't seen this article! I would say it's highly related, but doesn't contradict anything I'm saying. His examples are at a higher application level, in the domains of images and animation. I'm more focused the system level, like the basics of building code, distributing it, dynamically composing it, monitoring it, etc. There's definitely less need for types and schemas at that level, and more need for computation in his application domains. (I guess in networking terms this is a "smart middlebox" problem, as opposed to passive ones.) The commonality is that they are trying to avoid combinatorial explosions of code by having a shared representation, e.g. avoiding writing the same transformations for both PDF and JPG. I'd also say that he is overly negative on a few fronts -- I agree with a lot of it, but I'd frame the evolution as a success, not a failure. I guess my point the narrow waist is basically the only way we know how to build (really evolve) large scale systems over large time frames. We can do it well or do it badly, explicitly or implicitly... From the code and systems I've worked with, we could easily do a lot better. But I appreciate this article because identifying these kinds of problems is the first step. ---- Some random responses: Putting data inside a free-form key/value map doesn't change things much. It's barely an improvement over having a unknownData byte[] mix-in on each native type. It only pays off if you actually adopt a decomposable model and stick with it. That way the data is not unknown, but always provides a serviceable view on its own. Arguably this is the killer feature of a dynamic language. The benefit of "extensible data" is mainly "fully introspectable without recompilation." This part feels overly negative ... I would just frame this as a tradeoff between static and dynamic. There could be reasons that dynamic doesn't work in his domain, but that doesn't mean it doesn't work elsewhere. I have a bunch of material in drafts about types and distributed systems, sort of related to this "Maybe Not" discussion: https://lobste.rs/s/zdvg9y/maybe_not_rich_hickey https://lobste.rs/s/zdvg9y/maybe_not_rich_hickey (i.e. schemas/shapes should be decoupled from field presence; one is reusable and global and the other is local) From what I know Clojure has some good framings of the versioning problem, with Spec and RDF inspired schemas, but I haven't used it. Rather than the brittle versioning model, Hickey frames software evolution as "strengthen a promise" and "relax a guarantee". I don't know if it would be useful in those domains but it's worth addressing. I would be interested in unpacking the problem differing PDF and JPG metadata a little more... Off hand I'm not sure why it's so difficult; I feel like it's mostly a problem in certain statically-typed systems. I guess to put a big stake in the ground, you could say extensibility is inherently dynamic. Simply because you don't know what code is going to operate on your data 10 years from now. It hasn't been written yet, and the people who need it haven't been born yet. It's impossible in reality, not just in the type system :) I think the difference is his domains are living with the confines of a single machine and single application, so there is the expectation that you should be able to do better (have more type safety). While nobody expects to be able to reboot the Internet all at once and upgrade it. And even say at Google nobody wants to recompile all the code in a cluster at once and reboot it.
  - skybrian 5y ago
    Yes, I don’t agree with everything in the article, but it points out some sticky problems that can make things complicated. One example that’s related to programming languages is the evolution of an abstract syntax tree. We often expect new compilers to work with old code when a new kind of AST node is added, but to what extent should existing tools work with new code that has a node type that it doesn’t understand? Also, if your AST datatype is a library, does adding a new node type immediately cause compiler errors in every tool that uses the AST, or do those tools compile fine and break at runtime when given new code? Sometimes as a workaround, “metadata” gets put in specially formatted comments, as a way of saying that most language tools can ignore it. Alternately there can be general-purpose annotations (as Java has). Also, often an intermediate representation gets used as a “narrow waist” but they are tricky to design because “lowering” code will often discard high-level constructs.
    - chubot 5y ago
      Yes, the AST problem is one I've had a lot! It's hard. (The whole Oil language is defined as a big ML-style statically typed data structure, but it's not exposed: https://www.oilshell.org/release/0.9.8/source-code.wwz/frontend/syntax.asdl https://www.oilshell.org/release/0.9.8/source-code.wwz/front...) What I'd say is that we already have a narrow waist for every language -- text. That is the one that's stable! Having an additional stable AST format seems to be an insurmountable problem, especially in most statically typed languages. The basic reason is that it's easier to evolve text in a backward compatible way than types. This is probably obvious to some people, but controversial to others... It has a lot to do with the "ignoring new data" problem discussed in the blog post. It would be nice to unpack this a bit and find some references. ---- Related: I recently circulated this language implementation, which has a front end in OCaml, a backend in C++, and the IR is transmitted via protobuf. https://old.reddit.com/r/ProgrammingLanguages/comments/ss3w6n/an_accessible_introduction_to_type_theory_and/ https://old.reddit.com/r/ProgrammingLanguages/comments/ss3w6... Protobuf has the "runtime field presence" feature, which would be useful for making a backward compatible AST/ IR. That isn't done here, but it would be something to explore. ----- Also, thinking about the blog post a bit more, I'd say a huge number of decisions hinge on whether you can recompile the entire program or not. If you can, you should. But if you can't, a lot of programmers seem to have a hard time with runtime composition. Sometimes they even deny that it exists or should exist! In contrast, whereas admins/operators/SREs deal with runtime composition almost exclusively. Programmers are more familiar with a static view and build time composition. If I wanted to be ambitious, what I am getting at with this series is a theory of and guidelines for runtime composition, and versionless evolution. e.g. even in Windows, you never recompile your COM components together, even though they are written in statically typed languages. Global builds and atomic upgrades don't scale, even on a single computer.
jrm4 5y ago
As someone who teaches basic programming type things, and enjoys hacking away at things solo, I really like this reframing and reconsideration of that good ol' "Unix Way" type of thinking, and why I'm a huge fan of Bash for everything, and it makes me wonder why it isn't more "text everywhere." In the back of my head I don't think I'll ever be convinced that those reasons aren't mostly arbitrary. (i.e. language/tool developers for other developers subconsciously fomenting lock-in)
gwern 5y ago
https://en.wikipedia.org/wiki/End-to-end_principle https://en.wikipedia.org/wiki/End-to-end_principle
- thaumasiotes 5y ago
  I think the author is going for more of https://en.wikipedia.org/wiki/Star_topology https://en.wikipedia.org/wiki/Star_topology, just with the special case added that you're simplifying a bipartite graph rather than a general graph.
- chubot 5y ago
  (author here) How does that relate to the article? It's definitely an important principle of network design, but the narrow waist principle is distinct. The waist isn't the middle of the network as far as connectivity; it's the architectural middle with applications on top and transports on bottom. I think these are mostly separate issues, but I'm interested in any elaboration.
  - gwern 5y ago
    The narrow waist falls out of the end to end principle because when you push all the intelligence out to the ends, there is often only one reasonable intermediate/substrate; and all narrow waists in your article, whether TCP/IP or text formats, are driven by end-to-end concerns. The narrow waist is the special-case of the more important principle. Anyone who thinks about these issues will be thinking about end to end, so at the very least, you should discuss why you think narrow waist is more fundamental and important than end to end. BTW, all NAPs are available online as a federal government body, all you had to do is google the title, no need to buy it: https://www.nap.edu/read/4755/chapter/1 https://www.nap.edu/read/4755/chapter/1 While we're linking things, 'commoditize your complement' https://www.gwern.net/Complement https://www.gwern.net/Complement is very relevant to web browser dynamics, and 'bitrot'/'bitcreep' are relevant to upgrading systems and choice of standards https://www.gwern.net/Holy-wars https://www.gwern.net/Holy-wars .
    - chubot 5y ago
mypalmike 5y ago
Interesting article. There's a typo - "Instructure Set Architectures". The fingers just want to type the word "structure" I guess.
- chubot 5y ago
  Thanks, fixed!
hlandau 5y ago
C APIs like ioctl, setsockopt, etc. are perhaps also an example of narrow waists which aren't mentioned in the article. I've been looking for a term for these kinds of functions, which seem to be quite successful long-term in terms of API design.
- kevin_thibedeau 5y ago
  You can take it to an extreme like with XView which has a half dozen variadic functions for an entire GUI API.
- chubot 5y ago
  Yes, file descriptors and whole file systems are other narrow waists that Unix has, not just files themselves! I want to make diagrams for those too. The terminology I'd use: File descriptors are just a way to do polymorphism in the type system of C. Basically you fall back to an "untyped" integer (similar to void*). Go's interfaces solve this problem pretty directly, much better than C++, which can't really express it in the type system. That is, with the "-er" interface pattern like Reader and Writer. (I think type classes do better too, but I haven't used them much. I'd like to see an article that unpacks this better in the context of file descriptor APIs) This style relates to "versionless evolution". The key issue is that Go doesn't require you to declare that a type supports an interface. It's implicit and the compiler figures it out. That was controversial, but it is the key to enabling evolution. Most programmers are viewing things through the lense of "strictness" and "catching mistakes", rather than evolution. ioctl() is messy as heck but has allowed a lot of evolution. It's highly related to this discussion, which I've linked on the blog before: https://lobste.rs/s/izpz2n/on_missed_opportunities_static_types https://lobste.rs/s/izpz2n/on_missed_opportunities_static_ty... again linked from the main page: https://github.com/oilshell/oil/wiki/Perlis-Thompson-Principle https://github.com/oilshell/oil/wiki/Perlis-Thompson-Princip...
edent 5y ago
It's interesting to apply this at various levels of abstraction. Almost all raster image editors work on a grid of RGB(A) pixels. Whether it's PNG, JPG, WebP or whatever - they're all deflated to a pixel grid. Wireless technology strikes me as the opposite of the narrow waist. Sure, it is all just variants of basic radio tech. But Bluetooth audio and WiFi are fundamentally incompatible - despite using more-or-less the same spectrum. 2G phones can't send or receive 5G. What a great article. Good food for thought.
- chubot 5y ago
  Glad you liked the article! From a user perspective, bluetooth does seem to be a outlier ... it would be interesting to unpack the reasons. This post linked by skybrian is good, and discusses a sort of narrow waist for image transformations (and animation library linked at the end) https://acko.net/blog/on-variance-and-extensibility/ https://acko.net/blog/on-variance-and-extensibility/ My comments say where I disagree, but it's a good article obviously based on a lot of experience.
- CyberRabbi 5y ago
  Most wireless transmission protocols use generic modulation techniques that in theory can be put together using a software package like GNU Radio. In theory GNU Radio could be the narrow waist for all wireless transmission protocols. It isn’t practical yet but it’s a possibility. Instead of rewriting a new wireless stack each time we just use GNU radio and specify our specific signal blocks. The analogy is similar to IR being the compiler’s narrow waist.
- teleforce 5y ago
  Actually wireless is the classic example of the narrow waist as suggested by the sibling comments but it's not because of GNU Radio but I/Q signal. All wireless communication are based on EM waves and can be reduced to I/Q or quadrature signal [1]. But rather than design by human as IP or text mentioned by the OP it is found in nature similar to our chromosomes pair. Using this wonderful I/Q signal concept you can even digitally create and control not only digital modulations but all the conventional analog modulations for examples AM, FM and SSB. If you want you can even control the EM signal impedance itself by using the I/Q signals and in microwave measurement world this modern technique is known as digital load-pull or source-pull. The interoperability problems that you've observed is due to the Bluetooth and WiFi are using different modulation techniques. Bluetooth is using the Frequency Hopping Spread Spectrum (FHSS) and WiFi is using OFDM. The former is good for low power and the latter is good for bandwidth. Initially WiFi standard also support FHSS but due to its bandwidth limitation the later WiFi standards (and also 4G/5G) are based entirely on OFDM. [1]I/Q Data for Dummies: https://news.ycombinator.com/item?id=25134698 https://news.ycombinator.com/item?id=25134698
quickthrower2 5y ago
The bridge design pattern!
zokier 5y ago
Would sexprs also fall into this narrow waist category?
- chubot 5y ago
  Yes but only when you're programming in Lisp (which can obviously be extremely useful and productive). I want to get to this on the blog, but the narrow waist idea is actually hierarchical, with bytes at "level 0", and things like s-exprs and JSON at "level 1". Building up this claim, I'd say ALL good languages are defined by their core data structures. See my comment (oilshell) here: https://old.reddit.com/r/ProgrammingLanguages/comments/lliyuo/are_there_any_interesting_programming_languages/ https://old.reddit.com/r/ProgrammingLanguages/comments/lliyu... e.g. Lisp is defined by s-exprs, C by pointers and offsets, shell and Tcl by strings, etc. The lowest common denominator between all these "meta-models" languages is BYTES. S-expressions are expressed as bytes. In Python and JavaScript, you serialize to JSON, etc. ---- To support this point, I have a slogan that "XML was a failed narrow waist". It was designed for documents, but people tried to express everything in it -- records, tables, and even programs. But that was a bad idea! JSON took over some of that space since it's simply more appropriate. But JSON isn't good enough for documents, so we use HTML. Which means the lowest common denominator is again text. (That discussion appears here: https://github.com/oilshell/oil/wiki/Perlis-Thompson-Principle https://github.com/oilshell/oil/wiki/Perlis-Thompson-Princip...)
ignoramous 5y ago
Ha! Just a few months ago, I came across Cloudflare Research's Address Agility blog post (since published in SIGCOMM [0]) that squeezed Internet's waist down to a single IP (dubbed Ao1), and instead reasoned that perhaps with popularity of TLS, a naming scheme (like the SNI field) is all that's needed, because given the pigeon-hole principle, there will always be more names (variable length puny-codes) than IP addresses (fixed-length hex/octets). Telecom industry perhaps deployed something similar in Multi-protocol Label Switching. Other address translation schemes such as the HTTP Host header, and Google Maglev-esque Virtual IP load balancers (in-use at BigCloud providers) are other attempts at addressing hosts using something other than public IPs (which enabled domain-fronting, once employed by Signal before it was outlawed [1]), squeezing Internet's narrow waist further. I am mostly interested in Ao1 from an anti-censorship point-of-view (IP fronting) as that's the last frontier as far as stateless firewalls go, while names, DNS and SNI, can be hidden away today with DoH/DoT and ECH (encrypted client-hello), respectively. That said, I like the term Simplfying Assumption better as described in a series of posts on it by apenwarr [2][3] while the classic The Rise of Worse is Better pits C against Lisp, arguing that sacrificing correctness, consistency, completeness for the sake of simplicity of the implementation, is a worthwhile trade-off [4]. [0] https://research.cloudflare.com/publications/Fayed2021/ https://research.cloudflare.com/publications/Fayed2021/ [1] Amazon threatens to suspend Signal (2018), https://news.ycombinator.com/item?id=16970199 https://news.ycombinator.com/item?id=16970199 [2] The strange story of etherpad (2011), https://apenwarr.ca/log/20110313 https://apenwarr.ca/log/20110313 [3] Conspiracy theories (2004), https://apenwarr.ca/log/200411 https://apenwarr.ca/log/200411 [4] https://dreamsongs.com/RiseOfWorseIsBetter.html https://dreamsongs.com/RiseOfWorseIsBetter.html
kevincox 5y ago
I like the blog but they really need to set up a feed. They recommend subscribing on Reddit but that also has a lot of non-blog content.
- chubot 5y ago
  You can also use Twitter: https://twitter.com/oilshellblog https://twitter.com/oilshellblog Reddit and twitter both have some non-blog content, but extremely little. This isn't likely to be fixed because 1) I need to fix other things about the blog infrastructure first 2) I view it as a feature because I publish the posts first, then view them on a few devices, edit them, and then publish to reddit, twitter, etc. i.e. I don't want people to view it the second it's published In other words, it's surprisingly difficult to have a productive workflow for writing such articles, and I've mostly settled into it :)
  - kevincox 5y ago
    Twitter doesn't have an RSS/Atom feed :( I get that if you want to have a specific release schedule it could be annoying, but why not add it to the feed the same time you post to twitter?
    - dredmorbius 5y ago
      Nitter offers that. Find and follow an instance. At any given user, there's a feed link. Add that to your RSS reader.
bunbunbun 5y ago
Is it just me or does ‘narrow waist’ feel unnecessarily gendered when the name of that so-called waist is actually ‘neck’?
- CyberRabbi 5y ago
  Any person regardless of gender can have a narrow waist. A waist is not a gendered body part. https://en.wiktionary.org/wiki/waist https://en.wiktionary.org/wiki/waist
- oversocialized 5y ago
oversocialized 5y ago

109 points by crawshaw 5y ago

This works well for transport and storage, but not so well if you need to understand and transform the data. For the problems with ad-hoc extension mechanisms, see: On Variance and Extensibility <https://acko.net/blog/on-variance-and-extensibility/ https://acko.net/blog/on-variance-and-extensibility/> In particular, the section on "The Mirage of Extensibility", which uses ImageMagick as an example. > One solution is to do both. You declare a standard schema upfront, with common conventions that can be relied on by anyone. But you also provide the ability to extend it with custom data, so that specific pairs of Input/Process/Output can coordinate. HTTP and e-mail headers are X-able. > The problem is that there is no universal reason why something should be standard or not. Standard is the common set of functionality "we" are aware of today. Non-standard is what's unanticipated. This is entirely haphazard. [...] > X- fields and headers are the norm and have a habit of becoming defacto standards. When they do, you find it's too late to clean it up into a new standard. People try anyway, like with X-Forwarded-For vs Forwarded. Or -webkit-transform vs transform. New software must continue to accept old input. It must also produce output compatible with old software. This means old software never needs to be updated, which means new software can never ditch its legacy code. Also see: IPv6. IP packets being so universal makes upgrading very difficult.

chubot 5y ago
(author here) Hm very interesting, I hadn't seen this article! I would say it's highly related, but doesn't contradict anything I'm saying. His examples are at a higher application level, in the domains of images and animation. I'm more focused the system level, like the basics of building code, distributing it, dynamically composing it, monitoring it, etc. There's definitely less need for types and schemas at that level, and more need for computation in his application domains. (I guess in networking terms this is a "smart middlebox" problem, as opposed to passive ones.) The commonality is that they are trying to avoid combinatorial explosions of code by having a shared representation, e.g. avoiding writing the same transformations for both PDF and JPG. I'd also say that he is overly negative on a few fronts -- I agree with a lot of it, but I'd frame the evolution as a success, not a failure. I guess my point the narrow waist is basically the only way we know how to build (really evolve) large scale systems over large time frames. We can do it well or do it badly, explicitly or implicitly... From the code and systems I've worked with, we could easily do a lot better. But I appreciate this article because identifying these kinds of problems is the first step. ---- Some random responses: Putting data inside a free-form key/value map doesn't change things much. It's barely an improvement over having a unknownData byte[] mix-in on each native type. It only pays off if you actually adopt a decomposable model and stick with it. That way the data is not unknown, but always provides a serviceable view on its own. Arguably this is the killer feature of a dynamic language. The benefit of "extensible data" is mainly "fully introspectable without recompilation." This part feels overly negative ... I would just frame this as a tradeoff between static and dynamic. There could be reasons that dynamic doesn't work in his domain, but that doesn't mean it doesn't work elsewhere. I have a bunch of material in drafts about types and distributed systems, sort of related to this "Maybe Not" discussion: https://lobste.rs/s/zdvg9y/maybe_not_rich_hickey https://lobste.rs/s/zdvg9y/maybe_not_rich_hickey (i.e. schemas/shapes should be decoupled from field presence; one is reusable and global and the other is local) From what I know Clojure has some good framings of the versioning problem, with Spec and RDF inspired schemas, but I haven't used it. Rather than the brittle versioning model, Hickey frames software evolution as "strengthen a promise" and "relax a guarantee". I don't know if it would be useful in those domains but it's worth addressing. I would be interested in unpacking the problem differing PDF and JPG metadata a little more... Off hand I'm not sure why it's so difficult; I feel like it's mostly a problem in certain statically-typed systems. I guess to put a big stake in the ground, you could say extensibility is inherently dynamic. Simply because you don't know what code is going to operate on your data 10 years from now. It hasn't been written yet, and the people who need it haven't been born yet. It's impossible in reality, not just in the type system :) I think the difference is his domains are living with the confines of a single machine and single application, so there is the expectation that you should be able to do better (have more type safety). While nobody expects to be able to reboot the Internet all at once and upgrade it. And even say at Google nobody wants to recompile all the code in a cluster at once and reboot it.
- skybrian 5y ago
  Yes, I don’t agree with everything in the article, but it points out some sticky problems that can make things complicated. One example that’s related to programming languages is the evolution of an abstract syntax tree. We often expect new compilers to work with old code when a new kind of AST node is added, but to what extent should existing tools work with new code that has a node type that it doesn’t understand? Also, if your AST datatype is a library, does adding a new node type immediately cause compiler errors in every tool that uses the AST, or do those tools compile fine and break at runtime when given new code? Sometimes as a workaround, “metadata” gets put in specially formatted comments, as a way of saying that most language tools can ignore it. Alternately there can be general-purpose annotations (as Java has). Also, often an intermediate representation gets used as a “narrow waist” but they are tricky to design because “lowering” code will often discard high-level constructs.
  - chubot 5y ago
    Yes, the AST problem is one I've had a lot! It's hard. (The whole Oil language is defined as a big ML-style statically typed data structure, but it's not exposed: https://www.oilshell.org/release/0.9.8/source-code.wwz/frontend/syntax.asdl https://www.oilshell.org/release/0.9.8/source-code.wwz/front...) What I'd say is that we already have a narrow waist for every language -- text. That is the one that's stable! Having an additional stable AST format seems to be an insurmountable problem, especially in most statically typed languages. The basic reason is that it's easier to evolve text in a backward compatible way than types. This is probably obvious to some people, but controversial to others... It has a lot to do with the "ignoring new data" problem discussed in the blog post. It would be nice to unpack this a bit and find some references. ---- Related: I recently circulated this language implementation, which has a front end in OCaml, a backend in C++, and the IR is transmitted via protobuf. https://old.reddit.com/r/ProgrammingLanguages/comments/ss3w6n/an_accessible_introduction_to_type_theory_and/ https://old.reddit.com/r/ProgrammingLanguages/comments/ss3w6... Protobuf has the "runtime field presence" feature, which would be useful for making a backward compatible AST/ IR. That isn't done here, but it would be something to explore. ----- Also, thinking about the blog post a bit more, I'd say a huge number of decisions hinge on whether you can recompile the entire program or not. If you can, you should. But if you can't, a lot of programmers seem to have a hard time with runtime composition. Sometimes they even deny that it exists or should exist! In contrast, whereas admins/operators/SREs deal with runtime composition almost exclusively. Programmers are more familiar with a static view and build time composition. If I wanted to be ambitious, what I am getting at with this series is a theory of and guidelines for runtime composition, and versionless evolution. e.g. even in Windows, you never recompile your COM components together, even though they are written in statically typed languages. Global builds and atomic upgrades don't scale, even on a single computer.

jrm4 5y ago

As someone who teaches basic programming type things, and enjoys hacking away at things solo, I really like this reframing and reconsideration of that good ol' "Unix Way" type of thinking, and why I'm a huge fan of Bash for everything, and it makes me wonder why it isn't more "text everywhere." In the back of my head I don't think I'll ever be convinced that those reasons aren't mostly arbitrary. (i.e. language/tool developers for other developers subconsciously fomenting lock-in)

gwern 5y ago

https://en.wikipedia.org/wiki/End-to-end_principle https://en.wikipedia.org/wiki/End-to-end_principle

thaumasiotes 5y ago
I think the author is going for more of https://en.wikipedia.org/wiki/Star_topology https://en.wikipedia.org/wiki/Star_topology, just with the special case added that you're simplifying a bipartite graph rather than a general graph.
chubot 5y ago
(author here) How does that relate to the article? It's definitely an important principle of network design, but the narrow waist principle is distinct. The waist isn't the middle of the network as far as connectivity; it's the architectural middle with applications on top and transports on bottom. I think these are mostly separate issues, but I'm interested in any elaboration.
- gwern 5y ago
  The narrow waist falls out of the end to end principle because when you push all the intelligence out to the ends, there is often only one reasonable intermediate/substrate; and all narrow waists in your article, whether TCP/IP or text formats, are driven by end-to-end concerns. The narrow waist is the special-case of the more important principle. Anyone who thinks about these issues will be thinking about end to end, so at the very least, you should discuss why you think narrow waist is more fundamental and important than end to end. BTW, all NAPs are available online as a federal government body, all you had to do is google the title, no need to buy it: https://www.nap.edu/read/4755/chapter/1 https://www.nap.edu/read/4755/chapter/1 While we're linking things, 'commoditize your complement' https://www.gwern.net/Complement https://www.gwern.net/Complement is very relevant to web browser dynamics, and 'bitrot'/'bitcreep' are relevant to upgrading systems and choice of standards https://www.gwern.net/Holy-wars https://www.gwern.net/Holy-wars .
  - chubot 5y ago

mypalmike 5y ago

Interesting article. There's a typo - "Instructure Set Architectures". The fingers just want to type the word "structure" I guess.

chubot 5y ago
Thanks, fixed!

hlandau 5y ago

C APIs like ioctl, setsockopt, etc. are perhaps also an example of narrow waists which aren't mentioned in the article. I've been looking for a term for these kinds of functions, which seem to be quite successful long-term in terms of API design.

kevin_thibedeau 5y ago
You can take it to an extreme like with XView which has a half dozen variadic functions for an entire GUI API.
chubot 5y ago
Yes, file descriptors and whole file systems are other narrow waists that Unix has, not just files themselves! I want to make diagrams for those too. The terminology I'd use: File descriptors are just a way to do polymorphism in the type system of C. Basically you fall back to an "untyped" integer (similar to void*). Go's interfaces solve this problem pretty directly, much better than C++, which can't really express it in the type system. That is, with the "-er" interface pattern like Reader and Writer. (I think type classes do better too, but I haven't used them much. I'd like to see an article that unpacks this better in the context of file descriptor APIs) This style relates to "versionless evolution". The key issue is that Go doesn't require you to declare that a type supports an interface. It's implicit and the compiler figures it out. That was controversial, but it is the key to enabling evolution. Most programmers are viewing things through the lense of "strictness" and "catching mistakes", rather than evolution. ioctl() is messy as heck but has allowed a lot of evolution. It's highly related to this discussion, which I've linked on the blog before: https://lobste.rs/s/izpz2n/on_missed_opportunities_static_types https://lobste.rs/s/izpz2n/on_missed_opportunities_static_ty... again linked from the main page: https://github.com/oilshell/oil/wiki/Perlis-Thompson-Principle https://github.com/oilshell/oil/wiki/Perlis-Thompson-Princip...

edent 5y ago

It's interesting to apply this at various levels of abstraction. Almost all raster image editors work on a grid of RGB(A) pixels. Whether it's PNG, JPG, WebP or whatever - they're all deflated to a pixel grid. Wireless technology strikes me as the opposite of the narrow waist. Sure, it is all just variants of basic radio tech. But Bluetooth audio and WiFi are fundamentally incompatible - despite using more-or-less the same spectrum. 2G phones can't send or receive 5G. What a great article. Good food for thought.

chubot 5y ago
Glad you liked the article! From a user perspective, bluetooth does seem to be a outlier ... it would be interesting to unpack the reasons. This post linked by skybrian is good, and discusses a sort of narrow waist for image transformations (and animation library linked at the end) https://acko.net/blog/on-variance-and-extensibility/ https://acko.net/blog/on-variance-and-extensibility/ My comments say where I disagree, but it's a good article obviously based on a lot of experience.
CyberRabbi 5y ago
Most wireless transmission protocols use generic modulation techniques that in theory can be put together using a software package like GNU Radio. In theory GNU Radio could be the narrow waist for all wireless transmission protocols. It isn’t practical yet but it’s a possibility. Instead of rewriting a new wireless stack each time we just use GNU radio and specify our specific signal blocks. The analogy is similar to IR being the compiler’s narrow waist.
teleforce 5y ago
Actually wireless is the classic example of the narrow waist as suggested by the sibling comments but it's not because of GNU Radio but I/Q signal. All wireless communication are based on EM waves and can be reduced to I/Q or quadrature signal [1]. But rather than design by human as IP or text mentioned by the OP it is found in nature similar to our chromosomes pair. Using this wonderful I/Q signal concept you can even digitally create and control not only digital modulations but all the conventional analog modulations for examples AM, FM and SSB. If you want you can even control the EM signal impedance itself by using the I/Q signals and in microwave measurement world this modern technique is known as digital load-pull or source-pull. The interoperability problems that you've observed is due to the Bluetooth and WiFi are using different modulation techniques. Bluetooth is using the Frequency Hopping Spread Spectrum (FHSS) and WiFi is using OFDM. The former is good for low power and the latter is good for bandwidth. Initially WiFi standard also support FHSS but due to its bandwidth limitation the later WiFi standards (and also 4G/5G) are based entirely on OFDM. [1]I/Q Data for Dummies: https://news.ycombinator.com/item?id=25134698 https://news.ycombinator.com/item?id=25134698

quickthrower2 5y ago

The bridge design pattern!

zokier 5y ago

Would sexprs also fall into this narrow waist category?

chubot 5y ago
Yes but only when you're programming in Lisp (which can obviously be extremely useful and productive). I want to get to this on the blog, but the narrow waist idea is actually hierarchical, with bytes at "level 0", and things like s-exprs and JSON at "level 1". Building up this claim, I'd say ALL good languages are defined by their core data structures. See my comment (oilshell) here: https://old.reddit.com/r/ProgrammingLanguages/comments/lliyuo/are_there_any_interesting_programming_languages/ https://old.reddit.com/r/ProgrammingLanguages/comments/lliyu... e.g. Lisp is defined by s-exprs, C by pointers and offsets, shell and Tcl by strings, etc. The lowest common denominator between all these "meta-models" languages is BYTES. S-expressions are expressed as bytes. In Python and JavaScript, you serialize to JSON, etc. ---- To support this point, I have a slogan that "XML was a failed narrow waist". It was designed for documents, but people tried to express everything in it -- records, tables, and even programs. But that was a bad idea! JSON took over some of that space since it's simply more appropriate. But JSON isn't good enough for documents, so we use HTML. Which means the lowest common denominator is again text. (That discussion appears here: https://github.com/oilshell/oil/wiki/Perlis-Thompson-Principle https://github.com/oilshell/oil/wiki/Perlis-Thompson-Princip...)

ignoramous 5y ago

Ha! Just a few months ago, I came across Cloudflare Research's Address Agility blog post (since published in SIGCOMM [0]) that squeezed Internet's waist down to a single IP (dubbed Ao1), and instead reasoned that perhaps with popularity of TLS, a naming scheme (like the SNI field) is all that's needed, because given the pigeon-hole principle, there will always be more names (variable length puny-codes) than IP addresses (fixed-length hex/octets). Telecom industry perhaps deployed something similar in Multi-protocol Label Switching. Other address translation schemes such as the HTTP Host header, and Google Maglev-esque Virtual IP load balancers (in-use at BigCloud providers) are other attempts at addressing hosts using something other than public IPs (which enabled domain-fronting, once employed by Signal before it was outlawed [1]), squeezing Internet's narrow waist further. I am mostly interested in Ao1 from an anti-censorship point-of-view (IP fronting) as that's the last frontier as far as stateless firewalls go, while names, DNS and SNI, can be hidden away today with DoH/DoT and ECH (encrypted client-hello), respectively. That said, I like the term Simplfying Assumption better as described in a series of posts on it by apenwarr [2][3] while the classic The Rise of Worse is Better pits C against Lisp, arguing that sacrificing correctness, consistency, completeness for the sake of simplicity of the implementation, is a worthwhile trade-off [4]. [0] https://research.cloudflare.com/publications/Fayed2021/ https://research.cloudflare.com/publications/Fayed2021/ [1] Amazon threatens to suspend Signal (2018), https://news.ycombinator.com/item?id=16970199 https://news.ycombinator.com/item?id=16970199 [2] The strange story of etherpad (2011), https://apenwarr.ca/log/20110313 https://apenwarr.ca/log/20110313 [3] Conspiracy theories (2004), https://apenwarr.ca/log/200411 https://apenwarr.ca/log/200411 [4] https://dreamsongs.com/RiseOfWorseIsBetter.html https://dreamsongs.com/RiseOfWorseIsBetter.html

kevincox 5y ago

I like the blog but they really need to set up a feed. They recommend subscribing on Reddit but that also has a lot of non-blog content.

chubot 5y ago
You can also use Twitter: https://twitter.com/oilshellblog https://twitter.com/oilshellblog Reddit and twitter both have some non-blog content, but extremely little. This isn't likely to be fixed because 1) I need to fix other things about the blog infrastructure first 2) I view it as a feature because I publish the posts first, then view them on a few devices, edit them, and then publish to reddit, twitter, etc. i.e. I don't want people to view it the second it's published In other words, it's surprisingly difficult to have a productive workflow for writing such articles, and I've mostly settled into it :)
- kevincox 5y ago
  Twitter doesn't have an RSS/Atom feed :( I get that if you want to have a specific release schedule it could be annoying, but why not add it to the feed the same time you post to twitter?
  - dredmorbius 5y ago
    Nitter offers that. Find and follow an instance. At any given user, there's a feed link. Add that to your RSS reader.

bunbunbun 5y ago

Is it just me or does ‘narrow waist’ feel unnecessarily gendered when the name of that so-called waist is actually ‘neck’?

CyberRabbi 5y ago
Any person regardless of gender can have a narrow waist. A waist is not a gendered body part. https://en.wiktionary.org/wiki/waist https://en.wiktionary.org/wiki/waist
oversocialized 5y ago

oversocialized 5y ago