7 ms·

Don't tug on that, you never know what it might be attached to (2016)

143 points by todsacerdoti 11mo ago

markstos 11mo ago
And this was written 10 years ago, when computers were far less complicated and vibe coding sleeper bugs wasn't a thing.
- detourdog 11mo ago
  From my perspective vibe coding was always a thing.
- WJW 11mo ago
  Vibe coded sleeper bugs have always been a thing, they just came from the bosses' nephew who was still learning PHP at the time and left several years ago. Also, computers in 2015 were not meaningfully less complex than today. Certainly not when the topic is weird emacs and perl interactions.
  - marcosdumay 11mo ago
    Even if the topic was web applications (that are where Big Complexity thrives), 2015 was about peak complexity. Things have improved a bit since then.
  - add-sub-mul-div 11mo ago
    The problem isn't that AI is doing something new, we all know that it isn't. The problem is that the boss' nephew is becoming the rule now rather than the exception.
    - jama211 11mo ago
      It also makes bugs easier to find and resolve. You win some you lose some. Perhaps by the time it is the rule they’ll be better at writing safer code.
linsomniac 11mo ago
The Internet needs more Buckaroo Banzai references. Because wherever you go, there you are.
- neilk 11mo ago
  Yup. I nearly had this movie memorized when I was a child. https://www.youtube.com/watch?v=aWXuDNmO7j8 https://www.youtube.com/watch?v=aWXuDNmO7j8 Peter Weller, playing Buckaroo Banzai, is late for his military-particle-physics-interdimensional-jet-car test because he's helping Jeff Goldblum's character with neurosurgery. Later that day he will go play lead guitar in an ensemble. Scriptwriting gurus advise that your protagonist should have flaws and character progression. The writers of this movie disagree.
  - linsomniac 11mo ago
    Kevin Smith has an introduction to this movie where he calls it a true piece of art: "It doesn't care what you bring to the table, it bring itself to the table and says: figure it out". https://youtu.be/N8R8wmlggwc?si=sva2-jF1Kl5eFsU4 https://youtu.be/N8R8wmlggwc?si=sva2-jF1Kl5eFsU4
svat 11mo ago
(2016) Also, “direct” link: https://blog.plover.com/tech/tmpdir.html https://blog.plover.com/tech/tmpdir.html (This doesn't really matter, as the posted link is to https://blog.plover.com/2016/07/01/#tmpdir https://blog.plover.com/2016/07/01/#tmpdir i.e. the blog post named “tmpdir” posted on 2016-07-01 and there is only post posted on that date, so the content of the page is basically the same.)
adrianmonk 11mo ago
> This computer stuff is amazingly complicated. I don't know how anyone gets anything done. I wonder what could be done to make this type of problem less hidden and easier to diagnose. The one thing that comes to mind is to have the loader fail fast. For security reasons, the loader needs to ensure TMPDIR isn't set. Right now it accomplishes this by un-setting TMPDIR, which leads to silent failures. Instead, it could check if TMPDIR is set, and if so, give a fatal error. This would force you to unset TMPDIR yourself before you run a privileged program, which would be tedious, but at least you'd know it was happening because you'd be the one doing it. (To be clear, I'm not proposing actually doing this. It would break compatibility. It's just interesting to think about alternative designs.)
- tetha 11mo ago
  Mh, I am starting to dislike this kind of hyper-configurability. I know when this was necessary and used it myself quite a bit. But today, couldn't we just open up a mount namespace and bind-mount something else to /tmp, like SystemDs private tempdirs? (Which broke a lot of assumptions about tmpdirs and caused a bit of ruckus, but on the other hand, I see their point by now) I'm honestly starting to wonder about a lot of these really weird, prickly and fragile environment variables which cause security vulnerabilities, if low-overhead virtualization and namespacing/containers are available. This would also raise the security floor.
  - josephcsible 11mo ago
    > But today, couldn't we just open up a mount namespace and bind-mount something else to /tmp, like SystemDs private tempdirs? No, because unless you're already root (in which case you wouldn't have needed the binary with the capability in the first place), you can't make a mount namespace without also making a user namespace, and the counterproductive risk-averse craziness has led to removing unprivileged users' ability to make user namespaces.
    - kragen 11mo ago
      It's probably true that there are setuid programs that can be exploited if you run them in a user namespace. You probably need to remove setuid (and setgid) as Plan9 did in order to do this.
torben-friis 11mo ago
(Let me start clarifying that this is not at all a criticism of the author) I am usually amused by the way really competent people judge other's context. This post assumes understanding of: - emacs (what it is, and terminology like buffers) - strace - linux directories and "everything is a file" - environment variables - grep and similar - what git is - the fact that 'git whatever' works to run a custom script if git-whatever exists in the path (this one was a TIL for me!) - irc - CVEs - dynamic loaders - file priviledges but then feels important to explain to the audience that: >A socket is a facility that enables interprocess communication
- dwedge 11mo ago
  I found it interesting that they know how to use strace, but not how to list open files held by a process which to me seems simpler. Again, not criticism just an observation and I enjoyed the article
  - parliament32 11mo ago
    Given the "(hi Julia!)" immediately after the strace shenanigans, I interpreted this as a third-party hint; the author most likely had not used strace before. The author is both an example of and an example for how we can get caught in "bubbles" of tools/things we know and use and don't, and blog posts like this are great for discovery (I didn't know about git invoking a binary in the path like his "git re-edit", for example, until today).
    - dwedge 11mo ago
      I discovered that by accident, I had a script called git-pr that opened a pull request with github using the last commit message and then pushed it to slack for approval. I was trying to rewrite it to add a description and wondered why "git pr" pushed an empty message to slack
    - mjd 11mo ago
      I assure you that the author has been using strace since it was invented in the early 1990s.
deleted 11mo ago
[deleted]
jcynix 11mo ago
BTW, the author "mjd" is the author of the excellent book "Higher-Order Perl" which is available online at https://hop.perl.plover.com/book/ https://hop.perl.plover.com/book/
- pinkmuffinere 11mo ago
  I love mjd! He once replied to me on an HN thread and it lives forever in my memory :)
  - roywashere 11mo ago
    Me three! mjd fanboy here
  - mjd 11mo ago
    I LIKE PIE
    - deleted 11mo ago
      [deleted]
LordGrey 11mo ago
Buckaroo Banzai: You can check your anatomy all you want, and even though there may be normal variation, when it comes right down to it, this far inside the head it all looks the same. No, no, no, don’t tug on that. You never know what it might be attached to.
- buckaroobanzi 11mo ago
  See, this is the point, for me, where it started to look like a problem. You know, I wanted to sacrifice the precentral vein in order to get some exposure, but because of this guy's normal variation, I got excited, and all of a sudden I didn't know whether I was looking at the precentral vein, or one of the internal cerebral veins, or the vein of Galen, or the vascular vein of Rosenthal. So, on my own, to me, at this point, I was ready to say that's it, let's get out.
Scott-David 11mo ago
[dead]
thayne 11mo ago
Setting a capability on the perl executable seems like a very bad idea. That effectively grants tha capability to everything that is able to invoke perl (without being restricted to NO_NEW_PRIVILEGES).
- aftbit 11mo ago
  Yeah why did he want non-root perl to be able to bind to low-numbered ports? Seems like one of those typical footguns of applying non-standard configurations.
  - thayne 11mo ago
    My reading is the author didn't do that, rather his/her employers configuration system had done so. Setting TMPDIR to /mnt/tmp seems also to come from that. I would guess both were the result of someone who didn't really know what they were doing trying things until they found something that got what they needed to work, then pushed that out without understanding the broader implications.
lanthade 11mo ago
Don't tug on that applies to hardware too. Years ago I worked on contract for a large blue 3 letter company doing outsourced server management for the fancy credit card company. The incident in question happened before my time on the team but I heard about it first hand from the server admin (let's call him Ben) who had been at the center of it. The data center in question was (IIRC) 160K sqft of raised floor spread across multiple floors in a major metropolitan downtown area. It isn't there anymore. Windows, Unix, Linux, mainframe, San, all the associated fun stuff. Ben was working the day after thanksgiving decommissioning a system. Full software and physical decommission. Approved through all the proper change management procedures. As part of the decommission Ben removed the network cables from under raised floor. Standard snip the connector off and pull it back. Easy. Little did he know that network cable was ever so slightly entangled with another cable. Not enough to give him pause when pulling it though. It wouldn't have been an issue if the other cable had been properly latched in its ports. It wasn't. That little pull ended up pulling the network connection out of a completely unrelated system. A system managed by a completely different group. A system responsible for credit card processing. On USA Black Friday. Oops. CC processing went down. It took far too long to resolve. Amazingly Ben didnt loose his job. After all he followed all the processes and procedures. Kudos to the management team who kept him protected. Change management and change freezes were far more stringent by the time I joined the team. There was also now a raised floor infrastructure group and no one pulled a tile without their involvement. Be careful what you tug on!
klodolph 11mo ago
I have an overly reductive take on this—it’s Unix environment variables. You have your terminal window and your .bashrc (or equivalent), and that sets a bunch of environment variables. But your GUI runs with, most likely, different environment variables. It sucks. And here’s my controversial take on things—the “correct” resolution is to reify some higher-level concept of environment. Each process should not have its own separate copy of environment variables. Some things should be handled… ugh, I hate to say it… through RPC to some centralized system like systemd.
- rao-v 11mo ago
  Windows registry just sort of hovering in the backdrop
  - klodolph 11mo ago
    Something that is still inheritable, between “there is one and it is global” and “there is a separate copy for each process”.
eichin 11mo ago
A couple of paragraphs in I started wondering if it was going to turn out to be systemd-tmpfiles (in ubuntu, 16.04 I think? The symptom was "about 10 days after login, X11 forwarding over ssh stopped working but local apps could still open windows just fine." I remember it as an ubuntu-specific misconfiguration, though I think the systemd defaults were changed to be less of a footgun in response...) I was pleased that it was more interesting than that, and I want people to write more twitchy-detail-post-mortems like this :-)
Yehia_loay 11mo ago
Cool

143 points by todsacerdoti 11mo ago

And this was written 10 years ago, when computers were far less complicated and vibe coding sleeper bugs wasn't a thing.

detourdog 11mo ago
From my perspective vibe coding was always a thing.
WJW 11mo ago
Vibe coded sleeper bugs have always been a thing, they just came from the bosses' nephew who was still learning PHP at the time and left several years ago. Also, computers in 2015 were not meaningfully less complex than today. Certainly not when the topic is weird emacs and perl interactions.
- marcosdumay 11mo ago
  Even if the topic was web applications (that are where Big Complexity thrives), 2015 was about peak complexity. Things have improved a bit since then.
- add-sub-mul-div 11mo ago
  The problem isn't that AI is doing something new, we all know that it isn't. The problem is that the boss' nephew is becoming the rule now rather than the exception.
  - jama211 11mo ago
    It also makes bugs easier to find and resolve. You win some you lose some. Perhaps by the time it is the rule they’ll be better at writing safer code.

linsomniac 11mo ago

The Internet needs more Buckaroo Banzai references. Because wherever you go, there you are.

neilk 11mo ago
Yup. I nearly had this movie memorized when I was a child. https://www.youtube.com/watch?v=aWXuDNmO7j8 https://www.youtube.com/watch?v=aWXuDNmO7j8 Peter Weller, playing Buckaroo Banzai, is late for his military-particle-physics-interdimensional-jet-car test because he's helping Jeff Goldblum's character with neurosurgery. Later that day he will go play lead guitar in an ensemble. Scriptwriting gurus advise that your protagonist should have flaws and character progression. The writers of this movie disagree.
- linsomniac 11mo ago
  Kevin Smith has an introduction to this movie where he calls it a true piece of art: "It doesn't care what you bring to the table, it bring itself to the table and says: figure it out". https://youtu.be/N8R8wmlggwc?si=sva2-jF1Kl5eFsU4 https://youtu.be/N8R8wmlggwc?si=sva2-jF1Kl5eFsU4

svat 11mo ago

(2016) Also, “direct” link: https://blog.plover.com/tech/tmpdir.html https://blog.plover.com/tech/tmpdir.html (This doesn't really matter, as the posted link is to https://blog.plover.com/2016/07/01/#tmpdir https://blog.plover.com/2016/07/01/#tmpdir i.e. the blog post named “tmpdir” posted on 2016-07-01 and there is only post posted on that date, so the content of the page is basically the same.)

adrianmonk 11mo ago

> This computer stuff is amazingly complicated. I don't know how anyone gets anything done. I wonder what could be done to make this type of problem less hidden and easier to diagnose. The one thing that comes to mind is to have the loader fail fast. For security reasons, the loader needs to ensure TMPDIR isn't set. Right now it accomplishes this by un-setting TMPDIR, which leads to silent failures. Instead, it could check if TMPDIR is set, and if so, give a fatal error. This would force you to unset TMPDIR yourself before you run a privileged program, which would be tedious, but at least you'd know it was happening because you'd be the one doing it. (To be clear, I'm not proposing actually doing this. It would break compatibility. It's just interesting to think about alternative designs.)

tetha 11mo ago
Mh, I am starting to dislike this kind of hyper-configurability. I know when this was necessary and used it myself quite a bit. But today, couldn't we just open up a mount namespace and bind-mount something else to /tmp, like SystemDs private tempdirs? (Which broke a lot of assumptions about tmpdirs and caused a bit of ruckus, but on the other hand, I see their point by now) I'm honestly starting to wonder about a lot of these really weird, prickly and fragile environment variables which cause security vulnerabilities, if low-overhead virtualization and namespacing/containers are available. This would also raise the security floor.
- josephcsible 11mo ago
  > But today, couldn't we just open up a mount namespace and bind-mount something else to /tmp, like SystemDs private tempdirs? No, because unless you're already root (in which case you wouldn't have needed the binary with the capability in the first place), you can't make a mount namespace without also making a user namespace, and the counterproductive risk-averse craziness has led to removing unprivileged users' ability to make user namespaces.
  - kragen 11mo ago
    It's probably true that there are setuid programs that can be exploited if you run them in a user namespace. You probably need to remove setuid (and setgid) as Plan9 did in order to do this.

torben-friis 11mo ago

(Let me start clarifying that this is not at all a criticism of the author) I am usually amused by the way really competent people judge other's context. This post assumes understanding of: - emacs (what it is, and terminology like buffers) - strace - linux directories and "everything is a file" - environment variables - grep and similar - what git is - the fact that 'git whatever' works to run a custom script if git-whatever exists in the path (this one was a TIL for me!) - irc - CVEs - dynamic loaders - file priviledges but then feels important to explain to the audience that: >A socket is a facility that enables interprocess communication

dwedge 11mo ago
I found it interesting that they know how to use strace, but not how to list open files held by a process which to me seems simpler. Again, not criticism just an observation and I enjoyed the article
- parliament32 11mo ago
  Given the "(hi Julia!)" immediately after the strace shenanigans, I interpreted this as a third-party hint; the author most likely had not used strace before. The author is both an example of and an example for how we can get caught in "bubbles" of tools/things we know and use and don't, and blog posts like this are great for discovery (I didn't know about git invoking a binary in the path like his "git re-edit", for example, until today).
  - dwedge 11mo ago
    I discovered that by accident, I had a script called git-pr that opened a pull request with github using the last commit message and then pushed it to slack for approval. I was trying to rewrite it to add a description and wondered why "git pr" pushed an empty message to slack
  - mjd 11mo ago
    I assure you that the author has been using strace since it was invented in the early 1990s.

deleted 11mo ago

[deleted]

jcynix 11mo ago

BTW, the author "mjd" is the author of the excellent book "Higher-Order Perl" which is available online at https://hop.perl.plover.com/book/ https://hop.perl.plover.com/book/

pinkmuffinere 11mo ago
I love mjd! He once replied to me on an HN thread and it lives forever in my memory :)
- roywashere 11mo ago
  Me three! mjd fanboy here
- mjd 11mo ago
  I LIKE PIE
  - deleted 11mo ago
    [deleted]

LordGrey 11mo ago

Buckaroo Banzai: You can check your anatomy all you want, and even though there may be normal variation, when it comes right down to it, this far inside the head it all looks the same. No, no, no, don’t tug on that. You never know what it might be attached to.

buckaroobanzi 11mo ago
See, this is the point, for me, where it started to look like a problem. You know, I wanted to sacrifice the precentral vein in order to get some exposure, but because of this guy's normal variation, I got excited, and all of a sudden I didn't know whether I was looking at the precentral vein, or one of the internal cerebral veins, or the vein of Galen, or the vascular vein of Rosenthal. So, on my own, to me, at this point, I was ready to say that's it, let's get out.

Scott-David 11mo ago

[dead]

thayne 11mo ago

Setting a capability on the perl executable seems like a very bad idea. That effectively grants tha capability to everything that is able to invoke perl (without being restricted to NO_NEW_PRIVILEGES).

aftbit 11mo ago
Yeah why did he want non-root perl to be able to bind to low-numbered ports? Seems like one of those typical footguns of applying non-standard configurations.
- thayne 11mo ago
  My reading is the author didn't do that, rather his/her employers configuration system had done so. Setting TMPDIR to /mnt/tmp seems also to come from that. I would guess both were the result of someone who didn't really know what they were doing trying things until they found something that got what they needed to work, then pushed that out without understanding the broader implications.

lanthade 11mo ago

Don't tug on that applies to hardware too. Years ago I worked on contract for a large blue 3 letter company doing outsourced server management for the fancy credit card company. The incident in question happened before my time on the team but I heard about it first hand from the server admin (let's call him Ben) who had been at the center of it. The data center in question was (IIRC) 160K sqft of raised floor spread across multiple floors in a major metropolitan downtown area. It isn't there anymore. Windows, Unix, Linux, mainframe, San, all the associated fun stuff. Ben was working the day after thanksgiving decommissioning a system. Full software and physical decommission. Approved through all the proper change management procedures. As part of the decommission Ben removed the network cables from under raised floor. Standard snip the connector off and pull it back. Easy. Little did he know that network cable was ever so slightly entangled with another cable. Not enough to give him pause when pulling it though. It wouldn't have been an issue if the other cable had been properly latched in its ports. It wasn't. That little pull ended up pulling the network connection out of a completely unrelated system. A system managed by a completely different group. A system responsible for credit card processing. On USA Black Friday. Oops. CC processing went down. It took far too long to resolve. Amazingly Ben didnt loose his job. After all he followed all the processes and procedures. Kudos to the management team who kept him protected. Change management and change freezes were far more stringent by the time I joined the team. There was also now a raised floor infrastructure group and no one pulled a tile without their involvement. Be careful what you tug on!

klodolph 11mo ago

I have an overly reductive take on this—it’s Unix environment variables. You have your terminal window and your .bashrc (or equivalent), and that sets a bunch of environment variables. But your GUI runs with, most likely, different environment variables. It sucks. And here’s my controversial take on things—the “correct” resolution is to reify some higher-level concept of environment. Each process should not have its own separate copy of environment variables. Some things should be handled… ugh, I hate to say it… through RPC to some centralized system like systemd.

rao-v 11mo ago
Windows registry just sort of hovering in the backdrop
- klodolph 11mo ago
  Something that is still inheritable, between “there is one and it is global” and “there is a separate copy for each process”.

eichin 11mo ago

A couple of paragraphs in I started wondering if it was going to turn out to be systemd-tmpfiles (in ubuntu, 16.04 I think? The symptom was "about 10 days after login, X11 forwarding over ssh stopped working but local apps could still open windows just fine." I remember it as an ubuntu-specific misconfiguration, though I think the systemd defaults were changed to be less of a footgun in response...) I was pleased that it was more interesting than that, and I want people to write more twitchy-detail-post-mortems like this :-)

Yehia_loay 11mo ago

Cool