6 ms·

I saw the title and instantly thought of the "Load bearing mac mini" at Twitter. In our server closet there was a mac mini sitting on another rack mounted serv

I saw the title and instantly thought of the "Load bearing mac mini" at Twitter. In our server closet there was a mac mini sitting on another rack mounted server and plugged directly into a switch. IT found it, asked around and nobody knew what it was, so they unpugged it. Immediately the whole of engineering and support were basically offline. Despite the thing looking suspicious as possible, I had set this thing up as an employee a year before. We were not allowed direct network access to our hosted prod network so as a "stop gap" I setup a SSH tunnel that listened on the mini's IP. At first we used this for access to the support web interface so it could be taken off the internet. At the time my request for a server was rejected. One by one more things got added to the list of things proxied over the device, eventually including basically all internal pages, git access, and about a dozen other random services. I finally got it moved into the server room, but not to real hardware. Once we built a DC we got peered access and the mini finally died.

yjftsjthsd-h 3y ago
> so as a "stop gap" There's Nothing as Permanent as a Temporary Solution™:)
- voiceblue 3y ago
  As immortalized by my favorite case[0] from the codeless code (based on real code [1] from Apollo 11). [0] http://thecodelesscode.com/case/234 http://thecodelesscode.com/case/234 [1] https://github.com/chrislgarry/Apollo-11/blob/master/Luminary099/LUNAR_LANDING_GUIDANCE_EQUATIONS.agc#L179 https://github.com/chrislgarry/Apollo-11/blob/master/Luminar...
- luca3v 3y ago
  Somewhat relatex XKCD: https://xkcd.com/2730/ https://xkcd.com/2730/
orenlindsey 3y ago
I always thought the load bearing Mac mini was a myth, it's such an impossible story and it's been so widely shared. I guess it's just a case of programmers taking the easiest route and having it backfire.
- JohnFen 3y ago
  > I guess it's just a case of programmers taking the easiest route and having it backfire. Or a case of IT not actually providing needed services. That, too, is more common than it should be.
- echohack5 3y ago
  > request was denied > mac mini proxy server I love that the programmer's solution is more expensive than doing it the right way had OP's request been approved.
  - liquidgecka 3y ago
    ... At the end of the the deal with had two mac mini's with auto fail over configured via health checking. Thank goodness we got rid of the service provider that refused to let us pair with the network. Once that happened the networking team could just do normal peering with a standard router. But yea, in the early days that mac sat on my desk. It only got moved when I pointed out the issue to our new security team and their jaw hit the floor. =)
- dijit 3y ago
  Things like this happen all the time. "Shadow IT" is the official name for circumstances that lead to this and it's the consequence of: A) Deprioritising "non-urgent" but "important" tasks consistently and B) IT being bureaucratic and/or unable to allocate budget I had a Mac Pro (trashcan) at Ubisoft that was the only way people were able to play our studios live game from within the office. (Ubisoft had a "NO OUTBOUND CONNECTIONS TO INTERNET" policy for Studios) That same Mac Pro was running our internal slack bot to run Maintenances (and, insult people). I left Ubisoft 5 years ago, as far as I know that Mac Pro is still plugged in on my former desk chugging along, last time I asked my former manager about it was last year. ---- At another company (now owned by Oracle), we had an internal IRC (this was before Slack) and nobody thought about it. One day it went down, and traceroute had indicated it was in our server room; after checking every single server we could not find it, until someone noticed an ethernet cable that went through a run into the floor but didn't come back out. After opening the suspended floor we noticed a laptop running Solaris. That was our IRCd and the OS had an uptime of close to 8 years.
SoftTalker 3y ago
Also proving yet again that the best way to find out if something is in use is to unplug it and see who complains.
- gurchik 3y ago
  So-called "scream test."
  - csmattryder 3y ago
    Chesterton yelling from miles away "that's my damn fence, you moron!"
  - shagie 3y ago
    There was an article by Microsoft back in November about their scream tests - https://www.microsoft.com/insidetrack/blog/microsoft-uses-a-scream-test-to-silence-its-unused-servers/ https://www.microsoft.com/insidetrack/blog/microsoft-uses-a-...
    - notnaut 3y ago
      They have about 8 layers of useless customer service acting as ear muffs to deafen the noise, is the problem I’ve had when screaming at them.
  - Booourns 3y ago
    I once took over a decommissioning project of a DMZ set up for a connection to a third party no longer used. Everything documented showed that all connections were no longer active. Step 1 was to do the ‘scream test’. Some how the enterprise routed the most random traffic through there as we took offline random servers and people’s desktops. Turns out the screaming that happens is at you for making things break instead of a pat on the back for discovering bad networking.
    - euroderf 3y ago
      The information technology equivalent of popping pimples.
hinkley 3y ago
One of the last contracts I worked, they provided such awful hardware that the company and employees ended up BYOD. Not enough memory for a modern IDE to start up levels of awful. As the project ramped down they ended up having to buy an iMac off of us. Someone stopped using it and before the case even got cold I had installed a CI build agent on it to run Karma tests. Which ended up being invaluable but also never managed to get replaced with a real piece of hardware. So they just dragged that thing around as the project went into maintenance.
neilv 3y ago
I consider the labelmaker to be a crucial item of sysadmin gear. :)
- davely 3y ago
  To ensure no one will ever touch it, you can pull the hardware equivalent of adding a comment to a piece of code: // AUTOGENERATED DO NOT DELETE
- majewsky 3y ago
  We have a labelmaker in our hackerspace. The box containing it has a label on it "Please do not print unnecessary labels". Below it is a label reading "ok". :)

yjftsjthsd-h 3y ago

> so as a "stop gap" There's Nothing as Permanent as a Temporary Solution™:)

voiceblue 3y ago
As immortalized by my favorite case[0] from the codeless code (based on real code [1] from Apollo 11). [0] http://thecodelesscode.com/case/234 http://thecodelesscode.com/case/234 [1] https://github.com/chrislgarry/Apollo-11/blob/master/Luminary099/LUNAR_LANDING_GUIDANCE_EQUATIONS.agc#L179 https://github.com/chrislgarry/Apollo-11/blob/master/Luminar...
luca3v 3y ago
Somewhat relatex XKCD: https://xkcd.com/2730/ https://xkcd.com/2730/

orenlindsey 3y ago

I always thought the load bearing Mac mini was a myth, it's such an impossible story and it's been so widely shared. I guess it's just a case of programmers taking the easiest route and having it backfire.

JohnFen 3y ago
> I guess it's just a case of programmers taking the easiest route and having it backfire. Or a case of IT not actually providing needed services. That, too, is more common than it should be.
echohack5 3y ago
> request was denied > mac mini proxy server I love that the programmer's solution is more expensive than doing it the right way had OP's request been approved.
- liquidgecka 3y ago
  ... At the end of the the deal with had two mac mini's with auto fail over configured via health checking. Thank goodness we got rid of the service provider that refused to let us pair with the network. Once that happened the networking team could just do normal peering with a standard router. But yea, in the early days that mac sat on my desk. It only got moved when I pointed out the issue to our new security team and their jaw hit the floor. =)
dijit 3y ago
Things like this happen all the time. "Shadow IT" is the official name for circumstances that lead to this and it's the consequence of: A) Deprioritising "non-urgent" but "important" tasks consistently and B) IT being bureaucratic and/or unable to allocate budget I had a Mac Pro (trashcan) at Ubisoft that was the only way people were able to play our studios live game from within the office. (Ubisoft had a "NO OUTBOUND CONNECTIONS TO INTERNET" policy for Studios) That same Mac Pro was running our internal slack bot to run Maintenances (and, insult people). I left Ubisoft 5 years ago, as far as I know that Mac Pro is still plugged in on my former desk chugging along, last time I asked my former manager about it was last year. ---- At another company (now owned by Oracle), we had an internal IRC (this was before Slack) and nobody thought about it. One day it went down, and traceroute had indicated it was in our server room; after checking every single server we could not find it, until someone noticed an ethernet cable that went through a run into the floor but didn't come back out. After opening the suspended floor we noticed a laptop running Solaris. That was our IRCd and the OS had an uptime of close to 8 years.

SoftTalker 3y ago

Also proving yet again that the best way to find out if something is in use is to unplug it and see who complains.

gurchik 3y ago
So-called "scream test."
- csmattryder 3y ago
  Chesterton yelling from miles away "that's my damn fence, you moron!"
- shagie 3y ago
  There was an article by Microsoft back in November about their scream tests - https://www.microsoft.com/insidetrack/blog/microsoft-uses-a-scream-test-to-silence-its-unused-servers/ https://www.microsoft.com/insidetrack/blog/microsoft-uses-a-...
  - notnaut 3y ago
    They have about 8 layers of useless customer service acting as ear muffs to deafen the noise, is the problem I’ve had when screaming at them.
- Booourns 3y ago
  I once took over a decommissioning project of a DMZ set up for a connection to a third party no longer used. Everything documented showed that all connections were no longer active. Step 1 was to do the ‘scream test’. Some how the enterprise routed the most random traffic through there as we took offline random servers and people’s desktops. Turns out the screaming that happens is at you for making things break instead of a pat on the back for discovering bad networking.
  - euroderf 3y ago
    The information technology equivalent of popping pimples.

hinkley 3y ago

One of the last contracts I worked, they provided such awful hardware that the company and employees ended up BYOD. Not enough memory for a modern IDE to start up levels of awful. As the project ramped down they ended up having to buy an iMac off of us. Someone stopped using it and before the case even got cold I had installed a CI build agent on it to run Karma tests. Which ended up being invaluable but also never managed to get replaced with a real piece of hardware. So they just dragged that thing around as the project went into maintenance.

neilv 3y ago

I consider the labelmaker to be a crucial item of sysadmin gear. :)

davely 3y ago
To ensure no one will ever touch it, you can pull the hardware equivalent of adding a comment to a piece of code: // AUTOGENERATED DO NOT DELETE
majewsky 3y ago
We have a labelmaker in our hackerspace. The box containing it has a label on it "Please do not print unnecessary labels". Below it is a label reading "ok". :)