7 ms·

My OpenBSD Story

144 points by wmat 12y ago

wmat 12y ago
I loved this story. Anything similar out there?
- phiber_phreak 12y ago
  Michael W. Lucas has written many technical books, all useful and all with the same wry humor. His earlier books were published by No Starch Press. He has self-published more recent works at Tilted Windmill Press.
- dwc 12y ago
  Years ago I worked as a developer for a company that also owned a small ISP. The sysadmin for the ISP had quit and they didn't bother to replace him, as another guy could usually keep things running. Until a huge wave of spam hit and the mail server went unresponsive. Typical for that time, the mail server was sendmail with an unholy, gnarly config including SpamAssassin with an unholy, gnarly config. The guy handling sysadmin duties was out of his depth. He gave me access, showed me around the configs, and asked me to help. Sure, I could help. But not in any kind of acceptable time frame. So I went back to my desk and thought about it a bit. A while before I had installed OpenBSD on an old box to play with. I recalled seeing something about a transparent bridge setup, so I got the idea of putting a transparent bridge with greylisting in front of the mail server. So I grabbed a slow cast off desktop, slapped an extra NIC in it, got it configured so I thought it would work, and put it in place. A few tweaks later and the load was off the mail server and it began responding to legitimate requests and clearing out its queue. From idea to working solution took about 4 hours, including scrounging up parts, installing, and everything. I'm sure something similar could have been done with another OS, but what level of expertise would it have required? I was pretty much an OpenBSD newbie then, doing things I'd never even played with before, and that stuff just worked.
- kylec 12y ago
  I found this while Googling for unrelated OpenBSD stuff: http://www.skeptech.org/blog/2013/01/13/unscrewed-a-story-about-openbsd/ http://www.skeptech.org/blog/2013/01/13/unscrewed-a-story-ab...
louwrentius 12y ago
Love this war story. Partially because to a less dramatic extend, I solved a similar situation also with 'traffic shaping' using Linux a year ago. So that's why it resonates for me. I'm not an OpenBSD user but I do respect it's capabilities. I wrote a blog post on how I did it: http://louwrentius.com/how-traffic-shaping-can-dramatically-improve-internet-responsiveness.html http://louwrentius.com/how-traffic-shaping-can-dramatically-...
- grobinson 12y ago
  Can someone how traffic shaping reduces latency rather than simply lowering the amount of load required to cause significant latency? I understand that the aim here is to prevent the router from having to queue packets, but by doing this haven't we just shifted the queue and source at which packets are dropped from the router to the Linux box?
  - agshekeloh 12y ago
    (Note: This link points to my web site, it's my story.) You can't erase the capacity problems that cause issues like latency. Depending on your gear, however, you can shift them. In this case, I used traffic shaping to transfer the latency to the customer that generated the traffic. It let other customers get the normal usage that they'd paid for. ==ml
  - ploxiln 12y ago
    The problem is that the router does not drop packets (as soon as a proper traffic shaper does). It queues them up, megabytes at a time. Packets that do make it to their destination take longer and longer to get there (due to waiting in the queue). Eventually most of the buffered packets are so late that they're considered missing/dropped, though they took bandwidth to send and caused other packets to wait behind them. Then TCP adjusts to a long series of lost packets by dropping the rate dramatically. TCP isn't supposed to work that way :)
imanaccount247 12y ago
That's amazingly similar to a situation I dealt with, except I had already been using openbsd for a year or two. Cisco really like to sell incredibly underpowered hardware and claim it can do anything. And then you end up replacing your $60,000 cisco with an $1200 PC because they really meant "well in theory it could do it if we sold you hardware that could take more than 256MB of RAM".
GFK_of_xmaspast 12y ago
"If it didn’t work, I would either lay someone off or file for unemployment myself."" If I hadn’t already been stressed out, the prospect of choosing a minion to lay off would have done the trick. (Before any of those minions start to think I care about them personally: I work hard training minions, and swinging the Club of Correction makes my arms sore. Eventually. I don’t like to replace them.)" This dude must be a joy to work for.
- olefoo 12y ago
  Evidently you are not familiar with the sort of humor common to alt.sysadmin.recovery And next you'll be telling us that http://bofh.ntk.net/BOFH/index.php http://bofh.ntk.net/BOFH/index.php isn't fiction/satire/parody.
  - GFK_of_xmaspast 12y ago
    I used to post on the Scary Devil Monastery, actually. I wouldn't have wanted to work for a lot of those people either.
    - olefoo 12y ago
      You have a point. I used to work with/for a few of them, and sometimes the online persona was perhaps more aligned with their IRL personality than was good for them. But for the most part it was a way to vent frustration.
- chronid 12y ago
  In my current workplace the joke is for the newcomers to be called "undervisors" (in contrast to shift supervisors). I found it common to get called a "minion", "drone" or "padawan" while working in IT. No one usually gets offended (if someone does usually the nickname gets dropped immediately in his/her presence).
  - gaius 12y ago
    It's no worse than being called a "resource". Of course, it means you also get to call your boss "overhead" ;-)
joshbaptiste 12y ago
Should note that this is from (2011), guess it's not an issue as this is a testimonial which is timeless. Michael writes many *BSD flavored books, but my favorite book that every OpenSSH user should have is his "SSH Mastery" book. http://www.amazon.com/SSH-Mastery-OpenSSH-PuTTY-Tunnels/dp/1470069717/ http://www.amazon.com/SSH-Mastery-OpenSSH-PuTTY-Tunnels/dp/1...
- jcr 12y ago
  There is also Michael's "Sudo Mastery" book, and like his "SSH Mastery" book, it's full of useful lessons for just about anyone working with UNIX-based systems, including Linux and MacOS 10+. Earlier this month, Michael gave a talk based on the "Sudo Mastery" book, titled "Sudo: You're Doing it Wrong": https://www.youtube.com/watch?v=o0purspHg-o https://www.youtube.com/watch?v=o0purspHg-o
FeeTinesAMady 12y ago
I don't understand why the backup option wasn't to boot the one customer whose traffic was killing the company, instead of losing several other clients and going out of business. Yes, try to fix things, and he succeeded, but it seems like he didn't even consider getting rid of the problem customer to save the company.
- rasz_pl 12y ago
  or even throttling that one client, plugging 10Mbit hub between this clients box and router could do the job as a poor mans bw limiter.
  - bluedino 12y ago
    Hub? Why not just set the NIC of the customers machine (or their switch port) to 10mb.
    - phaemon 12y ago
      ADSL broadband was only introduced to the UK in the year 2000, so 10Mb might be pushing the boat out a bit. A better question is who on earth were they hosting who had a website that was that popular back then? Napster?
Touche 12y ago
Interesting. Any theories of why this setup was more efficient than the firewall it replaced? OpenBSD just than much better?
- alecco 12y ago
  OpenBSD is really good for firewalls. It's one of its most popular kinds of install. And very easy to configure.
- tw04 12y ago
  Better? Absolutely not. Cheaper? Definitely. In his story, however accurate, he didn't have the time to wait, or the money to spend, on a better firewall. OpenBSD has the advantage of being free, and instantly available.
  - Touche 12y ago
    Maybe I misread, but he had a professional firewall but just couldn't afford the traffic shaping feature.
- otis_inf 12y ago
  The feature he needed was an optional feature which did cost a lot of money and wasn't available right away so it was useless for this problem at hand as clients were threatening to leave the next day.
  - gaius 12y ago
    In my nearly 20 years in the industry, I can count on my fingers the number of times vendor support has been worth paying even a penny for.
    - otis_inf 12y ago
      You almost never had to consult with a vendor, development team who wrote the stuff you're using, or otherwise 3rd party who supplied you with a piece of software or hardware you're using with your own software in those 20 years or was it in almost all cases a miserable experience? I agree paying a lot of money for simple additional features (the well known 'enterprise features') is in most cases unjustified, but support is IMHO something else.
ptype 12y ago
What's the best way to try out Open BSD? I have not been able to find a VMWare image. Which VPS providers are recommended for OpenBSD?
- toyg 12y ago
  I've installed 5.6 on Vultr.com as a "Custom ISO" and seems to work fine, although I'm not exactly pushing it.
- brynet 12y ago
  By installing it. There are images for floppy/CD and USB media.

My OpenBSD Story

144 points by wmat 12y ago

wmat 12y ago

I loved this story. Anything similar out there?

phiber_phreak 12y ago
Michael W. Lucas has written many technical books, all useful and all with the same wry humor. His earlier books were published by No Starch Press. He has self-published more recent works at Tilted Windmill Press.
dwc 12y ago
Years ago I worked as a developer for a company that also owned a small ISP. The sysadmin for the ISP had quit and they didn't bother to replace him, as another guy could usually keep things running. Until a huge wave of spam hit and the mail server went unresponsive. Typical for that time, the mail server was sendmail with an unholy, gnarly config including SpamAssassin with an unholy, gnarly config. The guy handling sysadmin duties was out of his depth. He gave me access, showed me around the configs, and asked me to help. Sure, I could help. But not in any kind of acceptable time frame. So I went back to my desk and thought about it a bit. A while before I had installed OpenBSD on an old box to play with. I recalled seeing something about a transparent bridge setup, so I got the idea of putting a transparent bridge with greylisting in front of the mail server. So I grabbed a slow cast off desktop, slapped an extra NIC in it, got it configured so I thought it would work, and put it in place. A few tweaks later and the load was off the mail server and it began responding to legitimate requests and clearing out its queue. From idea to working solution took about 4 hours, including scrounging up parts, installing, and everything. I'm sure something similar could have been done with another OS, but what level of expertise would it have required? I was pretty much an OpenBSD newbie then, doing things I'd never even played with before, and that stuff just worked.
kylec 12y ago
I found this while Googling for unrelated OpenBSD stuff: http://www.skeptech.org/blog/2013/01/13/unscrewed-a-story-about-openbsd/ http://www.skeptech.org/blog/2013/01/13/unscrewed-a-story-ab...

louwrentius 12y ago

Love this war story. Partially because to a less dramatic extend, I solved a similar situation also with 'traffic shaping' using Linux a year ago. So that's why it resonates for me. I'm not an OpenBSD user but I do respect it's capabilities. I wrote a blog post on how I did it: http://louwrentius.com/how-traffic-shaping-can-dramatically-improve-internet-responsiveness.html http://louwrentius.com/how-traffic-shaping-can-dramatically-...

grobinson 12y ago
Can someone how traffic shaping reduces latency rather than simply lowering the amount of load required to cause significant latency? I understand that the aim here is to prevent the router from having to queue packets, but by doing this haven't we just shifted the queue and source at which packets are dropped from the router to the Linux box?
- agshekeloh 12y ago
  (Note: This link points to my web site, it's my story.) You can't erase the capacity problems that cause issues like latency. Depending on your gear, however, you can shift them. In this case, I used traffic shaping to transfer the latency to the customer that generated the traffic. It let other customers get the normal usage that they'd paid for. ==ml
- ploxiln 12y ago
  The problem is that the router does not drop packets (as soon as a proper traffic shaper does). It queues them up, megabytes at a time. Packets that do make it to their destination take longer and longer to get there (due to waiting in the queue). Eventually most of the buffered packets are so late that they're considered missing/dropped, though they took bandwidth to send and caused other packets to wait behind them. Then TCP adjusts to a long series of lost packets by dropping the rate dramatically. TCP isn't supposed to work that way :)

imanaccount247 12y ago

That's amazingly similar to a situation I dealt with, except I had already been using openbsd for a year or two. Cisco really like to sell incredibly underpowered hardware and claim it can do anything. And then you end up replacing your $60,000 cisco with an $1200 PC because they really meant "well in theory it could do it if we sold you hardware that could take more than 256MB of RAM".

GFK_of_xmaspast 12y ago

"If it didn’t work, I would either lay someone off or file for unemployment myself."" If I hadn’t already been stressed out, the prospect of choosing a minion to lay off would have done the trick. (Before any of those minions start to think I care about them personally: I work hard training minions, and swinging the Club of Correction makes my arms sore. Eventually. I don’t like to replace them.)" This dude must be a joy to work for.

olefoo 12y ago
Evidently you are not familiar with the sort of humor common to alt.sysadmin.recovery And next you'll be telling us that http://bofh.ntk.net/BOFH/index.php http://bofh.ntk.net/BOFH/index.php isn't fiction/satire/parody.
- GFK_of_xmaspast 12y ago
  I used to post on the Scary Devil Monastery, actually. I wouldn't have wanted to work for a lot of those people either.
  - olefoo 12y ago
    You have a point. I used to work with/for a few of them, and sometimes the online persona was perhaps more aligned with their IRL personality than was good for them. But for the most part it was a way to vent frustration.
chronid 12y ago
In my current workplace the joke is for the newcomers to be called "undervisors" (in contrast to shift supervisors). I found it common to get called a "minion", "drone" or "padawan" while working in IT. No one usually gets offended (if someone does usually the nickname gets dropped immediately in his/her presence).
- gaius 12y ago
  It's no worse than being called a "resource". Of course, it means you also get to call your boss "overhead" ;-)

joshbaptiste 12y ago

Should note that this is from (2011), guess it's not an issue as this is a testimonial which is timeless. Michael writes many *BSD flavored books, but my favorite book that every OpenSSH user should have is his "SSH Mastery" book. http://www.amazon.com/SSH-Mastery-OpenSSH-PuTTY-Tunnels/dp/1470069717/ http://www.amazon.com/SSH-Mastery-OpenSSH-PuTTY-Tunnels/dp/1...

jcr 12y ago
There is also Michael's "Sudo Mastery" book, and like his "SSH Mastery" book, it's full of useful lessons for just about anyone working with UNIX-based systems, including Linux and MacOS 10+. Earlier this month, Michael gave a talk based on the "Sudo Mastery" book, titled "Sudo: You're Doing it Wrong": https://www.youtube.com/watch?v=o0purspHg-o https://www.youtube.com/watch?v=o0purspHg-o

FeeTinesAMady 12y ago

I don't understand why the backup option wasn't to boot the one customer whose traffic was killing the company, instead of losing several other clients and going out of business. Yes, try to fix things, and he succeeded, but it seems like he didn't even consider getting rid of the problem customer to save the company.

rasz_pl 12y ago
or even throttling that one client, plugging 10Mbit hub between this clients box and router could do the job as a poor mans bw limiter.
- bluedino 12y ago
  Hub? Why not just set the NIC of the customers machine (or their switch port) to 10mb.
  - phaemon 12y ago
    ADSL broadband was only introduced to the UK in the year 2000, so 10Mb might be pushing the boat out a bit. A better question is who on earth were they hosting who had a website that was that popular back then? Napster?

Touche 12y ago

Interesting. Any theories of why this setup was more efficient than the firewall it replaced? OpenBSD just than much better?

alecco 12y ago
OpenBSD is really good for firewalls. It's one of its most popular kinds of install. And very easy to configure.
tw04 12y ago
Better? Absolutely not. Cheaper? Definitely. In his story, however accurate, he didn't have the time to wait, or the money to spend, on a better firewall. OpenBSD has the advantage of being free, and instantly available.
- Touche 12y ago
  Maybe I misread, but he had a professional firewall but just couldn't afford the traffic shaping feature.
otis_inf 12y ago
The feature he needed was an optional feature which did cost a lot of money and wasn't available right away so it was useless for this problem at hand as clients were threatening to leave the next day.
- gaius 12y ago
  In my nearly 20 years in the industry, I can count on my fingers the number of times vendor support has been worth paying even a penny for.
  - otis_inf 12y ago
    You almost never had to consult with a vendor, development team who wrote the stuff you're using, or otherwise 3rd party who supplied you with a piece of software or hardware you're using with your own software in those 20 years or was it in almost all cases a miserable experience? I agree paying a lot of money for simple additional features (the well known 'enterprise features') is in most cases unjustified, but support is IMHO something else.

ptype 12y ago

What's the best way to try out Open BSD? I have not been able to find a VMWare image. Which VPS providers are recommended for OpenBSD?

toyg 12y ago
I've installed 5.6 on Vultr.com as a "Custom ISO" and seems to work fine, although I'm not exactly pushing it.
brynet 12y ago
By installing it. There are images for floppy/CD and USB media.