12 ms·

Terminal Server on a Budget

92 points by Jaruzel 6y ago

xupybd 6y ago
I would love be to know what the family is using the servers for.
- dredmorbius 6y ago
  I run a number of services in my home covering all the essentials such as media serving, DNS, local cloud, PBX, Minecraft containers, etc. 'graph 2 of TFA.
  - sushshshsh 6y ago
    Is it really a time-savings to say that instead of "Paragraph 2 of the article"?
    - xupybd 6y ago
      Ah missed that, thanks for pointing that out.
twicetwice 6y ago
This is super cool! I'd love to get this working for my home servers as well one day (priority number one is making them accessible outside my LAN again).
- jjice 6y ago
  I'm currently using a Digital Ocean droplet with wireguard and my home server connects to that VPS via wireguard site-to-site. I have Nginx on the VPS and do a proxy_pass to my local services. Just set it up a week ago and works great, very simple once you understand the setup.
  - twicetwice 6y ago
    I was planning to do exactly that! I already have a Digital Ocean droplet with Nginx on it (serving a couple of very-low-traffic apps) ready to go, I just haven't had time to dive into the problem yet. Glad to hear it worked for you! I need to do this because port forwarding isn't working with my current ISP/router/etc setup (not sure where the problem lies). Did you forward the wireguard port to your home server, or another approach? I did some research and it looks like there's a "keep-alive" option that hopefully should get my wireguard connection open through NAT.
    - jjice 6y ago
      I didn't have to touch my router settings at all. There are a million guides out there, but I used this one [0]. I'm not sure about the keep alive, but I'm assuming it's this setting that they mention in the guide: # Ensures that your home router does not kill the tunnel, by sending a ping # every 25 seconds. PersistentKeepalive = 25 I set up the server .conf, ran wg-quick up wg0, and then did the same on the client and it worked like a charm. Just make sure to open your UDP port for Wireguard on your VPS, that cost me a solid 10 minutes of confusion. [0] https://zach.bloomqu.ist/blog/2019/11/site-to-site-wireguard-vpn.html https://zach.bloomqu.ist/blog/2019/11/site-to-site-wireguard...
  - wahern 6y ago
    Wireguard requires a more complex setup than for IPSec, believe it or not. To give my home router a static IP address, I use a Vultr VPS and pay for an additional IP address. I then use IPSec encapsulation to forward traffic for the second IP address directly to the router, which initiates the IPSec SA setup. The router has a loopback device configured with the static IP address; the routable static address is effectively local, even from the kernel's perspective. The IPSec rules handle forwarding of everything, so I don't even need packet filtering (e.g. OpenBSD PF in my case) rules. No need to fiddle with private address ranges, NAT'ing, reverse proxying, or packet filtering of any sort. The IKEv2 setup is a single line on the VPS and a single line on the router, though OpenBSD's OpenIKED (as well as their isakmpd fork for IKEv1) is awesome that way; StrongSwan and other Linux IKE daemons require a more verbose configuration on account of their key-value pair syntax, but in any event it's still relatively simple.
katzgrau 6y ago
> I no longer have that box of serial adaptors and my wife somehow can’t remember where it went since we got maried. I used to have a glorious box of ram, spare drives, and every cable and adapter imaginable. My box also mysteriously disappeared some time after I got married. I feel your pain :)
- bfuclusion 6y ago
  Mine went 3 moves back. I had a great collection of real RS232 cards and USB cables from various years and am now really missing them since I've gotten into ESP32.
- huxflux 6y ago
  Exactly the same thing happened to me when I got married. Head's up single fellas.
- Roritharr 6y ago
  My wife misplaced my very first modem, US Robotics 56k. Sometimes I wonder what would have happened if we didn't already have kids.
- C1sc0cat 6y ago
  And the joys of soldering up a rs232 cable including the swanky heat shrink tubing - only to find you had neglected to put the hoods on first. I recall at one job we had very long runs of rs232 cables to some vt100's which where very sensitive to electrical charges. To the point where one of our electronics shop brought the official Vt100 manuals so they could repair any blown chips. DEC field circus loved us :-) It was the 11/03 covered I black coal dust that really upset them.
- Cthulhu_ 6y ago
  Sounds like some serious boundary-crossing behaviour; have you ever discussed that with your spouse? Did some of their stuff "disappear" as well? I'm just saying, this is not normal or acceptable behaviour.
  - halfmatthalfcat 6y ago
nwmcsween 6y ago
Interesting but why not a kvm rack unit from ebay? Cheaper and easier only issue is most vendors stop supporting the switches and the software on them can be complete garbage (only works with firefox version x or IE version y).
- matt-attack 6y ago
  Yeah everyone I’ve used requires java in the browser. Feels like 1998.
  - nwmcsween 6y ago
    Later avocents have a sort of usable HTML5 implementation. It would be nice if there was something like OpenWRT for kvm switches
cl3misch 6y ago
Maybe I missed it, but what device is on the other end of the COM connection? His router? What's the reason against SSH'ing from the gateway to the servers instead of the physical COM connection?
- jlgaddis 6y ago
  > Maybe I missed it, but what device is on the other end of the COM connection? His router? The plan was to use a Raspberry Pi to access the two PCs (via USB-to-serial adapters), though he hasn't made it to that point yet ("I have not used it for this project yet, but it is still in my plans."). Right now, the two PCs are just connected to each other, back-to-back. > What's the reason against SSH'ing from the gateway to the servers instead of the physical COM connection? I would assume that he does just use SSH -- under normal circumstances. It's when SSH access isn't available that access via the serial console would be used -- such as the network connection being down or b0rked, when the kernel (or required) modules failed to boot/build/load properly, a bad firewall change has locked you out, and so on. This method even provides access to the bootloader. If your BIOS supports "console redirection" (to a serial port), you also gain access to the BIOS, can watch the startup (POST) process, or even change the boot device -- in order to fire off a kickstart installation, for example. He mentioned that his PCs don't have ILO/DRAC/IPMI, so this is his backup solution. -- remember that his machines are headless.
gatvol 6y ago
Why not use a jumpbox /ssh ?
- iso947 6y ago
  Because when the box doesn’t boot you can’t ssh in.
xiconfjs 6y ago
We are searching for a new serial to ethernet solution which is not as expensive as most B2B products. Right now this [1] would be our dream product. We are also looking into a combination of a mini-pc (like PC-Engines [2]) with two of these miniPCIe serial cards [3]. If you have better recommendations feel free to share them :) [1] https://freetserv.github.io/ https://freetserv.github.io/ [2] https://www.pcengines.ch/apu2c2.htm https://www.pcengines.ch/apu2c2.htm [3] https://www.delock.de/produkte/G_95244/merkmale.html https://www.delock.de/produkte/G_95244/merkmale.html
- jlgaddis 6y ago
  Old multi-port serial consoles can often be found used quite cheap. Personally, I'm a big fan of the OpenGear units. They're *much^ more than just basic serial consoles and are quite extendable. You can even write your own scripts to interact with them, trigger them on specific events, and so on.
  - cure 6y ago
    Yeah +1 for OpenGear console servers, I've used them for years. Looks like there are some bargains to be had on ebay, I see a 32 port one for $150...
- plett 6y ago
  I've used Startech FTDI based multi-port serial to USB adapters along with a Mikrotik Routerboard to make serial-to-ethernet or serial-to-LTE boxes. The Mikrotik boxes expose the serial port on a TCP port for you to connect to. https://imgur.com/a/fdMfjdl https://imgur.com/a/fdMfjdl
teddyh 6y ago
Note: Some server motherboards also have a setting in the BIOS to show boot-time text, and allow BIOS configuration, on a serial port.
- wokkel 6y ago
  Not only server. If you have a Q-series desktop motherboard with AMT you can enable Serial-Over-Lan and access the machine remotely (and power on/off or reset it as well). Q series chipsets are also found on Fujitsu/Dell/HP prebuilts which can be picked up quite cheaply when companies decide to upgrade. For servers you usually have IPMI/iDrac etc. which uses more power, but usually has a better (web)interface.
  - plett 6y ago
    Yep. I've just picked up a few second hand HP EliteDesk 800 G2 desktops with the intention of turning them into a homelab for k8s learning. They have AMT and the web power control, VNC based KVM and SoL work well. There is no BIOS serial redirection, so SoL is no use until the OS starts booting, and grub has a bug[0][1][2] which means it can't find the AMT SoL when booting in EFI mode. But SoL works fine as soon as the kernel starts, and the KVM VNC works fine during BIOS and GRUB if I have an emergency. One gotcha which would catch someone out who wasn't expecting it - the onboard video disables itself when no monitor is connected, so the AMT KVM just shows a black screen. This can be solved by DisplayPort or VGA "ghost" dongle in the back of the machine to talk EDID and pretend to be a screen. I'm a big fan of conserver, and intend to use it in this project to spawn amtterm processes to connect to the SoL ports. 0: https://savannah.gnu.org/bugs/?42026 https://savannah.gnu.org/bugs/?42026 1: https://community.intel.com/t5/Intel-vPro-Platform/No-AMT-serial-port-in-Linux-grub2/td-p/720528 https://community.intel.com/t5/Intel-vPro-Platform/No-AMT-se... 2: https://wiki.networksecuritytoolkit.org/index.php/HowTo_Headless_Intel_NUC_vPro_AMT#AMT_Serial_Port https://wiki.networksecuritytoolkit.org/index.php/HowTo_Head...
- jlgaddis 6y ago
  ... and, if you don't have Serial-over-LAN (SOL) access via IPMI, etc., you can always just wire the serial port up to another host to give yourself access to that serial port remotely. Then, just SSH in to the other host and use something like "minicom" to connect directly to the serial port -- or use "conserver" (mentioned in TFA) when you've got multiple consoles and/or multiple users to deal with simultaneously. Plenty of HN'ers are all-in on "cloud" but for anyone dealing with physical devices (routers, servers, whatever), remote access to the console can be a lifesaver at times!
jlgaddis 6y ago
"conserver" (which the author is hsing) is really handy for quickly and easily accessing and/or monitoring the (serial) console of not just physical machines (like the author is doing) but also virtual machines as well. I've used it with virtual machines running on both vSphere and QEMU/KVM to make their serial consoles available over the network... to access the serial console you can just connect to the corresponding TCP port. Just make sure to lock down / restrict access appropriately and ensure you don't send sensitive info over the network in clear-text! It works the same way with physical devices too (Cisco devices, primarily). It's great when you want to a "permanent record" (on disk) of any output that is sent to the console. As mentioned, it's really nice when you need to provide "shared" console access and/or access to multiple devices and/or users simultaneously. It's really an under-rated tool.
Harvesterify 6y ago
So, why not a KVM ?
dxld 6y ago
> One small disadvantage not addressed is that neither of the two motherboards support BIOS access via COM1, which is a bummer. I've always wondered if you could burn SGABIOS (https://code.google.com/archive/p/sgabios/ https://code.google.com/archive/p/sgabios/) to a PCI(e) card ROM to get this working on real hardware instead of just in QEMU. Apparently someone did try this, so it might just work: https://www.flashrom.org/User:GNUtoo/Howto_flash_sgabios_on_an_Nvidia_video_card https://www.flashrom.org/User:GNUtoo/Howto_flash_sgabios_on_... Maybe I should dig up some old flashrom supported NIC card or something.
drcross 6y ago
Why not use a Raspberry Pi populated with several USB to console adaptors, there are USB console cables with up to four serial ports per connection which would result in 16 console ports per Pi.
tyingq 6y ago
A fair amount of PC's already have remote access built in, via Intel's vPro. Might be worth checking for that if you're on a budget.
ed25519FUUU 6y ago
> more than a few times I have gotten into a situation where I have attempted to preform a system update on one of the pair remotely from cafe or sofa and the PC failed to come back up after a reboot. This happened to me as well. In fact it happened so often that I took all of the critical network software off the home server and run it on an edge router-x. Also, setting “power on after power loss” and running the computer through a smart plug is a good idea. When it gets jammed, you can at least power cycle it remotely and it attempts to reboot.

92 points by Jaruzel 6y ago

xupybd 6y ago

I would love be to know what the family is using the servers for.

dredmorbius 6y ago
I run a number of services in my home covering all the essentials such as media serving, DNS, local cloud, PBX, Minecraft containers, etc. 'graph 2 of TFA.
- sushshshsh 6y ago
  Is it really a time-savings to say that instead of "Paragraph 2 of the article"?
  - xupybd 6y ago
    Ah missed that, thanks for pointing that out.

twicetwice 6y ago

This is super cool! I'd love to get this working for my home servers as well one day (priority number one is making them accessible outside my LAN again).

jjice 6y ago
I'm currently using a Digital Ocean droplet with wireguard and my home server connects to that VPS via wireguard site-to-site. I have Nginx on the VPS and do a proxy_pass to my local services. Just set it up a week ago and works great, very simple once you understand the setup.
- twicetwice 6y ago
  I was planning to do exactly that! I already have a Digital Ocean droplet with Nginx on it (serving a couple of very-low-traffic apps) ready to go, I just haven't had time to dive into the problem yet. Glad to hear it worked for you! I need to do this because port forwarding isn't working with my current ISP/router/etc setup (not sure where the problem lies). Did you forward the wireguard port to your home server, or another approach? I did some research and it looks like there's a "keep-alive" option that hopefully should get my wireguard connection open through NAT.
  - jjice 6y ago
    I didn't have to touch my router settings at all. There are a million guides out there, but I used this one [0]. I'm not sure about the keep alive, but I'm assuming it's this setting that they mention in the guide: # Ensures that your home router does not kill the tunnel, by sending a ping # every 25 seconds. PersistentKeepalive = 25 I set up the server .conf, ran wg-quick up wg0, and then did the same on the client and it worked like a charm. Just make sure to open your UDP port for Wireguard on your VPS, that cost me a solid 10 minutes of confusion. [0] https://zach.bloomqu.ist/blog/2019/11/site-to-site-wireguard-vpn.html https://zach.bloomqu.ist/blog/2019/11/site-to-site-wireguard...
- wahern 6y ago
  Wireguard requires a more complex setup than for IPSec, believe it or not. To give my home router a static IP address, I use a Vultr VPS and pay for an additional IP address. I then use IPSec encapsulation to forward traffic for the second IP address directly to the router, which initiates the IPSec SA setup. The router has a loopback device configured with the static IP address; the routable static address is effectively local, even from the kernel's perspective. The IPSec rules handle forwarding of everything, so I don't even need packet filtering (e.g. OpenBSD PF in my case) rules. No need to fiddle with private address ranges, NAT'ing, reverse proxying, or packet filtering of any sort. The IKEv2 setup is a single line on the VPS and a single line on the router, though OpenBSD's OpenIKED (as well as their isakmpd fork for IKEv1) is awesome that way; StrongSwan and other Linux IKE daemons require a more verbose configuration on account of their key-value pair syntax, but in any event it's still relatively simple.

katzgrau 6y ago

> I no longer have that box of serial adaptors and my wife somehow can’t remember where it went since we got maried. I used to have a glorious box of ram, spare drives, and every cable and adapter imaginable. My box also mysteriously disappeared some time after I got married. I feel your pain :)

bfuclusion 6y ago
Mine went 3 moves back. I had a great collection of real RS232 cards and USB cables from various years and am now really missing them since I've gotten into ESP32.
huxflux 6y ago
Exactly the same thing happened to me when I got married. Head's up single fellas.
Roritharr 6y ago
My wife misplaced my very first modem, US Robotics 56k. Sometimes I wonder what would have happened if we didn't already have kids.
C1sc0cat 6y ago
And the joys of soldering up a rs232 cable including the swanky heat shrink tubing - only to find you had neglected to put the hoods on first. I recall at one job we had very long runs of rs232 cables to some vt100's which where very sensitive to electrical charges. To the point where one of our electronics shop brought the official Vt100 manuals so they could repair any blown chips. DEC field circus loved us :-) It was the 11/03 covered I black coal dust that really upset them.
Cthulhu_ 6y ago
Sounds like some serious boundary-crossing behaviour; have you ever discussed that with your spouse? Did some of their stuff "disappear" as well? I'm just saying, this is not normal or acceptable behaviour.
- halfmatthalfcat 6y ago

nwmcsween 6y ago

Interesting but why not a kvm rack unit from ebay? Cheaper and easier only issue is most vendors stop supporting the switches and the software on them can be complete garbage (only works with firefox version x or IE version y).

matt-attack 6y ago
Yeah everyone I’ve used requires java in the browser. Feels like 1998.
- nwmcsween 6y ago
  Later avocents have a sort of usable HTML5 implementation. It would be nice if there was something like OpenWRT for kvm switches

cl3misch 6y ago

Maybe I missed it, but what device is on the other end of the COM connection? His router? What's the reason against SSH'ing from the gateway to the servers instead of the physical COM connection?

jlgaddis 6y ago
> Maybe I missed it, but what device is on the other end of the COM connection? His router? The plan was to use a Raspberry Pi to access the two PCs (via USB-to-serial adapters), though he hasn't made it to that point yet ("I have not used it for this project yet, but it is still in my plans."). Right now, the two PCs are just connected to each other, back-to-back. > What's the reason against SSH'ing from the gateway to the servers instead of the physical COM connection? I would assume that he does just use SSH -- under normal circumstances. It's when SSH access isn't available that access via the serial console would be used -- such as the network connection being down or b0rked, when the kernel (or required) modules failed to boot/build/load properly, a bad firewall change has locked you out, and so on. This method even provides access to the bootloader. If your BIOS supports "console redirection" (to a serial port), you also gain access to the BIOS, can watch the startup (POST) process, or even change the boot device -- in order to fire off a kickstart installation, for example. He mentioned that his PCs don't have ILO/DRAC/IPMI, so this is his backup solution. -- remember that his machines are headless.

gatvol 6y ago

Why not use a jumpbox /ssh ?

iso947 6y ago
Because when the box doesn’t boot you can’t ssh in.

xiconfjs 6y ago

We are searching for a new serial to ethernet solution which is not as expensive as most B2B products. Right now this [1] would be our dream product. We are also looking into a combination of a mini-pc (like PC-Engines [2]) with two of these miniPCIe serial cards [3]. If you have better recommendations feel free to share them :) [1] https://freetserv.github.io/ https://freetserv.github.io/ [2] https://www.pcengines.ch/apu2c2.htm https://www.pcengines.ch/apu2c2.htm [3] https://www.delock.de/produkte/G_95244/merkmale.html https://www.delock.de/produkte/G_95244/merkmale.html

jlgaddis 6y ago
Old multi-port serial consoles can often be found used quite cheap. Personally, I'm a big fan of the OpenGear units. They're *much^ more than just basic serial consoles and are quite extendable. You can even write your own scripts to interact with them, trigger them on specific events, and so on.
- cure 6y ago
  Yeah +1 for OpenGear console servers, I've used them for years. Looks like there are some bargains to be had on ebay, I see a 32 port one for $150...
plett 6y ago
I've used Startech FTDI based multi-port serial to USB adapters along with a Mikrotik Routerboard to make serial-to-ethernet or serial-to-LTE boxes. The Mikrotik boxes expose the serial port on a TCP port for you to connect to. https://imgur.com/a/fdMfjdl https://imgur.com/a/fdMfjdl

teddyh 6y ago

Note: Some server motherboards also have a setting in the BIOS to show boot-time text, and allow BIOS configuration, on a serial port.

wokkel 6y ago
Not only server. If you have a Q-series desktop motherboard with AMT you can enable Serial-Over-Lan and access the machine remotely (and power on/off or reset it as well). Q series chipsets are also found on Fujitsu/Dell/HP prebuilts which can be picked up quite cheaply when companies decide to upgrade. For servers you usually have IPMI/iDrac etc. which uses more power, but usually has a better (web)interface.
- plett 6y ago
  Yep. I've just picked up a few second hand HP EliteDesk 800 G2 desktops with the intention of turning them into a homelab for k8s learning. They have AMT and the web power control, VNC based KVM and SoL work well. There is no BIOS serial redirection, so SoL is no use until the OS starts booting, and grub has a bug[0][1][2] which means it can't find the AMT SoL when booting in EFI mode. But SoL works fine as soon as the kernel starts, and the KVM VNC works fine during BIOS and GRUB if I have an emergency. One gotcha which would catch someone out who wasn't expecting it - the onboard video disables itself when no monitor is connected, so the AMT KVM just shows a black screen. This can be solved by DisplayPort or VGA "ghost" dongle in the back of the machine to talk EDID and pretend to be a screen. I'm a big fan of conserver, and intend to use it in this project to spawn amtterm processes to connect to the SoL ports. 0: https://savannah.gnu.org/bugs/?42026 https://savannah.gnu.org/bugs/?42026 1: https://community.intel.com/t5/Intel-vPro-Platform/No-AMT-serial-port-in-Linux-grub2/td-p/720528 https://community.intel.com/t5/Intel-vPro-Platform/No-AMT-se... 2: https://wiki.networksecuritytoolkit.org/index.php/HowTo_Headless_Intel_NUC_vPro_AMT#AMT_Serial_Port https://wiki.networksecuritytoolkit.org/index.php/HowTo_Head...
jlgaddis 6y ago
... and, if you don't have Serial-over-LAN (SOL) access via IPMI, etc., you can always just wire the serial port up to another host to give yourself access to that serial port remotely. Then, just SSH in to the other host and use something like "minicom" to connect directly to the serial port -- or use "conserver" (mentioned in TFA) when you've got multiple consoles and/or multiple users to deal with simultaneously. Plenty of HN'ers are all-in on "cloud" but for anyone dealing with physical devices (routers, servers, whatever), remote access to the console can be a lifesaver at times!

jlgaddis 6y ago

"conserver" (which the author is hsing) is really handy for quickly and easily accessing and/or monitoring the (serial) console of not just physical machines (like the author is doing) but also virtual machines as well. I've used it with virtual machines running on both vSphere and QEMU/KVM to make their serial consoles available over the network... to access the serial console you can just connect to the corresponding TCP port. Just make sure to lock down / restrict access appropriately and ensure you don't send sensitive info over the network in clear-text! It works the same way with physical devices too (Cisco devices, primarily). It's great when you want to a "permanent record" (on disk) of any output that is sent to the console. As mentioned, it's really nice when you need to provide "shared" console access and/or access to multiple devices and/or users simultaneously. It's really an under-rated tool.

Harvesterify 6y ago

So, why not a KVM ?

dxld 6y ago

> One small disadvantage not addressed is that neither of the two motherboards support BIOS access via COM1, which is a bummer. I've always wondered if you could burn SGABIOS (https://code.google.com/archive/p/sgabios/ https://code.google.com/archive/p/sgabios/) to a PCI(e) card ROM to get this working on real hardware instead of just in QEMU. Apparently someone did try this, so it might just work: https://www.flashrom.org/User:GNUtoo/Howto_flash_sgabios_on_an_Nvidia_video_card https://www.flashrom.org/User:GNUtoo/Howto_flash_sgabios_on_... Maybe I should dig up some old flashrom supported NIC card or something.

drcross 6y ago

Why not use a Raspberry Pi populated with several USB to console adaptors, there are USB console cables with up to four serial ports per connection which would result in 16 console ports per Pi.

tyingq 6y ago

A fair amount of PC's already have remote access built in, via Intel's vPro. Might be worth checking for that if you're on a budget.

ed25519FUUU 6y ago

> more than a few times I have gotten into a situation where I have attempted to preform a system update on one of the pair remotely from cafe or sofa and the PC failed to come back up after a reboot. This happened to me as well. In fact it happened so often that I took all of the critical network software off the home server and run it on an edge router-x. Also, setting “power on after power loss” and running the computer through a smart plug is a good idea. When it gets jammed, you can at least power cycle it remotely and it attempts to reboot.