7 ms·

This looks like one of those home server infrastructure setup rabbit holes I would've bailed out of within the first 10 minutes of Googling to research the pro

by DCKing 3y ago

This looks like one of those home server infrastructure setup rabbit holes I would've bailed out of within the first 10 minutes of Googling to research the problem. I commend the author for their perseverance and the writeup - rabbit hole writeups are their own form of entertainment. That alone was worth it. I could never run anything that looks like such a conceptual pain the butt. A pain to maintain and troubleshoot to boot. Especially since the setup requires a Raspberry Pi to run permanently. A Raspberry Pi that itself would work just as fine as the Time Machine server the author seems to need on demand?

ttanev 3y ago
My solution to the "10 years old PC used as a backup target" was to just donate/sell-off-as-parts the old PC, and replace with a 10 year old laptop, with an external USB enclosure. From 180-200 Watts down to 10-25, but it was not as much fun with two weeks of a rabbit whole digging :)
louwrentius 3y ago
I’m using A Raspberry Pi4 as a dedicated Timemachine backup server with an SSD via USB3. Works perfectly for a year or so.
dolmen 3y ago
However that requires the disk used for backup storage to also be always on.
- aequitas 3y ago
  Disk have power modes and can be spun down to save energy. A Raspberry Pi can run the OS on an SD card and mount the disk for backup storage. Leaving it in low power for a long period of time.
  - nwb99 3y ago
    I'm worried about the wear on my drives spinning up and down so much? Maybe it isn't such a big deal. It is the most stressful time for spinning drives, though.
- MattPalmer1086 3y ago
  I use my pi as a backup server. The disks spin down when not in use.
dolmen 3y ago
Ideally I would like the "wake-up / fake on" infrastructure to be a feature of my wifi router.
- xur17 3y ago
  I wonder if you could build this as a plugin for opnsense.
txdv 3y ago
i run an rpi to host stuff for cheap (the energy drain of the pi is low), my main machine goes to sleep after 15min to save costs. seemes like a neat trick to remotely wake it up
aequitas 3y ago
I agree, but you could argue that the Raspberry Pi is not performant enough during restores or browsing the backups. During the backup itself performance should not have to be that big of an issue as it's mostly a background operation you will not be waiting on. You could also use the RPi as a stepping stone which syncs it's files to the 'real' backup server which is on a daily/weekly timer instead of on-demand setup. I'm using this concept for the problem that I want to collect metrics all day long, but not have a power hungry server running 24/7. So my RPi collects all metrics and streams then directly to my server for me to view realtime. But whenever the server is unavailable (on schedule or for maintenance) the RPi just buffers the metrics until the server is available again. Also worth considering is just not using a single server for a single purpose but just cramming it full with utility to justify the power costs.
- manmal 3y ago
  Then how about an N95/N100 (recent Celeron successor) mini PC, like the ones from Beelink? They don’t draw that much power, but are 2-3x faster than a Raspi, have HW supported video de/encoding and can even run Windows if you needed that.
  - DJBunnies 3y ago
    "that much power" is pretty subjective, is it 2x or 10x a pi? Might be a turnoff for some.
    - manmal 3y ago
      The N100 has a TDP of 6W. I haven't measured power draw of the whole system yet though, I imagine that the mainboard will pull another couple watts at least. It shines especially in scenarios where the Pi is just a tad too slow. E.g. you can't live transcode 4k videos on a Pi, while the N100 has latest UHD graphics and therefore ships with a decent HW-accelerated H265 en/decoder.
      - nottorp 3y ago
MattPalmer1086 3y ago
Meh, I struggled to get my pi to work as a time machine server. It would work for a few weeks and then just stop working and I would have to start the backup from scratch previous backups becoming unusable. In the end I just plugged an external hard disk into the mac. I have had similar issues in the past even with such simple things as SSH to a Mac. It would be easy to blame the pi or Linux, but as far as I can tell, Apple just does not play nicely with anything not Apple. It's just that little bit different. I have no such problems interfacing Linux and the pi with anything else.
- testfoobar 3y ago
  Time Machine over NAS has always been unstable for me. As you experienced, my backup would be regularly corrupted and would need to restart. Alternatively, a normal <10G backup could take in excess of 24h. I eventually gave up on the NAS and now backup to external drives at home. This site has excellent articles on Time Machine - scroll down to Time Machine Problems: https://eclecticlight.co/mac-troubleshooting-summary/ https://eclecticlight.co/mac-troubleshooting-summary/
  - jdminhbg 3y ago
    I had the exact same problem and ended up with the same solution. I know people who have had somewhat better experiences backing up to something like a Synology, but the $Xk investment wasn't worth it for me without also needing it for big file storage or media serving or whatever.
    - nottorp 3y ago
      > backing up to something like a Synology How about QNAP? I happen to have one gathering dust, if you can reliably Time Machine to it i'd plug it in.
  - bobleeswagger 3y ago
    There are some quirks to setting up SMB so Time Machine is happy, but I have had great success with ZFS+SMB hosting my TM backups. Have been running for a little over a month and nothing is corrupt. Self hosting is still somewhat hard, I don't blame you for bailing on something so critical to uptime in the first place.
OJFord 3y ago
You could run avahi and 'a simple ruby script' on the router instead (ok not most retail routers as stock, but e.g. OpenWRT) and do away with the Pi.
thelittleone 3y ago
Nice to know I'm not alone. If I could reclaim time spent on rabbit hole projects that concluded with bail out, I feel I'd have a whole other lifetime. Yet somehow, each new rabbit hole project lures me in.
- deleted 3y ago
  [deleted]
elevation 3y ago
If an idling raspberry pi is too power hungry, a microcontroller like an ESP32 could be made to respond to ARPs. $8 for the MCU and a couple of afternoons of noodling. It's not going to run avahi for you though.
- cozzyd 3y ago
  Use a beaglebone black. modern rpis use too much power.
rendaw 3y ago
You could have an Arduino to wake the Pi.
- TheHippo 3y ago
  You can't sleep a Pi.
- fuzzy2 3y ago
  But then, why not just use an ESP32 which can replace the Pi entirely in the proposed setup. ;-) I would have an ESP32 impersonate the MAC address of the server passively (ie. only in the Wi-Fi association), then “push the wake button” if it found traffic that is allowed to wake the server. A wake-up firewall, if you will.
- scottyah 3y ago
  Or a "smart plug"
nottorp 3y ago
That's what I was thinking too. My home server consumes like 15 W and is silent (and saturates gigabit). If you get a rack mounted server made for data centers and stick it in a closet so you can't hear it then yes, i guess this approach makes sense.
- m463 3y ago
  > and stick it in a closet went down that route years and years ago and... it heats up the closet. Actually, (though not a server) I remember one place I lived, I just put my monitor, keyboard and mouse in the bedroom at a nice desk. Then I ran the cables through the wall to the computer in the next room where it could be loud and cool. (quiet, fast, cool: pick two)
  - nottorp 3y ago
    Haha when I was young and didn't know better i splurged on a dual socket AMD desktop. Athlon 1600M it was, I think. Ended up physically removing a CPU because the noise and heat was... not... worth... it.
gertlex 3y ago
> This looks like one of those home server infrastructure setup rabbit holes I would've bailed out of within the first 10 minutes of Googling to research the problem. Absolutely have felt the same often, in the past. Good news! There's ChatGPT now! It's so much quicker than google for figuring out approaches; it's great for explaining concepts and "how to do x with tech". And it's great at writing the scripts for you too. I set up a prometheus/grafana dashboard with scripts for various temperatures from various APIs with its help. Not trying to shill, but just sharing that this type of project is something that's a lot more feasible to implement in limited timeframes than it used to be. (and get the satisfaction of bringing the idea to fruition) But I reckon I'll always want content like this that covers the journey of uncovering various edge-cases, too...
- hammyhavoc 3y ago
  ChatGPT can't even do sensible NGINX configs, despite feeding it documentation. Gets URL rewrites wrong et al, hallucinates things that aren't even there. Troubleshooting its output is more labour-intensive than just doing it right in the first place, but most folks' Dunning-Kruger complex prevents them from knowing when the output is wrong in the first place.
- monocasa 3y ago
  So far my experience has been the opposite with ChapGPT FWIW. Overall it has cost me more time sending me down rabbit holes that it totally made up, but said with absolute confidence.
  - gertlex 3y ago
    This is interesting to hear! A key caveat in much of what I'm doing, is I'm implementing things I previously did for other projects in the past 15 years... so I'm using old technology, know the terminology, and can troubleshoot 90% correct stuff that chatGPT gives me. I haven't done as much "I have no idea where to start" stuff.
    - monocasa 3y ago
qwertox 3y ago
I had a somewhat similar setup and it worked pretty good as long as I needed it. A Raspi, which would also do other stuff, like provide the DNS service, would wake up a backup server, which was a Xeon machine and consumed a lot of power, also was running Windows. The Raspi would wake it up 4 times a day, the server had a Task set up to run some Python scripts which invoked robocopy so that the server pulled the data off of another machine which was always on and required to be backed up. When ready it shut itself down and all was good.
rektide 3y ago
Personally I like having courage to try. I don't tend to see things as requiring maintenance. This looks like an afternoon of messing around and should just work. I don't expect troubleshooting to be hard. I really love people who can show courage & interest. I feel like computing has decayed into vapid consumerism. Products no longer even try to be semi-technical. There's a whole culture built around telling & selling ourselves the idea that computers are hard & difficult. All it takes is one person with a good blog post to demystify & make accessible. You are smart enough. You can be that person. You can Augment Human Intellect by talking through it.
hulitu 3y ago
> A Raspberry Pi that itself would work just as fine as the Time Machine server the author seems to need on demand? A Raspberry Pi has one big problem: no SATA ports.
- gambiting 3y ago
  Yes but RPI 4 has usb 3.0 ports that you'd really struggle to saturate with a gigabit connection - I used to see the lack of sata as a major problem with that platform but nowadays it really doesn't matter.
  - hulitu 3y ago
    AFAIK the USB ports on RPi are shared with other peripherals. And there is no way you can connect more than 2 SATA drives without additional HW .
    - gambiting 3y ago
      On Rpi 4 the USB 3.0 ports get their own dedicated USB 3.0 controller that's connected through a PCIe 3.0 1x bus. They don't share bandwidth with anything else.
thefz 3y ago
OTOH, there's a ton to learn going down rabbit holes, and these are the reason why I stick to Linux.
fuzzy2 3y ago
I think it’s not too bad. Of course, I knew most of the ingredients before reading this article. I think my Router OS router can also do the ARP thing.
TacticalCoder 3y ago
> Especially since the setup requires a Raspberry Pi to run permanently. Which they'll gladly do no? If (poor quality) SD card corruption is a concern, you can have the RPi mount the SD card read-only or just keep a full backup of the SD card. I've had a RPi 1 power 4 VoIP Cisco phones for years. Zero issue. I've got another one constantly establishing a remote SSH-tunnel so that I can easily troubleshoot my brother, who lives in another country, 's LAN. The uptime is more than a year. One is routing (well, doing SNAT really) and firewalling between two non routable LANs. I set up the thing once and it's so reliable there's neither a screen nor a monitor hooked to it: just the power adapter and two ethernet cables (including one in a USB-to-ethernet adapter). Another one is running unbound. Same: no monitor, no keyboard, no nothing. The power adapter and one ethernet cable. Simple. Not much can go wrong. No moving parts. It's happily null routing hundreds of thousands of crappy domain names 24/7. I've got another one in a vintage arcade cab with a Pi2JAMMA adapter. In years it never failed to boot once. As we the others, just in case, I still have a "dd" of the entire SD card. Should something happen I'd just dd the image to a new SD card. The shortage of RPi is apparently due to companies snapping them all for industrial uses. I haven't heard of many people complaining "my RPi died overnight" (unlike, say, the Apple M1 laptop's screens). They're incredibly reliable pieces of equipment for their gentle price. In case something goes wrong on one of mine, I just have a dump of the entire SD card. I don't disagree that they may be a pain in the butt to set up, just like any server. But once setup, they don't move under your feet so the maintenance is really minimal. I hate that there's a shortage of Pi but I'm not sure that criticizing them as "non reliable" is a criticism made in good faith.
- JosephRedfern 3y ago
  I'm not sure OP was necessarily implying that the Pi specifically wasn't suitable for the job -- perhaps just that requiring /any/ other additional computer to achieve this is not without additional administrative overhead and power costs. It does seems likely that a Pi is going to consume a fair bit less energy than the x86 SFF PC, so it's probably a net gain there, but that admin overhead thing isn't necessarily one to ignore -- it's not entirely set and forget.
- kayge 3y ago

by DCKing 3y ago

ttanev 3y ago

My solution to the "10 years old PC used as a backup target" was to just donate/sell-off-as-parts the old PC, and replace with a 10 year old laptop, with an external USB enclosure. From 180-200 Watts down to 10-25, but it was not as much fun with two weeks of a rabbit whole digging :)

louwrentius 3y ago

I’m using A Raspberry Pi4 as a dedicated Timemachine backup server with an SSD via USB3. Works perfectly for a year or so.

dolmen 3y ago

However that requires the disk used for backup storage to also be always on.

aequitas 3y ago
Disk have power modes and can be spun down to save energy. A Raspberry Pi can run the OS on an SD card and mount the disk for backup storage. Leaving it in low power for a long period of time.
- nwb99 3y ago
  I'm worried about the wear on my drives spinning up and down so much? Maybe it isn't such a big deal. It is the most stressful time for spinning drives, though.
MattPalmer1086 3y ago
I use my pi as a backup server. The disks spin down when not in use.

dolmen 3y ago

Ideally I would like the "wake-up / fake on" infrastructure to be a feature of my wifi router.

xur17 3y ago
I wonder if you could build this as a plugin for opnsense.

txdv 3y ago

i run an rpi to host stuff for cheap (the energy drain of the pi is low), my main machine goes to sleep after 15min to save costs. seemes like a neat trick to remotely wake it up

aequitas 3y ago

I agree, but you could argue that the Raspberry Pi is not performant enough during restores or browsing the backups. During the backup itself performance should not have to be that big of an issue as it's mostly a background operation you will not be waiting on. You could also use the RPi as a stepping stone which syncs it's files to the 'real' backup server which is on a daily/weekly timer instead of on-demand setup. I'm using this concept for the problem that I want to collect metrics all day long, but not have a power hungry server running 24/7. So my RPi collects all metrics and streams then directly to my server for me to view realtime. But whenever the server is unavailable (on schedule or for maintenance) the RPi just buffers the metrics until the server is available again. Also worth considering is just not using a single server for a single purpose but just cramming it full with utility to justify the power costs.

manmal 3y ago
Then how about an N95/N100 (recent Celeron successor) mini PC, like the ones from Beelink? They don’t draw that much power, but are 2-3x faster than a Raspi, have HW supported video de/encoding and can even run Windows if you needed that.
- DJBunnies 3y ago
  "that much power" is pretty subjective, is it 2x or 10x a pi? Might be a turnoff for some.
  - manmal 3y ago
    The N100 has a TDP of 6W. I haven't measured power draw of the whole system yet though, I imagine that the mainboard will pull another couple watts at least. It shines especially in scenarios where the Pi is just a tad too slow. E.g. you can't live transcode 4k videos on a Pi, while the N100 has latest UHD graphics and therefore ships with a decent HW-accelerated H265 en/decoder.
    - nottorp 3y ago

MattPalmer1086 3y ago

Meh, I struggled to get my pi to work as a time machine server. It would work for a few weeks and then just stop working and I would have to start the backup from scratch previous backups becoming unusable. In the end I just plugged an external hard disk into the mac. I have had similar issues in the past even with such simple things as SSH to a Mac. It would be easy to blame the pi or Linux, but as far as I can tell, Apple just does not play nicely with anything not Apple. It's just that little bit different. I have no such problems interfacing Linux and the pi with anything else.

testfoobar 3y ago
Time Machine over NAS has always been unstable for me. As you experienced, my backup would be regularly corrupted and would need to restart. Alternatively, a normal <10G backup could take in excess of 24h. I eventually gave up on the NAS and now backup to external drives at home. This site has excellent articles on Time Machine - scroll down to Time Machine Problems: https://eclecticlight.co/mac-troubleshooting-summary/ https://eclecticlight.co/mac-troubleshooting-summary/
- jdminhbg 3y ago
  I had the exact same problem and ended up with the same solution. I know people who have had somewhat better experiences backing up to something like a Synology, but the $Xk investment wasn't worth it for me without also needing it for big file storage or media serving or whatever.
  - nottorp 3y ago
    > backing up to something like a Synology How about QNAP? I happen to have one gathering dust, if you can reliably Time Machine to it i'd plug it in.
- bobleeswagger 3y ago
  There are some quirks to setting up SMB so Time Machine is happy, but I have had great success with ZFS+SMB hosting my TM backups. Have been running for a little over a month and nothing is corrupt. Self hosting is still somewhat hard, I don't blame you for bailing on something so critical to uptime in the first place.

OJFord 3y ago

You could run avahi and 'a simple ruby script' on the router instead (ok not most retail routers as stock, but e.g. OpenWRT) and do away with the Pi.

thelittleone 3y ago

Nice to know I'm not alone. If I could reclaim time spent on rabbit hole projects that concluded with bail out, I feel I'd have a whole other lifetime. Yet somehow, each new rabbit hole project lures me in.

deleted 3y ago
[deleted]

elevation 3y ago

If an idling raspberry pi is too power hungry, a microcontroller like an ESP32 could be made to respond to ARPs. $8 for the MCU and a couple of afternoons of noodling. It's not going to run avahi for you though.

cozzyd 3y ago
Use a beaglebone black. modern rpis use too much power.

rendaw 3y ago

You could have an Arduino to wake the Pi.

TheHippo 3y ago
You can't sleep a Pi.
fuzzy2 3y ago
But then, why not just use an ESP32 which can replace the Pi entirely in the proposed setup. ;-) I would have an ESP32 impersonate the MAC address of the server passively (ie. only in the Wi-Fi association), then “push the wake button” if it found traffic that is allowed to wake the server. A wake-up firewall, if you will.
scottyah 3y ago
Or a "smart plug"

nottorp 3y ago

That's what I was thinking too. My home server consumes like 15 W and is silent (and saturates gigabit). If you get a rack mounted server made for data centers and stick it in a closet so you can't hear it then yes, i guess this approach makes sense.

m463 3y ago
> and stick it in a closet went down that route years and years ago and... it heats up the closet. Actually, (though not a server) I remember one place I lived, I just put my monitor, keyboard and mouse in the bedroom at a nice desk. Then I ran the cables through the wall to the computer in the next room where it could be loud and cool. (quiet, fast, cool: pick two)
- nottorp 3y ago
  Haha when I was young and didn't know better i splurged on a dual socket AMD desktop. Athlon 1600M it was, I think. Ended up physically removing a CPU because the noise and heat was... not... worth... it.

gertlex 3y ago

> This looks like one of those home server infrastructure setup rabbit holes I would've bailed out of within the first 10 minutes of Googling to research the problem. Absolutely have felt the same often, in the past. Good news! There's ChatGPT now! It's so much quicker than google for figuring out approaches; it's great for explaining concepts and "how to do x with tech". And it's great at writing the scripts for you too. I set up a prometheus/grafana dashboard with scripts for various temperatures from various APIs with its help. Not trying to shill, but just sharing that this type of project is something that's a lot more feasible to implement in limited timeframes than it used to be. (and get the satisfaction of bringing the idea to fruition) But I reckon I'll always want content like this that covers the journey of uncovering various edge-cases, too...

hammyhavoc 3y ago
ChatGPT can't even do sensible NGINX configs, despite feeding it documentation. Gets URL rewrites wrong et al, hallucinates things that aren't even there. Troubleshooting its output is more labour-intensive than just doing it right in the first place, but most folks' Dunning-Kruger complex prevents them from knowing when the output is wrong in the first place.
monocasa 3y ago
So far my experience has been the opposite with ChapGPT FWIW. Overall it has cost me more time sending me down rabbit holes that it totally made up, but said with absolute confidence.
- gertlex 3y ago
  This is interesting to hear! A key caveat in much of what I'm doing, is I'm implementing things I previously did for other projects in the past 15 years... so I'm using old technology, know the terminology, and can troubleshoot 90% correct stuff that chatGPT gives me. I haven't done as much "I have no idea where to start" stuff.
  - monocasa 3y ago

qwertox 3y ago

I had a somewhat similar setup and it worked pretty good as long as I needed it. A Raspi, which would also do other stuff, like provide the DNS service, would wake up a backup server, which was a Xeon machine and consumed a lot of power, also was running Windows. The Raspi would wake it up 4 times a day, the server had a Task set up to run some Python scripts which invoked robocopy so that the server pulled the data off of another machine which was always on and required to be backed up. When ready it shut itself down and all was good.

rektide 3y ago

Personally I like having courage to try. I don't tend to see things as requiring maintenance. This looks like an afternoon of messing around and should just work. I don't expect troubleshooting to be hard. I really love people who can show courage & interest. I feel like computing has decayed into vapid consumerism. Products no longer even try to be semi-technical. There's a whole culture built around telling & selling ourselves the idea that computers are hard & difficult. All it takes is one person with a good blog post to demystify & make accessible. You are smart enough. You can be that person. You can Augment Human Intellect by talking through it.

hulitu 3y ago

> A Raspberry Pi that itself would work just as fine as the Time Machine server the author seems to need on demand? A Raspberry Pi has one big problem: no SATA ports.

gambiting 3y ago
Yes but RPI 4 has usb 3.0 ports that you'd really struggle to saturate with a gigabit connection - I used to see the lack of sata as a major problem with that platform but nowadays it really doesn't matter.
- hulitu 3y ago
  AFAIK the USB ports on RPi are shared with other peripherals. And there is no way you can connect more than 2 SATA drives without additional HW .
  - gambiting 3y ago
    On Rpi 4 the USB 3.0 ports get their own dedicated USB 3.0 controller that's connected through a PCIe 3.0 1x bus. They don't share bandwidth with anything else.

thefz 3y ago

OTOH, there's a ton to learn going down rabbit holes, and these are the reason why I stick to Linux.

fuzzy2 3y ago

I think it’s not too bad. Of course, I knew most of the ingredients before reading this article. I think my Router OS router can also do the ARP thing.

TacticalCoder 3y ago

> Especially since the setup requires a Raspberry Pi to run permanently. Which they'll gladly do no? If (poor quality) SD card corruption is a concern, you can have the RPi mount the SD card read-only or just keep a full backup of the SD card. I've had a RPi 1 power 4 VoIP Cisco phones for years. Zero issue. I've got another one constantly establishing a remote SSH-tunnel so that I can easily troubleshoot my brother, who lives in another country, 's LAN. The uptime is more than a year. One is routing (well, doing SNAT really) and firewalling between two non routable LANs. I set up the thing once and it's so reliable there's neither a screen nor a monitor hooked to it: just the power adapter and two ethernet cables (including one in a USB-to-ethernet adapter). Another one is running unbound. Same: no monitor, no keyboard, no nothing. The power adapter and one ethernet cable. Simple. Not much can go wrong. No moving parts. It's happily null routing hundreds of thousands of crappy domain names 24/7. I've got another one in a vintage arcade cab with a Pi2JAMMA adapter. In years it never failed to boot once. As we the others, just in case, I still have a "dd" of the entire SD card. Should something happen I'd just dd the image to a new SD card. The shortage of RPi is apparently due to companies snapping them all for industrial uses. I haven't heard of many people complaining "my RPi died overnight" (unlike, say, the Apple M1 laptop's screens). They're incredibly reliable pieces of equipment for their gentle price. In case something goes wrong on one of mine, I just have a dump of the entire SD card. I don't disagree that they may be a pain in the butt to set up, just like any server. But once setup, they don't move under your feet so the maintenance is really minimal. I hate that there's a shortage of Pi but I'm not sure that criticizing them as "non reliable" is a criticism made in good faith.

JosephRedfern 3y ago
I'm not sure OP was necessarily implying that the Pi specifically wasn't suitable for the job -- perhaps just that requiring /any/ other additional computer to achieve this is not without additional administrative overhead and power costs. It does seems likely that a Pi is going to consume a fair bit less energy than the x86 SFF PC, so it's probably a net gain there, but that admin overhead thing isn't necessarily one to ignore -- it's not entirely set and forget.
kayge 3y ago