8 ms·

A Raspberry Pi as a decent residential proxy

98 points by AlexITC 6y ago

tzs 6y ago
> Don’t expose the proxy to the world or attackers will be able to interact with your home devices Or use it to get stuff on the internet that they don't want to get directly themselves. Once upon a time I accidentally enabled open proxying on the web server on my Linux box. This was before smart phones and tablets and such, when my Linux box was the only network client I had, and so I didn't have a NAT router. Just modem straight to Linux box. One day I checked my Apache logs to see if my extremely low traffic web site had any visitors. First impression: "What the hell is all this traffic? This is orders of magnitude more than I should have!". Second impression: "Why the hell is someone trying to get horse_fucking.mov from my server?". Third impression: "That didn't give a 404. Why is my server actually providing horse_fucking.mov?". (No, the orders of magnitude extra traffic were not all people getting horse_fucking.mov. There was a whole plethora of bestiality titles, plus a lot of other porn that you would not want to fetch directly from your own IP address either out of embarrassment if you got caught or because it was almost certainly illegal).
- AlexITC 6y ago
  > Or use it to get stuff on the internet that they don't want to get directly themselves That's a good point, another one is people using your proxy to attack other servers. I find the proxy area to be interesting but something I'm unlikely to use for business, you are quoting some examples that made me laugh, and I'd certainly laugh if it happens to me, but I know I will remove the public access, if I were involved in business with this, the story would be different. Thanks for sharing.
- oefrha 6y ago
  I once accidentally left privoxy open on one of my Linux servers and misconfigured ACL. A couple hours later it started getting hit with loads of traffic. A Google search quickly revealed that my ip:port ended up on several “free proxy” lists. They probably found it on Shodan or something. I quickly fixed acl to drop the traffic, but IIRC my server persisted on the free proxy lists for a couple days afterwards. That was the first hand experience that confirmed my suspicions of what those “free proxies” really are.
  - AlexITC 6y ago
    Actually, you are giving one advantage to the custom-made proxy, it's not easily discoverable by bots, nor it could be used right away.
    - heavyset_go 6y ago
      Just using non-default settings for your proxy would be enough to thwart 99.9% of automated attacks, and you wouldn't have to rely on a hand-rolled proxy.
      - Topgamer7 6y ago
        This is the same for ssh. If you want to decrease log size, change the port used away from default.
- AlexITC 6y ago
  Adding on this, I just checked the logs and looks like some persons started playing: Calling: https://nsa.gov, headers = BOMBTHEMOSQUE -> 911 Calling: https://fbi.net Calling: file:///bin/sh Calling: file:///etc/passwd Calling: http://127.0.0.1:80
- neilv 6y ago
  20 years ago, I thought it would be cool to monitor Web and other port traffic on my Linux home PC by logging it to my X root window... That's when I realized from the scrolling log messages that even residental ADSL was already under heavy probing from around the world. I decided I didn't have the time or the stomach to try to deal with that at home, and there wasn't much I could do, so I disabled the logging.
yozel 6y ago
Why not use a normal HTTP proxy like Squid? It would be more easy to use it since almost all languages and clients support HTTP proxy built-in.
- AlexITC 6y ago
  I evaluated Squid which was my first option, due to the existing code on the project I integrated the proxy, it wasn't that straightforward to interact with Squid. I'm still thinking to use several Pies and it's simpler to evolve the custom-made proxy to support that, if you see the Github repo, it's pretty simple. Last, getting a Scala project to run on the old Pi was a nice experiment worth trying.
iamphilrae 6y ago
I run a number of news website scrapers for various projects. Originally I ran them all on AWS EC2 instances and racked up sizeable bills. Recently though, I pulled an unused Raspberry Pi from my desk drawer, installed them all onto the Pi, and now run them for near free from my desk! Granted I need to be a little bit more efficient with memory and processor cycles, but that’s a fun little challenge. I find if you respect the websites in terms of not hitting them multiple times a minute (or only once an hour which I do) then you’ll very seldom get banned.
- AlexITC 6y ago
  It's interesting how simple is to get small services out of the cloud now. I'm not that worried about websites banning the Pi as the traffic is really low, on concurrent calls it's unlikely to be a problem as I don't expect the Pi to query several websites in parallel. Certainly it's something worth implementing if I ever start getting more traffic. Thanks for sharing.
- city41 6y ago
  I did the same thing when I was running a scraper. I throttled the requests quite low to be respectful so it took a while to gather all the data, but it worked just fine. For this article, does the OP feel they need the proxy due to scraping rapidly?
  - AlexITC 6y ago
    The problem isn't the speed, if it were, I would certainly use something more powerful than an old Pi. The actual problem is that some websites block IPs from known cloud providers, so, it doesn't matter which headers you use because what's blacklisted is the IP address.
    - city41 6y ago
      Myself and the person I was replying to ran our scrapers from our homes. One benefit being the IP address doesn’t get flagged. Curious why scraping from your home network doesn’t work for you?
ysleepy 6y ago
Why not just use the SSH dynamic SOCKS5 proxy? Just ssh -D 12345 raspberry and then just use localhost:12345 as the scrapers SOCKS5 proxy. No need to involve a http proxy. EDIT: AAh, ssh port is not accessible from outside and the raspi does a reverse connect.
- AlexITC 6y ago
  That can certainly work. A reason to writing the custom-made proxy is mostly to evolve it supporting several pies, the tunnel being the simpler workaround for now. Also, I'm a coder and wanted to try getting Scala running on my old pi.
amaccuish 6y ago
I still keep a server running in my basement doing WireGuard since Netflix et al. all block cloud IPs/ASNs
Snawoot 6y ago
> Surprisingly, 256 MB in RAM have been enough to maintain the app running for months until now. What is total RSS of all your proxy processes?
- AlexITC 6y ago
  What's RSS?
  - Snawoot 6y ago
    Resident Set Size, a value reported by programs like top/htop with amount of physical RAM used by process.
- AlexITC 6y ago
  $ free -h total used free shared buff/cache available Mem: 432Mi 194Mi 26Mi 20Mi 211Mi 164Mi Swap: 99Mi 2.0Mi 97Mi $ top Tasks: 69 total, 1 running, 68 sleeping, 0 stopped, 0 zombie %Cpu(s): 7.3 us, 4.3 sy, 0.0 ni, 88.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 432.4 total, 26.2 free, 194.7 used, 211.5 buff/cache MiB Swap: 100.0 total, 98.0 free, 2.0 used. 164.7 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2915 pi 20 0 299.9m 173.8m 13.6m S 1.6 40.2 447:45.82 java 32009 pi 20 0 7.5m 2.6m 2.2m R 1.0 0.6 0:00.31 top 31991 pi 20 0 11.9m 3.8m 3.0m S 0.3 0.9 0:00.10 sshd
  - Snawoot 6y ago
    Thank you!
  - boring_twenties 6y ago
    Don't forget to set gpu_mem in config.txt. It looks like you are running with the default, which is 64MB for the GPU. I'm guessing you are not using the GPU in this application and should reduce this value to the bare minimum (which I don't recall, sorry)
hjb 6y ago
That is a very retro raspberry pi :-D
- AlexITC 6y ago
  Indeed, I was planning to buy one until my wife complained that I already bought one years ago, it was worth trying.
vivekf 6y ago
Have you looked at running wireguard on your server and connecting to it from the pi. Then you have a vpn between server and the pi where you can send data in any protocol
- AlexITC 6y ago
  I haven't but I'll think about it, thanks for the idea. Mainly the custom-made proxy allows scaling the app to several pies if I ever need to.
- bonyt 6y ago
  I do this to expose services to the internet in a limited way. Wireguard link between a pi running, say, octoprint, and a vps. Then, the vps is running nginx or caddy as a reverse proxy over that wireguard link, giving me https access and even letting me add basic auth if I want another layer of authentication.
  - kawsper 6y ago
    It's a shame that client-certificates is implemented in such a clunky way in most browsers and operating systems, because that could also be an easy way to achieve this even without installing anything.
donaldihunter 6y ago
For more reliable ssh tunnels you can use autossh. https://www.harding.motd.ca/autossh/ https://www.harding.motd.ca/autossh/
- AlexITC 6y ago
  The first time I hit an issue, autossh was the first proposed solution but I preferred to not include more dependencies. Also, I believed it should be possible to handle this with plain ssh + systemd. Do you have any experiences with autossh/ssh to share?
Ralo 6y ago
I setup an Openvpn server on my LAN, with username/password credentials so I can access my LAN from anywhere. I have multiple vpn configurations for my cell phone/laptop to either map only 192.x through the VPN or all traffic through it. It's very very useful, and seems like a much more secure option in my opinion.
- AlexITC 6y ago
  Can you please elaborate on why it's more secure?
tails4e 6y ago
Is there a solid solution to the SD card problem? Every Rpi I've tried to use as a server eventually kills its SD card, probably as running an os on SD is destined for failure with write wear. I'm guessing using SD for boot only and then using a home NAS would be an option, but I've not come across a dummies guide to making a Rpi reliable long term.
- Gibbon1 6y ago
  Years ago a friend mentioned her company would spring for very expensive industrial grade SD cards. My company up and refuses to have anything to do with SD cards.
- kiliankoe 6y ago
  I used to run the boot partition from the SD card and everything else over a mounted USB flash drive. That seemed to work quite ok for a long time, but maybe I just got lucky.
- deeblering4 6y ago
  Spending a little extra on good quality SD cards helps both durability and performance. Also making a backup image of the SD card once configured and using a NAS or external disk for important info helps mitigate the data loss risk, and gives you a quick path to restore onto a fresh SD. With that said, I have a few raspberry pis that have been running for years without SD failures. Also, make sure swap is not enabled on the SD

98 points by AlexITC 6y ago

tzs 6y ago

> Don’t expose the proxy to the world or attackers will be able to interact with your home devices Or use it to get stuff on the internet that they don't want to get directly themselves. Once upon a time I accidentally enabled open proxying on the web server on my Linux box. This was before smart phones and tablets and such, when my Linux box was the only network client I had, and so I didn't have a NAT router. Just modem straight to Linux box. One day I checked my Apache logs to see if my extremely low traffic web site had any visitors. First impression: "What the hell is all this traffic? This is orders of magnitude more than I should have!". Second impression: "Why the hell is someone trying to get horse_fucking.mov from my server?". Third impression: "That didn't give a 404. Why is my server actually providing horse_fucking.mov?". (No, the orders of magnitude extra traffic were not all people getting horse_fucking.mov. There was a whole plethora of bestiality titles, plus a lot of other porn that you would not want to fetch directly from your own IP address either out of embarrassment if you got caught or because it was almost certainly illegal).

AlexITC 6y ago
> Or use it to get stuff on the internet that they don't want to get directly themselves That's a good point, another one is people using your proxy to attack other servers. I find the proxy area to be interesting but something I'm unlikely to use for business, you are quoting some examples that made me laugh, and I'd certainly laugh if it happens to me, but I know I will remove the public access, if I were involved in business with this, the story would be different. Thanks for sharing.
oefrha 6y ago
I once accidentally left privoxy open on one of my Linux servers and misconfigured ACL. A couple hours later it started getting hit with loads of traffic. A Google search quickly revealed that my ip:port ended up on several “free proxy” lists. They probably found it on Shodan or something. I quickly fixed acl to drop the traffic, but IIRC my server persisted on the free proxy lists for a couple days afterwards. That was the first hand experience that confirmed my suspicions of what those “free proxies” really are.
- AlexITC 6y ago
  Actually, you are giving one advantage to the custom-made proxy, it's not easily discoverable by bots, nor it could be used right away.
  - heavyset_go 6y ago
    Just using non-default settings for your proxy would be enough to thwart 99.9% of automated attacks, and you wouldn't have to rely on a hand-rolled proxy.
    - Topgamer7 6y ago
      This is the same for ssh. If you want to decrease log size, change the port used away from default.
AlexITC 6y ago
Adding on this, I just checked the logs and looks like some persons started playing: Calling: https://nsa.gov, headers = BOMBTHEMOSQUE -> 911 Calling: https://fbi.net Calling: file:///bin/sh Calling: file:///etc/passwd Calling: http://127.0.0.1:80
neilv 6y ago
20 years ago, I thought it would be cool to monitor Web and other port traffic on my Linux home PC by logging it to my X root window... That's when I realized from the scrolling log messages that even residental ADSL was already under heavy probing from around the world. I decided I didn't have the time or the stomach to try to deal with that at home, and there wasn't much I could do, so I disabled the logging.

yozel 6y ago

Why not use a normal HTTP proxy like Squid? It would be more easy to use it since almost all languages and clients support HTTP proxy built-in.

AlexITC 6y ago
I evaluated Squid which was my first option, due to the existing code on the project I integrated the proxy, it wasn't that straightforward to interact with Squid. I'm still thinking to use several Pies and it's simpler to evolve the custom-made proxy to support that, if you see the Github repo, it's pretty simple. Last, getting a Scala project to run on the old Pi was a nice experiment worth trying.

iamphilrae 6y ago

I run a number of news website scrapers for various projects. Originally I ran them all on AWS EC2 instances and racked up sizeable bills. Recently though, I pulled an unused Raspberry Pi from my desk drawer, installed them all onto the Pi, and now run them for near free from my desk! Granted I need to be a little bit more efficient with memory and processor cycles, but that’s a fun little challenge. I find if you respect the websites in terms of not hitting them multiple times a minute (or only once an hour which I do) then you’ll very seldom get banned.

AlexITC 6y ago
It's interesting how simple is to get small services out of the cloud now. I'm not that worried about websites banning the Pi as the traffic is really low, on concurrent calls it's unlikely to be a problem as I don't expect the Pi to query several websites in parallel. Certainly it's something worth implementing if I ever start getting more traffic. Thanks for sharing.
city41 6y ago
I did the same thing when I was running a scraper. I throttled the requests quite low to be respectful so it took a while to gather all the data, but it worked just fine. For this article, does the OP feel they need the proxy due to scraping rapidly?
- AlexITC 6y ago
  The problem isn't the speed, if it were, I would certainly use something more powerful than an old Pi. The actual problem is that some websites block IPs from known cloud providers, so, it doesn't matter which headers you use because what's blacklisted is the IP address.
  - city41 6y ago
    Myself and the person I was replying to ran our scrapers from our homes. One benefit being the IP address doesn’t get flagged. Curious why scraping from your home network doesn’t work for you?

ysleepy 6y ago

Why not just use the SSH dynamic SOCKS5 proxy? Just ssh -D 12345 raspberry and then just use localhost:12345 as the scrapers SOCKS5 proxy. No need to involve a http proxy. EDIT: AAh, ssh port is not accessible from outside and the raspi does a reverse connect.

AlexITC 6y ago
That can certainly work. A reason to writing the custom-made proxy is mostly to evolve it supporting several pies, the tunnel being the simpler workaround for now. Also, I'm a coder and wanted to try getting Scala running on my old pi.

amaccuish 6y ago

I still keep a server running in my basement doing WireGuard since Netflix et al. all block cloud IPs/ASNs

Snawoot 6y ago

> Surprisingly, 256 MB in RAM have been enough to maintain the app running for months until now. What is total RSS of all your proxy processes?

AlexITC 6y ago
What's RSS?
- Snawoot 6y ago
  Resident Set Size, a value reported by programs like top/htop with amount of physical RAM used by process.
AlexITC 6y ago
$ free -h total used free shared buff/cache available Mem: 432Mi 194Mi 26Mi 20Mi 211Mi 164Mi Swap: 99Mi 2.0Mi 97Mi $ top Tasks: 69 total, 1 running, 68 sleeping, 0 stopped, 0 zombie %Cpu(s): 7.3 us, 4.3 sy, 0.0 ni, 88.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 432.4 total, 26.2 free, 194.7 used, 211.5 buff/cache MiB Swap: 100.0 total, 98.0 free, 2.0 used. 164.7 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2915 pi 20 0 299.9m 173.8m 13.6m S 1.6 40.2 447:45.82 java 32009 pi 20 0 7.5m 2.6m 2.2m R 1.0 0.6 0:00.31 top 31991 pi 20 0 11.9m 3.8m 3.0m S 0.3 0.9 0:00.10 sshd
- Snawoot 6y ago
  Thank you!
- boring_twenties 6y ago
  Don't forget to set gpu_mem in config.txt. It looks like you are running with the default, which is 64MB for the GPU. I'm guessing you are not using the GPU in this application and should reduce this value to the bare minimum (which I don't recall, sorry)

hjb 6y ago

That is a very retro raspberry pi :-D

AlexITC 6y ago
Indeed, I was planning to buy one until my wife complained that I already bought one years ago, it was worth trying.

vivekf 6y ago

Have you looked at running wireguard on your server and connecting to it from the pi. Then you have a vpn between server and the pi where you can send data in any protocol

AlexITC 6y ago
I haven't but I'll think about it, thanks for the idea. Mainly the custom-made proxy allows scaling the app to several pies if I ever need to.
bonyt 6y ago
I do this to expose services to the internet in a limited way. Wireguard link between a pi running, say, octoprint, and a vps. Then, the vps is running nginx or caddy as a reverse proxy over that wireguard link, giving me https access and even letting me add basic auth if I want another layer of authentication.
- kawsper 6y ago
  It's a shame that client-certificates is implemented in such a clunky way in most browsers and operating systems, because that could also be an easy way to achieve this even without installing anything.

donaldihunter 6y ago

For more reliable ssh tunnels you can use autossh. https://www.harding.motd.ca/autossh/ https://www.harding.motd.ca/autossh/

AlexITC 6y ago
The first time I hit an issue, autossh was the first proposed solution but I preferred to not include more dependencies. Also, I believed it should be possible to handle this with plain ssh + systemd. Do you have any experiences with autossh/ssh to share?

Ralo 6y ago

I setup an Openvpn server on my LAN, with username/password credentials so I can access my LAN from anywhere. I have multiple vpn configurations for my cell phone/laptop to either map only 192.x through the VPN or all traffic through it. It's very very useful, and seems like a much more secure option in my opinion.

AlexITC 6y ago
Can you please elaborate on why it's more secure?

tails4e 6y ago

Is there a solid solution to the SD card problem? Every Rpi I've tried to use as a server eventually kills its SD card, probably as running an os on SD is destined for failure with write wear. I'm guessing using SD for boot only and then using a home NAS would be an option, but I've not come across a dummies guide to making a Rpi reliable long term.

Gibbon1 6y ago
Years ago a friend mentioned her company would spring for very expensive industrial grade SD cards. My company up and refuses to have anything to do with SD cards.
kiliankoe 6y ago
I used to run the boot partition from the SD card and everything else over a mounted USB flash drive. That seemed to work quite ok for a long time, but maybe I just got lucky.
deeblering4 6y ago
Spending a little extra on good quality SD cards helps both durability and performance. Also making a backup image of the SD card once configured and using a NAS or external disk for important info helps mitigate the data loss risk, and gives you a quick path to restore onto a fresh SD. With that said, I have a few raspberry pis that have been running for years without SD failures. Also, make sure swap is not enabled on the SD