10 ms·

Surviving the front page of Hacker News on a 50 Mbps uplink

261 points by hddherman 5y ago

hddherman 5y ago
Wayback Machine link, just in case: https://web.archive.org/web/20220209050643/https://ounapuu.ee/posts/2022/02/09/hn-stats-analytics/ https://web.archive.org/web/20220209050643/https://ounapuu.e...
charcircuit 5y ago
There didn't seem to be any data in the article analyzing how much bandwidth was being used under load. I'm not sure if for example compressing those images made it possible to now survive.
- hddherman 5y ago
  Good point, I’ll try to see if I can visualise that based on the logs that I have.
bawolff 5y ago
I wonder how much bandwidth a hn hug normally is. 50 mbps sounds like a lot to me unless you have heavy images/movies/not text, but i'm out of my area of expertise.
- mrb 5y ago
  Some of my blog posts have been #1 on HN for a couple hours. I have lightweight pages, about 100 kB including images and all resources. IIRC traffic peaked at 30 page hits per second. So about 3 MB/s or 24 Mbit/s
ComputerGuru 5y ago
This is a joke, right? We colo at a location where the standard is 100mbit and the websites hosted there are doing just fine. Just slim down your pages and stop generating 25MiB landing page bundles.
skilled 5y ago
Ok, what is the color theme you're using for the terminal?
- S0und 5y ago
  Judging by the colors: it's monokai
  - jcun4128 5y ago
    Love Monokai Pro Spectrum been using it for a few years now in VSC
- Godel_unicode 5y ago
  That's just what goaccess looks like, it's pretty sweet though: goaccess.io
  - skilled 5y ago
    Thanks!
  - ZuLuuuuuu 5y ago
    I was expecting that it is written in Go (because of the name), but apparently it is written in C. Looks like a nice program.
    - re 5y ago
      I had assumed that too, but guess it's G.O. Access, for the author's initials: https://github.com/allinurl https://github.com/allinurl
- jamil7 5y ago
  Asking the important questions
invalidator 5y ago
I haven't been hugged by HN yet, but I've survived being featured on a couple other front pages long ago. It's easy if it's a lightweight page and you have something faster than a 486. If it's heavy (lots of images, videos, downloads, etc) you can run out of bandwidth pretty quickly. However, that doesn't necessarily sink the ship! Traffic shaping makes a big difference. Without it, the modem buffer quickly fills causing multi-second ping times. This easily leads to congestion collapse. TCP grows the receive window to large values, and the congestion avoidance algorithm doesn't do well when the feedback is delayed due to the latency. TCP starts sawtoothing, and you end up with occasional moments where a few pages get served, interspersed with so much loss that connections get dropped. We used htb [1] which tightly controls the outbound traffic rate, preferring to drop packets instead of delay them. That prevents the modem's buffer from filling up, which keeps the latency down. This creates a smoother flow. Pages load slowly, but at least they keep loading, instead of wasting bandwidth on retries. It also prioritizes traffic: other services are guaranteed their minimum rates. We were using ssh through the line and didn't even realize we were getting hit until someone asked why the site was running slowly. Newer algorithms are even better, especially for keeping jitter down, but htb is simple and vastly better than letting your router blast packets at full speed. OpenWRT and all the other open source router firmwares can do it these days. It improves things for even normal end-user uses, but if you're running a server at home, you really want to set up some QOS before someone hugs you to death. [1] https://linux.die.net/man/8/tc-htb https://linux.die.net/man/8/tc-htb
- zekica 5y ago
  Codel, Fq_Codel and especially Cake are worth looking into.
  - throwaway48375 5y ago
    I wish OPNsense had cake.
  - mrg2k8 5y ago
    I used tc, CAKE and ifb for a small ISP router-on-a-stick implementation (around 2000 end-users) and it did wonders with the limited bandwidth I had to work with (around 3Gbps). More simple to configure than HTB, no more playing around with lots of knobs. Even during high peak traffic the latency would stay low.
tsak 5y ago
I'm using the exact same setup as yourself. I have a script that runs every 10 minutes that checks if my external IP has changed and then updates my DNS records via the GoDaddy API. I also decided to use Hugo to generate my blog (https://tsak.dev https://tsak.dev) And I use `goaccess` to check my logs as well.
- nicbou 5y ago
  There is a similar docker image that does this for DigitalOcean.
miyuru 5y ago
Any story on why the blog does not support IPv6? Your ISP seems to support it. https://stats.labs.apnic.net/ipv6/AS2586 https://stats.labs.apnic.net/ipv6/AS2586 I am asking because you did not even mention it in the "Residential connections and DNS" section. Depending on the ISP you might get a static IPv6 prefix.
- hddherman 5y ago
  My ISP only hands out an IPv4 address, unfortunately. Once I can change to another provider here, I’ll also be able to get an IPv6 address.
  - miyuru 5y ago
    thats a bummer. thanks for the info!
rufname 5y ago
Since it appears to be a statically generated site it might not matter, but do you use any caching in your NGINX config?
- hddherman 5y ago
  Not entirely sure, since I rely on the defaults provided by LinuxServer.io SWAG Docker image. The backing store for my site runs on ZFS though and it’s highly likely that web assets are cached in memory.
jfkimmes 5y ago
Quick Tip for your <figure> inconvenience with markdown: Just write a script that copies whatever is in your clipboard to a ./images/ directory and replace the content of your clipboard with the markdown you usually write pointing to that image. See my hacky version, here [1]. I have that on a keyboard shortcut but you can also just alias/source it and run it when needed. [1]: https://gitlab.com/jfkimmes/dotfiles/-/blob/master/scripts/markdown_image_include.sh https://gitlab.com/jfkimmes/dotfiles/-/blob/master/scripts/m... (this requires an `activate` command that symlinks my current project directory to a path in my home directory so that all my scripts can work there. You can probably make this more elegant but it works for me.)
calmingsolitude 5y ago
I've had the experience of being on the opposite end of the spectrum when I made a joke site[0] that kept the connection open to every connected browser. Needless to say, it got hugged to death in the very first ten minutes. What got it back up was switching from my homebrew webserver to Nginx and increasing the file descriptor limits. Switching to a battle-tested webserver made all the difference. I'm still not sure how many connections it can keep open now, but I don't have to worry about the static parts of the site going down. [0] https://youcantdownloadthisimage.online/ https://youcantdownloadthisimage.online/
- TT-392 5y ago
  Downloading your mona lisa worked just fine on my device (Android), in whatever browser telegram opens links in.
  - deleted 5y ago
    [deleted]
- unnouinceput 5y ago
  Firefox user here. Right click on image -> Open image in new tab. Then kill the connection and could "Save Image as" just fine. Or print screen it and get it in PBrush.
  - foxfluff 5y ago
    > Firefox user here. Right click on image -> Open image in new tab. Then kill the connection and could "Save Image as" just fine. That doesn't work for me.. Firefox just starts a new download for the image and it never finishes. Sometimes it seems to flush about 32KB to the disk, giving you half of the image, but the rest must still be waiting in buffers.
    - unnouinceput 5y ago
      Maybe because in my case Firefox is augmented with uBlock Origin + NoScript + Privacy Badger? I don't know, maybe because I'm on desktop and you're on mobile? No idea, but that's how I can do it.
moreentropy 5y ago
I'm in the same situation (50 Mbps uplink at one place, 100Mbps at another) and that's enough to do all hosting for hobby projects at home, which I really love. Instead of exposing my home directly using DynDNS, I got a really cheap low end server (currently a VM w/ 1 CPU, 512MB RAM and 400mbps traffic flat) for 1€/month that proxies all traffic to the (hidden) servers hosted at home. The spec is enough to reliably run a HAproxy that can max out the available bandwidth w/o sweat and it allows me to do failover between servers in my two "home datacenters" + possibly cache assets.
- dTal 5y ago
  May I ask where you rent your ultra-cheap server from?
  - moreentropy 5y ago
    I'm using a Ionos (oneandone) VPS S: https://www.ionos.de/server/vps https://www.ionos.de/server/vps 1&1/Ionos is one of the largest and best connected ISP and Hosting providers in Germany, not some small shady shop. I see they doubled the price to 2€ though :)
    - wackro 5y ago
      I made a WordPress site for a friend through Ionos's managed offering. After a couple of months of the site being up, they decided to detach the database for literally no reason. I had to go through support to get it back up. I'm guessing it got erroneously flagged as over its quota, but they couldn't explain why it happened. Just my experience.
    - sirsinsalot 5y ago
      I've had nothing but hell from 1&1/ionos ... if your billing has issues, be prepared to be treated like a criminal with few paths to easy resolution.
      - littlecranky67 5y ago
zuzun 5y ago
You can get the web fonts below 20 KB each if you subset them. You can also check if they are present on the system with `local()` before `url()` in the font-face declaration. You could also give the Apple system fonts a higher priority than Inter, because Inter and San Francisco are pretty similar.
adam-a 5y ago
I did run a few semi-private sites from my home but I soon found the incoming traffic was causing my outbound connection to fall over. Lots of dropped video calls and spotty vpn connections. I assume because my ISP provided router was not handling the incoming requests very well. I wonder if anyone here has any good advice to get this sort of setup to run well?
- hddherman 5y ago
  Do you use your own router behind the box provided by the ISP? And is it DOCSIS? If so, using the ISP-provided box in bridge mode could help. This was the cause of many woes in my setup, whenever there were a lot of connections open, the internet connectivity would drop off. But once I changed the ISP-provided box to be a dumb bridge between the internet and my own router, this stopped being an issue.
  - adam-a 5y ago
    I was just using the ISP router on its own. Any recommendations for a good router?
    - hddherman 5y ago
      If you're into tinkering, then anything that has good-enough specs and has great openWRT support is a safe bet. Personally I use a TP-Link Archer C7 v5, but there might be better options out there by now.
  - xuki 5y ago
    Yeah, always bridge the ISP box if possible. Get an EdgeRouter or run a decent router with OpenWRT or run pfSense/OPNSense on a spare x86 machine, those should be plenty fast for moderate usage.
marginalia_nu 5y ago
I've been hugged twice on 100 mbit domestic broadband, and it's honestly not been all that bad. In general I've seen somewhere between 1-5 visitors per second, a couple of million over a day. I think keeping the number of requests per visit down is a big key to surviving this traffic with low resources. The first hug was search.marginalia.nu, which used to load fonts and had a load of about 50 Kb/visit; but still only generated in maybe 4-5 GETs per visit. I measured about 2 Mbps. The second time was search.marginalia.nu/browse/random, which loads 75 images at a resolution of 512x348 (mostly WebP; so anywhere between 2 and 30 Kb). These were randomly picked from a very large pool, so they can't be meaningfully cached. I presciently set them to lazy loading and I think that made a big difference, especially from mobile visitors as they'd only load a few at a time. I didn't measure the bandwidth usage at this point, but it stayed up and stayed responsive throughout the day. I think something that possibly also makes a difference is that I don't do any sort of session cookies or mutable non-ephemeral state in my backend. I can only imagine having to validate sessions and manage access/roles for every request would make for a lot more work. I don't think it's necessarily the bandwidth that's going to be the bottleneck, rather than something like contested locks in the database, resulting in back pressure in turn causing traffic jam of lingering connections. Reducing mutable server state helps a lot with that.
- sgtnoodle 5y ago
  I made a very pretty retro control panel for Kerbal Space Program a few years back, and made a nice writeup on my personal website about it, hosted from a low power Intel baytrail based server I keep in my garage. It got picked up by hackaday, then popular mechanics and a few other similar sites, and hacker news. I vaguely recall my upload rate was about 5 Mbps at the time, and my connection was quite saturated for a few days. The page has half a dozen jpg photos, so I suspect that was the bulk of the data.
- _xnmw 5y ago
  Wait, what, one of the best text search engines for the whole internet runs on a residential broadband connection on a home server? My mind is blown.
  - 5y ago
at0mic22 5y ago
If you struggle with bandwidth, why not serving gzipped content? By response headers it looks like you are serving plain html.
- jspash 5y ago
  Self-hosting and service 300kb for font files also seems excessive. Especially for a Helvetica clone.
  - hddherman 5y ago
    Yep, that's why I am planning on getting rid of them completely. I did a quick test this morning, seems like removing references to the fonts from the theme and adjusting some font weights should do the trick.
  - nicbou 5y ago
    I use type subsetting and only serve about 50kb worth of fonts (4 variants in total). 300kb seems excessive.
- MrDOS 5y ago
  Actually, it looks to me like the HTML response is the only thing that's gzip'd. It doesn't make sense to waste time gzip'ing the PNGs, but the CSS and JS files should definitely be compressed. (And maybe the fonts? Not sure.) The default `gzip_types` setting in the nginx config shipped by Debian/Ubuntu does include `text/css` and `application/javascript`, but not `text/html`. Could be that OP is using a non-distro config that only enables gzip for `text/html`, or that OP accidentally disabled gzip'ing for the other MIME types while enabling it for HTML.
thomasfromcdnjs 5y ago
Sidenote: You pretty much generated traffic analytics anyway, and nginx logs have referrer information if you want to see where the traffic is coming from.
wink 5y ago
Author named ~70k hits per day. If we calculate 72000/12 that's 6000 - which is 1.6 req/s. Why are we talking about "surviving" anything here? Here some numbers from a Wordpress installation on a 3 EUR Scaleway VPS where zero time was spent optimizing anything: $ siege -c 3 -r 10 https://example.org ** SIEGE 4.0.4 ** Preparing 3 concurrent users for battle. The server is now under siege... Transactions: 360 hits Availability: 100.00 % Elapsed time: 13.89 secs Data transferred: 24.65 MB Response time: 0.11 secs Transaction rate: 25.92 trans/sec
- xuki 5y ago
  People see servers with multi Gbps and think it’s a requirement for high traffic websites. Unless you’re serving videos, bandwidth is almost never the bottleneck.
  - Nextgrid 5y ago
    High bandwidth protects you against basic DoS attacks that just fill up the pipe with garbage traffic. The other problem is that bandwidth is usually harder to scale than let's say CPU or memory, so having a large buffer there is helpful. Let's say the aforementioned VPS is suddenly under a lot of CPU load and legitimately can't keep up - Nginx can be trivially installed on it to load-balance between many downstream VPSes. Now let's say you are running out of bandwidth. You may need to switch to a different connection with a different IP, change DNS records and wait for them to propagate, etc.
- hddherman 5y ago
  It's mainly down to these points: * I've seen a lot of sites fall over when posted on HN or other link aggregators, like Reddit. * Cloud providers advertise gigabit or bigger pipes for networking, 50 Mbps is not a lot in comparison. * Images and other assets can take up a good chunk of this available bandwidth, which is why I keep an eye on the size of the end result as well. * It's fun.
  - wink 5y ago
zild3d 5y ago
not mentioned in the article, but since its a static blog the answer in how to survive a hug of death is: stick a CDN in front of it
- hddherman 5y ago
  If availability is your goal, then yes, a CDN is a good idea. My setup is optimized more towards self-hosting and just doing it as a hobby. The web is centralized enough already, I will do my best to balance that out.
notyourday 5y ago
The problem is pretty much never the bandwidth. It is nearly always the boneheaded way nearly static content is generated. If you don't bake your pages into static files, just throw Varnsh in front of your site and make dynamically generated static content static.
- nicbou 5y ago
  The nginx proxy cache can also handle this pretty well. However it might be a bit difficult to set up cache purging.
asiekierka 5y ago
Regarding image compression, have you considered using jpegrescan on top of mogrify resizing? In my experience, it can shave off a few percent losslessly. https://github.com/kud/jpegrescan https://github.com/kud/jpegrescan
kragen 5y ago
A million hits from 75k views, seems pretty reasonable. Taking as an example page http://canonical.org/~kragen/memory-models/ http://canonical.org/~kragen/memory-models/, the last thing of mine that hit the front page, that's 248.7 KiB, including all the images and other liabilities, so if that were distributed over a single day, that would be about 18 GiB, or 1.8 megabits per second, maybe with a peak of a few times that. The numbers provided here seem pretty comparable, maybe a factor of 2 or 3 bigger. Things can vary by an order of magnitude or more from this. The HTML of that page is 17 KiB, so bandwidth could be an order lower, and as Õunapuu points out, it's also easy to drop in a few multi-megabyte photos or videos that bulk up a page to 25 or 25 megabytes instead of 0.25. http://canonical.org/~kragen/larvacut.webm http://canonical.org/~kragen/larvacut.webm is 640×480 × 13 seconds and encodes to 1.8 megabytes.
tyingq 5y ago
Having correct cache headers helps as well, as a single visitor probably comes back to the page fairly often, or visits another page with shared assets. It's surprising how often some things that should be cached come back with headers that make it uncacheable by the browser. KeyCDN has a pretty comprehensive guide: https://www.keycdn.com/blog/http-cache-headers https://www.keycdn.com/blog/http-cache-headers
timw4mail 5y ago
I wish I had 50Mbps uplink. My web server (https://timshome.page https://timshome.page) runs on a ~15Mbps uplink. The hardware is still overpowered for it's usage, though, with a Ryzen 9 3900x (the server gets my gaming rig hand-me-downs) and 16GB of ram. In my case, the limited bandwidth probably protects my server in some respect.
Havoc 5y ago
Bit confused as to why set up a static website and then host it from home? Could have just used one of the CDNs?
- hddherman 5y ago
  The choice is ideological, not practical. I like hosting all my services all from one box at my home.
apoland 5y ago
When I began hosting web sites around 2002 we started with a 1.5mbit leased line. It's amazing how many requests we handled with what is now considered so little bandwidth. Of course the media attached to web sites has grown larger since then.
louwrentius 5y ago
I’ve had a few visits on my solar powered Pi3b+ on a 50 mbit uplink and because it’s a static site both pi and my internet had no issues whatsoever (looking at Grafana).
- hddherman 5y ago
  Have you heard about the Low-tech Magazine? https://solar.lowtechmagazine.com/ https://solar.lowtechmagazine.com/ This site is also hosted on an SBC running on batteries, and powered by solar power. Just wanted to mention this in case you haven't heard about them.
  - louwrentius 5y ago
    They inspired me to do this, goole my username. LTM credits me back on the site, love them.
sT370ma2 5y ago
I've survived the front page of Hacker News more than once with an uplink speed that averages only 0.75 Mbps. You just have to know how to do it.

261 points by hddherman 5y ago

Wayback Machine link, just in case: https://web.archive.org/web/20220209050643/https://ounapuu.ee/posts/2022/02/09/hn-stats-analytics/ https://web.archive.org/web/20220209050643/https://ounapuu.e...

charcircuit 5y ago

There didn't seem to be any data in the article analyzing how much bandwidth was being used under load. I'm not sure if for example compressing those images made it possible to now survive.

hddherman 5y ago
Good point, I’ll try to see if I can visualise that based on the logs that I have.

bawolff 5y ago

I wonder how much bandwidth a hn hug normally is. 50 mbps sounds like a lot to me unless you have heavy images/movies/not text, but i'm out of my area of expertise.

mrb 5y ago
Some of my blog posts have been #1 on HN for a couple hours. I have lightweight pages, about 100 kB including images and all resources. IIRC traffic peaked at 30 page hits per second. So about 3 MB/s or 24 Mbit/s

ComputerGuru 5y ago

This is a joke, right? We colo at a location where the standard is 100mbit and the websites hosted there are doing just fine. Just slim down your pages and stop generating 25MiB landing page bundles.

skilled 5y ago

Ok, what is the color theme you're using for the terminal?

S0und 5y ago
Judging by the colors: it's monokai
- jcun4128 5y ago
  Love Monokai Pro Spectrum been using it for a few years now in VSC
Godel_unicode 5y ago
That's just what goaccess looks like, it's pretty sweet though: goaccess.io
- skilled 5y ago
  Thanks!
- ZuLuuuuuu 5y ago
  I was expecting that it is written in Go (because of the name), but apparently it is written in C. Looks like a nice program.
  - re 5y ago
    I had assumed that too, but guess it's G.O. Access, for the author's initials: https://github.com/allinurl https://github.com/allinurl
jamil7 5y ago
Asking the important questions

invalidator 5y ago

I haven't been hugged by HN yet, but I've survived being featured on a couple other front pages long ago. It's easy if it's a lightweight page and you have something faster than a 486. If it's heavy (lots of images, videos, downloads, etc) you can run out of bandwidth pretty quickly. However, that doesn't necessarily sink the ship! Traffic shaping makes a big difference. Without it, the modem buffer quickly fills causing multi-second ping times. This easily leads to congestion collapse. TCP grows the receive window to large values, and the congestion avoidance algorithm doesn't do well when the feedback is delayed due to the latency. TCP starts sawtoothing, and you end up with occasional moments where a few pages get served, interspersed with so much loss that connections get dropped. We used htb [1] which tightly controls the outbound traffic rate, preferring to drop packets instead of delay them. That prevents the modem's buffer from filling up, which keeps the latency down. This creates a smoother flow. Pages load slowly, but at least they keep loading, instead of wasting bandwidth on retries. It also prioritizes traffic: other services are guaranteed their minimum rates. We were using ssh through the line and didn't even realize we were getting hit until someone asked why the site was running slowly. Newer algorithms are even better, especially for keeping jitter down, but htb is simple and vastly better than letting your router blast packets at full speed. OpenWRT and all the other open source router firmwares can do it these days. It improves things for even normal end-user uses, but if you're running a server at home, you really want to set up some QOS before someone hugs you to death. [1] https://linux.die.net/man/8/tc-htb https://linux.die.net/man/8/tc-htb

zekica 5y ago
Codel, Fq_Codel and especially Cake are worth looking into.
- throwaway48375 5y ago
  I wish OPNsense had cake.
- mrg2k8 5y ago
  I used tc, CAKE and ifb for a small ISP router-on-a-stick implementation (around 2000 end-users) and it did wonders with the limited bandwidth I had to work with (around 3Gbps). More simple to configure than HTB, no more playing around with lots of knobs. Even during high peak traffic the latency would stay low.

tsak 5y ago

I'm using the exact same setup as yourself. I have a script that runs every 10 minutes that checks if my external IP has changed and then updates my DNS records via the GoDaddy API. I also decided to use Hugo to generate my blog (https://tsak.dev https://tsak.dev) And I use `goaccess` to check my logs as well.

nicbou 5y ago
There is a similar docker image that does this for DigitalOcean.

miyuru 5y ago

Any story on why the blog does not support IPv6? Your ISP seems to support it. https://stats.labs.apnic.net/ipv6/AS2586 https://stats.labs.apnic.net/ipv6/AS2586 I am asking because you did not even mention it in the "Residential connections and DNS" section. Depending on the ISP you might get a static IPv6 prefix.

hddherman 5y ago
My ISP only hands out an IPv4 address, unfortunately. Once I can change to another provider here, I’ll also be able to get an IPv6 address.
- miyuru 5y ago
  thats a bummer. thanks for the info!

rufname 5y ago

Since it appears to be a statically generated site it might not matter, but do you use any caching in your NGINX config?

hddherman 5y ago
Not entirely sure, since I rely on the defaults provided by LinuxServer.io SWAG Docker image. The backing store for my site runs on ZFS though and it’s highly likely that web assets are cached in memory.

jfkimmes 5y ago

Quick Tip for your <figure> inconvenience with markdown: Just write a script that copies whatever is in your clipboard to a ./images/ directory and replace the content of your clipboard with the markdown you usually write pointing to that image. See my hacky version, here [1]. I have that on a keyboard shortcut but you can also just alias/source it and run it when needed. [1]: https://gitlab.com/jfkimmes/dotfiles/-/blob/master/scripts/markdown_image_include.sh https://gitlab.com/jfkimmes/dotfiles/-/blob/master/scripts/m... (this requires an `activate` command that symlinks my current project directory to a path in my home directory so that all my scripts can work there. You can probably make this more elegant but it works for me.)

calmingsolitude 5y ago

I've had the experience of being on the opposite end of the spectrum when I made a joke site[0] that kept the connection open to every connected browser. Needless to say, it got hugged to death in the very first ten minutes. What got it back up was switching from my homebrew webserver to Nginx and increasing the file descriptor limits. Switching to a battle-tested webserver made all the difference. I'm still not sure how many connections it can keep open now, but I don't have to worry about the static parts of the site going down. [0] https://youcantdownloadthisimage.online/ https://youcantdownloadthisimage.online/

TT-392 5y ago
Downloading your mona lisa worked just fine on my device (Android), in whatever browser telegram opens links in.
- deleted 5y ago
  [deleted]
unnouinceput 5y ago
Firefox user here. Right click on image -> Open image in new tab. Then kill the connection and could "Save Image as" just fine. Or print screen it and get it in PBrush.
- foxfluff 5y ago
  > Firefox user here. Right click on image -> Open image in new tab. Then kill the connection and could "Save Image as" just fine. That doesn't work for me.. Firefox just starts a new download for the image and it never finishes. Sometimes it seems to flush about 32KB to the disk, giving you half of the image, but the rest must still be waiting in buffers.
  - unnouinceput 5y ago
    Maybe because in my case Firefox is augmented with uBlock Origin + NoScript + Privacy Badger? I don't know, maybe because I'm on desktop and you're on mobile? No idea, but that's how I can do it.

moreentropy 5y ago

I'm in the same situation (50 Mbps uplink at one place, 100Mbps at another) and that's enough to do all hosting for hobby projects at home, which I really love. Instead of exposing my home directly using DynDNS, I got a really cheap low end server (currently a VM w/ 1 CPU, 512MB RAM and 400mbps traffic flat) for 1€/month that proxies all traffic to the (hidden) servers hosted at home. The spec is enough to reliably run a HAproxy that can max out the available bandwidth w/o sweat and it allows me to do failover between servers in my two "home datacenters" + possibly cache assets.

dTal 5y ago
May I ask where you rent your ultra-cheap server from?
- moreentropy 5y ago
  I'm using a Ionos (oneandone) VPS S: https://www.ionos.de/server/vps https://www.ionos.de/server/vps 1&1/Ionos is one of the largest and best connected ISP and Hosting providers in Germany, not some small shady shop. I see they doubled the price to 2€ though :)
  - wackro 5y ago
    I made a WordPress site for a friend through Ionos's managed offering. After a couple of months of the site being up, they decided to detach the database for literally no reason. I had to go through support to get it back up. I'm guessing it got erroneously flagged as over its quota, but they couldn't explain why it happened. Just my experience.
  - sirsinsalot 5y ago
    I've had nothing but hell from 1&1/ionos ... if your billing has issues, be prepared to be treated like a criminal with few paths to easy resolution.
    - littlecranky67 5y ago

zuzun 5y ago

You can get the web fonts below 20 KB each if you subset them. You can also check if they are present on the system with `local()` before `url()` in the font-face declaration. You could also give the Apple system fonts a higher priority than Inter, because Inter and San Francisco are pretty similar.

adam-a 5y ago

I did run a few semi-private sites from my home but I soon found the incoming traffic was causing my outbound connection to fall over. Lots of dropped video calls and spotty vpn connections. I assume because my ISP provided router was not handling the incoming requests very well. I wonder if anyone here has any good advice to get this sort of setup to run well?

hddherman 5y ago
Do you use your own router behind the box provided by the ISP? And is it DOCSIS? If so, using the ISP-provided box in bridge mode could help. This was the cause of many woes in my setup, whenever there were a lot of connections open, the internet connectivity would drop off. But once I changed the ISP-provided box to be a dumb bridge between the internet and my own router, this stopped being an issue.
- adam-a 5y ago
  I was just using the ISP router on its own. Any recommendations for a good router?
  - hddherman 5y ago
    If you're into tinkering, then anything that has good-enough specs and has great openWRT support is a safe bet. Personally I use a TP-Link Archer C7 v5, but there might be better options out there by now.
- xuki 5y ago
  Yeah, always bridge the ISP box if possible. Get an EdgeRouter or run a decent router with OpenWRT or run pfSense/OPNSense on a spare x86 machine, those should be plenty fast for moderate usage.

marginalia_nu 5y ago

I've been hugged twice on 100 mbit domestic broadband, and it's honestly not been all that bad. In general I've seen somewhere between 1-5 visitors per second, a couple of million over a day. I think keeping the number of requests per visit down is a big key to surviving this traffic with low resources. The first hug was search.marginalia.nu, which used to load fonts and had a load of about 50 Kb/visit; but still only generated in maybe 4-5 GETs per visit. I measured about 2 Mbps. The second time was search.marginalia.nu/browse/random, which loads 75 images at a resolution of 512x348 (mostly WebP; so anywhere between 2 and 30 Kb). These were randomly picked from a very large pool, so they can't be meaningfully cached. I presciently set them to lazy loading and I think that made a big difference, especially from mobile visitors as they'd only load a few at a time. I didn't measure the bandwidth usage at this point, but it stayed up and stayed responsive throughout the day. I think something that possibly also makes a difference is that I don't do any sort of session cookies or mutable non-ephemeral state in my backend. I can only imagine having to validate sessions and manage access/roles for every request would make for a lot more work. I don't think it's necessarily the bandwidth that's going to be the bottleneck, rather than something like contested locks in the database, resulting in back pressure in turn causing traffic jam of lingering connections. Reducing mutable server state helps a lot with that.

sgtnoodle 5y ago
I made a very pretty retro control panel for Kerbal Space Program a few years back, and made a nice writeup on my personal website about it, hosted from a low power Intel baytrail based server I keep in my garage. It got picked up by hackaday, then popular mechanics and a few other similar sites, and hacker news. I vaguely recall my upload rate was about 5 Mbps at the time, and my connection was quite saturated for a few days. The page has half a dozen jpg photos, so I suspect that was the bulk of the data.
_xnmw 5y ago
Wait, what, one of the best text search engines for the whole internet runs on a residential broadband connection on a home server? My mind is blown.
- 5y ago

at0mic22 5y ago

If you struggle with bandwidth, why not serving gzipped content? By response headers it looks like you are serving plain html.

jspash 5y ago
Self-hosting and service 300kb for font files also seems excessive. Especially for a Helvetica clone.
- hddherman 5y ago
  Yep, that's why I am planning on getting rid of them completely. I did a quick test this morning, seems like removing references to the fonts from the theme and adjusting some font weights should do the trick.
- nicbou 5y ago
  I use type subsetting and only serve about 50kb worth of fonts (4 variants in total). 300kb seems excessive.
MrDOS 5y ago
Actually, it looks to me like the HTML response is the only thing that's gzip'd. It doesn't make sense to waste time gzip'ing the PNGs, but the CSS and JS files should definitely be compressed. (And maybe the fonts? Not sure.) The default `gzip_types` setting in the nginx config shipped by Debian/Ubuntu does include `text/css` and `application/javascript`, but not `text/html`. Could be that OP is using a non-distro config that only enables gzip for `text/html`, or that OP accidentally disabled gzip'ing for the other MIME types while enabling it for HTML.

thomasfromcdnjs 5y ago

Sidenote: You pretty much generated traffic analytics anyway, and nginx logs have referrer information if you want to see where the traffic is coming from.

wink 5y ago

Author named ~70k hits per day. If we calculate 72000/12 that's 6000 - which is 1.6 req/s. Why are we talking about "surviving" anything here? Here some numbers from a Wordpress installation on a 3 EUR Scaleway VPS where zero time was spent optimizing anything: $ siege -c 3 -r 10 https://example.org ** SIEGE 4.0.4 ** Preparing 3 concurrent users for battle. The server is now under siege... Transactions: 360 hits Availability: 100.00 % Elapsed time: 13.89 secs Data transferred: 24.65 MB Response time: 0.11 secs Transaction rate: 25.92 trans/sec

xuki 5y ago
People see servers with multi Gbps and think it’s a requirement for high traffic websites. Unless you’re serving videos, bandwidth is almost never the bottleneck.
- Nextgrid 5y ago
  High bandwidth protects you against basic DoS attacks that just fill up the pipe with garbage traffic. The other problem is that bandwidth is usually harder to scale than let's say CPU or memory, so having a large buffer there is helpful. Let's say the aforementioned VPS is suddenly under a lot of CPU load and legitimately can't keep up - Nginx can be trivially installed on it to load-balance between many downstream VPSes. Now let's say you are running out of bandwidth. You may need to switch to a different connection with a different IP, change DNS records and wait for them to propagate, etc.
hddherman 5y ago
It's mainly down to these points: * I've seen a lot of sites fall over when posted on HN or other link aggregators, like Reddit. * Cloud providers advertise gigabit or bigger pipes for networking, 50 Mbps is not a lot in comparison. * Images and other assets can take up a good chunk of this available bandwidth, which is why I keep an eye on the size of the end result as well. * It's fun.
- wink 5y ago

zild3d 5y ago

not mentioned in the article, but since its a static blog the answer in how to survive a hug of death is: stick a CDN in front of it

hddherman 5y ago
If availability is your goal, then yes, a CDN is a good idea. My setup is optimized more towards self-hosting and just doing it as a hobby. The web is centralized enough already, I will do my best to balance that out.

notyourday 5y ago

The problem is pretty much never the bandwidth. It is nearly always the boneheaded way nearly static content is generated. If you don't bake your pages into static files, just throw Varnsh in front of your site and make dynamically generated static content static.

nicbou 5y ago
The nginx proxy cache can also handle this pretty well. However it might be a bit difficult to set up cache purging.

asiekierka 5y ago

Regarding image compression, have you considered using jpegrescan on top of mogrify resizing? In my experience, it can shave off a few percent losslessly. https://github.com/kud/jpegrescan https://github.com/kud/jpegrescan

kragen 5y ago

A million hits from 75k views, seems pretty reasonable. Taking as an example page http://canonical.org/~kragen/memory-models/ http://canonical.org/~kragen/memory-models/, the last thing of mine that hit the front page, that's 248.7 KiB, including all the images and other liabilities, so if that were distributed over a single day, that would be about 18 GiB, or 1.8 megabits per second, maybe with a peak of a few times that. The numbers provided here seem pretty comparable, maybe a factor of 2 or 3 bigger. Things can vary by an order of magnitude or more from this. The HTML of that page is 17 KiB, so bandwidth could be an order lower, and as Õunapuu points out, it's also easy to drop in a few multi-megabyte photos or videos that bulk up a page to 25 or 25 megabytes instead of 0.25. http://canonical.org/~kragen/larvacut.webm http://canonical.org/~kragen/larvacut.webm is 640×480 × 13 seconds and encodes to 1.8 megabytes.

tyingq 5y ago

Having correct cache headers helps as well, as a single visitor probably comes back to the page fairly often, or visits another page with shared assets. It's surprising how often some things that should be cached come back with headers that make it uncacheable by the browser. KeyCDN has a pretty comprehensive guide: https://www.keycdn.com/blog/http-cache-headers https://www.keycdn.com/blog/http-cache-headers

timw4mail 5y ago

I wish I had 50Mbps uplink. My web server (https://timshome.page https://timshome.page) runs on a ~15Mbps uplink. The hardware is still overpowered for it's usage, though, with a Ryzen 9 3900x (the server gets my gaming rig hand-me-downs) and 16GB of ram. In my case, the limited bandwidth probably protects my server in some respect.

Havoc 5y ago

Bit confused as to why set up a static website and then host it from home? Could have just used one of the CDNs?

hddherman 5y ago
The choice is ideological, not practical. I like hosting all my services all from one box at my home.

apoland 5y ago

When I began hosting web sites around 2002 we started with a 1.5mbit leased line. It's amazing how many requests we handled with what is now considered so little bandwidth. Of course the media attached to web sites has grown larger since then.

louwrentius 5y ago

I’ve had a few visits on my solar powered Pi3b+ on a 50 mbit uplink and because it’s a static site both pi and my internet had no issues whatsoever (looking at Grafana).

hddherman 5y ago
Have you heard about the Low-tech Magazine? https://solar.lowtechmagazine.com/ https://solar.lowtechmagazine.com/ This site is also hosted on an SBC running on batteries, and powered by solar power. Just wanted to mention this in case you haven't heard about them.
- louwrentius 5y ago
  They inspired me to do this, goole my username. LTM credits me back on the site, love them.

sT370ma2 5y ago

I've survived the front page of Hacker News more than once with an uplink speed that averages only 0.75 Mbps. You just have to know how to do it.