9 ms·

Slashdot effect

55 points by firefax 11mo ago

lionkor 11mo ago
Can someone with some more experience explain why this happens? The most basic webserver, like a TCP socket served from a C program, serving some files etc. can take tens of thousands of requests per second, per core, on reasonably recent hardware. What's going on that's taking down these sites, really, when its not a quota limit?
- Ameo 11mo ago
  In my experience, it's usually the database that gives out first. It's often a shared database, or one running on overprovisioned hardware. The kinds of sites that go down when receiving unexpected traffic are usually built in such a way where they're making multiple DB requests to render each page. Or they have a dozen poorly configured perf-expensive WordPress plugins running. Or likely all of the above
- mikece 11mo ago
  The webserver was fast but PHP in the days before fast CGI didn't take all that much load to bring the server to its knees. And we didn't have cloud hosting at the time either so dynamic scaling behind a network load balancer wasn't a thing (and running multiple servers behind a reverse proxy was uncommon and expensive).
- gusmas 11mo ago
  In the days of Apache 1.3 in the early 2000s for example every client session had a memory footprint (and in the case of php processes could be in the 1-2 megabytes easy). So there you are on a dedicated piece of hardware with limited RAM and a 10,000 connections hit you at the same time. And you have no caching strategy because you're an idiot child trying your best. Boom - Slashdotted
- aleph_minus_one 11mo ago
  > Can someone with some more experience explain why this happens? The most basic webserver, like a TCP socket served from a C program, serving some files etc. can take tens of thousands of requests per second, per core, on reasonably recent hardware. At that time, it was quite common to start a new thread or even fork a new process per request/connection. Apache was prone to this: > https://www.digitalocean.com/community/tutorials/apache-vs-nginx-practical-considerations https://www.digitalocean.com/community/tutorials/apache-vs-n... "mpm_prefork: This processing module spawns processes with a single thread each to handle requests. Each child can handle a single connection at a time. [...] mpm_worker: This module spawns processes that can each manage multiple threads. Each of these threads can handle a single connection. [...]" (this website also mentions the mpm_event MPM). So, if you have ten thousands of processes or threads on some web server, this can easily overstrain the resources of the web server. Additionally keep in mind that at that time servers were a lot less beefy than today.
  - deleted 11mo ago
    [deleted]
  - grepfru_it 11mo ago
    mpm_worker was notoriously bad back then and would frequently deadlock the entire Apache process. So the default was to either use prefork or some sort of monitoring to restart stuck processes
- JoshTriplett 11mo ago
  Hit a webserver with many thousands or tens of thousands of requests per second, and whatever the weakest link is will break: memory usage, CPU usage, database, bandwidth, billing, third-party service quotas...
  - Sesse__ 11mo ago
    At the time when the Slashdot effect was at its peak, the limiting factors were probably RAM or bandwidth (third-party services was not as much a thing, and CPUs were generally able to push 10 Mbit/sec without much of a problem -- but 10 Mbit/sec Internet lines would be a luxury). Like, you could reasonably host Apache on 16 MB of RAM on a 2 Mbit/sec line for your hobby site with a couple of photos of your car or whatever, and then ten thousand people would come along at once and find it interesting, and your site would be dead for a while.
- metajack 11mo ago
  Many blogs are completely dynamic and grab content out of a database on every request. The static site generation style fell out of early fashion when wordpress took over from moveable type, and didn't really return much until jekyll. Even today I think most blogs you see are mostly dynamic with the platforms using caching to avoid hitting the database every time. Most people don't do performance tuning of their blogs because the average traffic is miniscule. This means that any configuration issues rear their heads when load happens. For example, having your max connections to the web server be far more than the database can support. Perhaps the machine could handle the load if properly configured, but no one did so because there was no need.
  - xp84 11mo ago
    This is very true. A lot of corpsites of SaaS and similar companies run wordpress - i think because it's easy to find a "web developer" who can build very impressive custom themes that you wouldn't even know are blogs. My last company was paying about $500 a month to various "Professional Wordpress Hosting" companies to host their .com -- and they all had spotty downtime records because they were serving everything dynamically. My strategy to fix these has always been to install a "static site generation" plugin, stick the wordpress instance itself on a cheap host behind an IP restriction, and have the site generator plugin publish the site (on every change, which is never even that often) to some static hosting (S3, etc). Saves 99% of the cost and makes the site 50x faster and able to handle infinite traffic. But most companies don't seem to do this, and individual blogs are even less likely to.
  - mbreese 11mo ago
    Don’t forget - at the time, even if you were pulling a static file, you were pulling it from disk. And disk was much slower… IDE was still common. If you had a good server, it could have been SCSI, but a white label Linux box could be quite slow. So, unless you had a good caching setup, you could still have issues sending even static data. Also, when slashdotting was common, networks weren’t always the most robust. It wasn’t uncommon in the late 90’s for companies to have a single a T1 line for access to the internet. So, unless you had a good, well peered network provider, that was another potential bottleneck. We worked with what we had. Thankfully, everything is more robust now.
- jayd16 11mo ago
  Hell, if the connection timeouts are long enough and you're waiting for some pooled DB connection, you could run out of ports even if your throughput was high enough. There are actually a lot of little things to tune if you can't just auto-scale out of a bottleneck.
- 0xbadcafebee 11mo ago
  That's what's possible today. It wasn't back then (see: https://en.wikipedia.org/wiki/C10k_problem https://en.wikipedia.org/wiki/C10k_problem) The other reason is, not every web server, web application, database, etc is optimized for large number of connections. Sometimes they take up too much CPU and memory. Sometimes each page request triggers a connection to a database, so the database connection limit is hit. Sometimes the queries are inefficient and slow the database to a crawl. Sometimes the sheer number of packets causes so many interrupts that it sucks CPU away from the application/database. And sometimes the network connection of the webserver was just slow. Most people hit by the /. effect were not just a simple web server serving static content. Often it was a php website with a mysql database, on a server less powerful than a 2015 smartphone.
  - yobbo 11mo ago
    A 2015 smartphone could be eight 2 GHz cores with 2-4 GB memory and 128 GB SSD. By far faster and more powerful than anything but the heaviest 2000-era servers if that.
- locknitpicker 11mo ago
  > Can someone with some more experience explain why this happens? The most basic webserver, like a TCP socket served from a C program, serving some files etc. can take tens of thousands of requests per second, per core, on reasonably recent hardware. You're grossly overstating the capacity of a server to meet peak load demands, specially when hit with unexpected load spikes. The slashdot effect was a phenomenon observed two decades ago, in a time when the bulk of the web was served as Apache running CGI scripts. The rate of requests any random server could process was around ~300requests per second, and some of the links being hugged to death were literally served from a box under some dude's desk.
  - phkahler 11mo ago
    The Apache web server was popular because it was one of the first MVP in its category. I think it was weird that the solution developed back then was to figure out load balancing among more servers, rather than optimizing the web server itself.
    - toast0 11mo ago
      People did both. And you want load balancing for operations/redundancy anyway. Or because once you get the max requests per sec out of a machine, you can often get a second machine to double rps for much cheaper than a machine that can do double the rps. In the 20-30 year old time frame, easy to obtain servers are one to two cores, often one core per socket. SMP scalability was not a given; you could certainly run on SMP and get benefits, but depending on your load you might not get anywhere close to 2x the throughput. A single core webserver serving static files isn't going to be much faster than 1 ms per file, so you're around 1k rps. And that's assuming you files are all cached. If you do dynamic pages, they're more likely to start closer to 10 ms, and can easily be much longer, so 100 rps is a good target. But, if your pages are sizable, you can easily run out of bandwidth and then everything gets slow and you can hit too much concurrency and things get worse. A T3 was a very good connection back then, and it was 45 mbps. 1000 rps of 50k images is going to fill up your T3. Load balancing, done right, is very scalable for typical http loads where client sends little and server sends lots. Direct Server Return means the load balancer only sees the traffic from the client, which significantly reduces the effort for the load balancer. Of course, you can't do a lot of fancy things with DSR, but that means your balancers aren't overloaded with fancy things. If your DSR load balancer fails, you can fail over to DNS round robin.
- dakna 11mo ago
  At the time, a standard LAMP stack setup wasn't prepared for the https://en.wikipedia.org/wiki/C10k_problem https://en.wikipedia.org/wiki/C10k_problem. Quite often you had no DB connection pool and every request for dynamic data opened/closed a DB connection. Having a fixed number of connections available also quickly hit the limits unless it was a dynamically configured pool. Which also filled up eventually. If you were lucky you could reduce load by caching the URLs taking up the most resources and generate a webpage copy in the filesystem, then add some URL rewrite logic to Apache to skip going through your application logic and bypass the DB. Then you discovered there is a limit of open file descriptors in Linux. After updating this and also shutting down all log files you ran out of things to change pretty quick and started pricing a beefier server. For next time.
- hnlmorg 11mo ago
  - slower, often bare metal hardware - lack of scalability (you can’t dynamically spin up a bare metal server if you don’t already have one in your rack) - slower memory models (CGI, and forking meant it took longer for the OS kernel to allocate system resources) - improvements to TCP and fork performances in the Linux kernel over the years. Such as changes to how TCP negotiation happens - improvements to HTTP transit such as improved compression performance, HTTP/3, etc. - CDNs weren’t common place, and were super expensive. - caching was in its infancy. Tools like varnish (I think it was called) were complicated and setting up memcached, varnish, etc all required additional hardware, that added to hosting costs. - HTTP servers weren’t as mature. Plus weren’t included in the language runtime (eg you’d run httpd that would fork an instance of PHP rather than run the HTTP server as part of the Python, et al) - serving static content of NFSs mounted SAN was slow (the POSIX overhead, plus NFS protocol, plus network roundtrip, plus disk access times) - mechanical drives were slow And so on and so forth. The biggest difference I think was a combination of affordable or free CDNs, reducing pressure on the backend, and the easy of scaling (VM, Docker, etc) meaning you don’t need to find a happy middle ground between costs and high availability.
- dminvs 11mo ago
  there used to be a lot more shared hosting in the world when Slashdot ruled the geek news roost it'd be fine if it's one site on a dedicated machine, but these shared webhosts would routinely cram 500-1000 hosting accounts onto something like a single-core, HT-less 2.4GHz Pentium 4 with two gigs of RAM, running mpm_prefork on Apache 2.0, with single-node MySQL 5.x (maybe even 4.x!) on the same box... which was not terribly efficient as others observed you carry about 20x the compute power of that machine in your pocket, even a Raspberry Pi 4 could triple the workload
- baq 11mo ago
  /. effect is basically a ddos, same principles apply. basically sockets need to be created and torn down faster than incoming requests and they just aren't; they get queued up for a while and/or kill the host if it isn't configured correctly and keeps trying serving traffic which it can't due to resource exhaustion. whether it's a 'modern stack' requiring hundreds of MB of ram per request or a C CGI program matters only quantitatively.
- marcosdumay 11mo ago
  At the time Slashdot was a thing, web servers could serve dozens, maybe a hundred requests per second. Near to the end of (real) Slashdot's life, project 10k was happening. People were rewriting even the Linux process management to make servers perform close to their theoretic capacities.
  - esseph 11mo ago
    twitches Muscle memory still takes me there several times a day...
- jjcm 11mo ago
  You forget that the vast majority of the web in the 2005 era was blogs hosted on wordpress by either service providers or shared hosting providers, neither of which dedicated significant attention to performance. Many of them were hitting quota limits, but many still were truly just going down because of the oversubscribed hardware they were running on. Even an independent server that was properly set up though could hit issues. I had a few of my posts get slashdotted at that time, and the spike was real. Apache hosting wordpress couldn't quite keep up with the spike - I had to use varnish to set up a reverse proxy cache to properly handle the influx.
- Johnny555 11mo ago
  Oh I remember those days. In our case, it was because we had so much website data that we couldn't keep it all in cache, so when the site became busy enough, performance fell off a cliff as we became disk I/O bound. We had maxed out our main server with 512MB of RAM. We couldn't afford the new Sun Ultra 2's that would have given us a spacious 2GB of RAM, so we ended up coding a front-end to shard requests across the 2 web servers and used a Sparcstation 5 to run that front-end. Eventually we rearchitected the entire site to move the static HTML websites into an Oracle database. (by then we had upgraded to a Sun Ultra Enterprise 3000, which could handle up to 6GB of memory, more than we could ever use)
- storyinmemo 11mo ago
  2003: http://www.geology.smu.edu/~dpa-www/attention_span http://www.geology.smu.edu/~dpa-www/attention_span. 22 Requests / second was a big deal. Perhaps the best benchmark for the maximum potential traffic of the community at that time comes from a Slashdot post by Rob Malda (CmdrTaco) on September 14, 2001 (https://news.slashdot.org/story/01/09/13/154222/handling-the-loads https://news.slashdot.org/story/01/09/13/154222/handling-the...), detailing the site's traffic following the 9/11 attacks. This load on Slashdot itself represents the full attention of the userbase and was higher than the site normally experienced: Normal Load: 18–20 dynamic page views per second. Peak Load: 60–70 page views per second. Daily Volume: 3 million page views (up from a daily average of 1.4 million). To handle this higher load, Slashdot itself had to disable dynamic content and serve static HTML to survive this load. Their database could not handle the query volume. A typical "Slashdotting" pushed 5 to 10 Mbps of traffic. Since many sites were hosted on T1 lines (1.5 Mbps), the pipe was instantly clogged, resulting in 100% packet loss for legitimate users. I remember when Slashdot broke because they used an Unsigned Mediumint counter for a primary key and finally overflowed it with 16 million comments over the life of the site on Nov 6, 2006. It was a different time back then when we still measured our installed RAM in megabytes.
  - iso1631 11mo ago
    I remember the Tuesday well - pretty much every website was down due to the load, only Slashdot remained (with static pages, downloading over my 4kbit/second modem connection)
- nailer 11mo ago
  > Can someone with some more experience explain why this happens? The most basic webserver, like a TCP socket served from a C program, serving some files etc. can take tens of thousands of requests per second, per core, on reasonably recent hardware. Back then it was Wordpress/PHP with no caching. But: you're right. Once upon a time named Maxim made a web page that pointed out that a Pentium III should be able to handle 10K simultaneous requests, then said Apache was shit, and that he had made his own webserver, and because he was Russian he filled the webpage with animated hammers and sickles. This was how NGINX was created.
virgil_disgr4ce 11mo ago
Kind of funny to find this on the front page of HN. Makes me wonder what percentage of today's HN readers didn't live through the Slashdot era. (I'm aware it's still around.)
- SirFatty 11mo ago
  yeah, it's around, but a ghost of its former glory.
  - ricksunny 11mo ago
    I still check t regularly alongside HN. Need a counterpoint to HN submission coverage and crowd-editorial zeitgeist.
    - bartvk 11mo ago
      Yup, me too. What’s funny is that your slashdot account can make friends (or enemies!) with other accounts, and there’s a limit of 200. Sometimes I spot comments of friends I made like 20 years ago.
    - SirFatty 11mo ago
      Same.. however I've been using Alterslash as a story summary for years. https://alterslash.org/ https://alterslash.org/
    - jiehong 11mo ago
      Maybe lobste.rs?
      - esseph 11mo ago
        Mostly HN reposts and extremely limited ability to post there if you don't have Bay Area social circles.
  - theideaofcoffee 11mo ago
    It's sad to see its decline. I started reading right around 2000 and even noticed that it started to decline even as early as 2008, after and around some of its redesigns. I know I quit regularly stopping around there probably around that time too.
josefritzishere 11mo ago
I have only ever heard this called the "Hug of Death" not specifically attributed to slashdot.
- gadders 11mo ago
  Back in the early 2000s there weren't that many sites that could direct that sort of traffic to a website.
- rvbissell 11mo ago
  The name changed when Slashdot lost significant mindshare among tech nerds.
  - joshdavham 11mo ago
    Why did it lose this mindshare?
    - hnlmorg 11mo ago
      Because all social platforms and message boards are like living entities: They have their infancy and grow into something popular (relatively speaking) then your older members either get bored or jaded, often both. The culture of the platform starts to change while more people leave disliking the change than new people discover the platform. Resulting in the platform steadily shrinking in popularity. Every social network and forum has experienced the same phenomenon.
    - marcosdumay 11mo ago
      It was sold and the new owner didn't like the way it was.
    - tedunangst 11mo ago
      Stories were selected by site mods (editors) and that's censorship, man!
    - zajio1am 11mo ago
      IMHO people moved to Hacker News. I would guess one of the reasons is better comment system here. Algorithmic ordering of comments based on up/down is important for scaling to thousands of comments. Slashdot had scoring system, but comments were still ordered by time, which makes it less practical.
trumbitta2 11mo ago
"I was there... 3000 years ago."
- trumbitta2 11mo ago
  Jesus Christ you guys need a life
dbg31415 11mo ago
Better to show up on Slashdot, instead of Fucked Company.
schmuckonwheels 11mo ago
No one in 2025 should be allowed to criticize early 2000s web practices as we have come full circle in many ways. They were limited by tools and resources. Now you'll just see some async "API endpoint error -32768 timeout" and 8,000 spinning pinwheels on the page instead of it simply hanging. Because now you need to allocate 650MB RAM so you can output Hello World in some buggy "stack" with "memory safety". The smart ones were using Java 25 years ago and what needed some 80lb Solaris box could run off a mobile phone now.
- andrewmcwatters 11mo ago
  One of the most popular API frameworks is called “FastAPI” but in any meaningful benchmark it’s one of the slowest commercially used pieces of web software the industry has adopted. There is no hope for large segments of the industry. We actively choose to make bad decisions all the time, and then people rationalize them.
  - babylon5 11mo ago
    Fast to develop, fast to run under modest to reasonable requirements. Few things need to scale further.
    - phyrog 11mo ago
      It may not have to scale, but being able to use the same hardware for more (different things) is nice nevertheless.
  - georgeburdell 11mo ago
    FastAPI is the best combination of an easy and a flexible Python library for web servers, from my experience. If I need performance, I will rewrite it in Go.
    - amypetrik8 11mo ago
      the slowest API i ever seen is FastAPI
compsciphd 11mo ago
anyone else where have a 4 digit slashdot ID? (or heck, even less digits?)
- iso1631 11mo ago
  You must be new here
- beej71 11mo ago
  Not me. I'm 86,000-something.
JSR_FDED 11mo ago
Kuro5hin was also from that era. I miss it!

55 points by firefax 11mo ago

Can someone with some more experience explain why this happens? The most basic webserver, like a TCP socket served from a C program, serving some files etc. can take tens of thousands of requests per second, per core, on reasonably recent hardware. What's going on that's taking down these sites, really, when its not a quota limit?

Ameo 11mo ago
In my experience, it's usually the database that gives out first. It's often a shared database, or one running on overprovisioned hardware. The kinds of sites that go down when receiving unexpected traffic are usually built in such a way where they're making multiple DB requests to render each page. Or they have a dozen poorly configured perf-expensive WordPress plugins running. Or likely all of the above
mikece 11mo ago
The webserver was fast but PHP in the days before fast CGI didn't take all that much load to bring the server to its knees. And we didn't have cloud hosting at the time either so dynamic scaling behind a network load balancer wasn't a thing (and running multiple servers behind a reverse proxy was uncommon and expensive).
gusmas 11mo ago
In the days of Apache 1.3 in the early 2000s for example every client session had a memory footprint (and in the case of php processes could be in the 1-2 megabytes easy). So there you are on a dedicated piece of hardware with limited RAM and a 10,000 connections hit you at the same time. And you have no caching strategy because you're an idiot child trying your best. Boom - Slashdotted
aleph_minus_one 11mo ago
> Can someone with some more experience explain why this happens? The most basic webserver, like a TCP socket served from a C program, serving some files etc. can take tens of thousands of requests per second, per core, on reasonably recent hardware. At that time, it was quite common to start a new thread or even fork a new process per request/connection. Apache was prone to this: > https://www.digitalocean.com/community/tutorials/apache-vs-nginx-practical-considerations https://www.digitalocean.com/community/tutorials/apache-vs-n... "mpm_prefork: This processing module spawns processes with a single thread each to handle requests. Each child can handle a single connection at a time. [...] mpm_worker: This module spawns processes that can each manage multiple threads. Each of these threads can handle a single connection. [...]" (this website also mentions the mpm_event MPM). So, if you have ten thousands of processes or threads on some web server, this can easily overstrain the resources of the web server. Additionally keep in mind that at that time servers were a lot less beefy than today.
- deleted 11mo ago
  [deleted]
- grepfru_it 11mo ago
  mpm_worker was notoriously bad back then and would frequently deadlock the entire Apache process. So the default was to either use prefork or some sort of monitoring to restart stuck processes
JoshTriplett 11mo ago
Hit a webserver with many thousands or tens of thousands of requests per second, and whatever the weakest link is will break: memory usage, CPU usage, database, bandwidth, billing, third-party service quotas...
- Sesse__ 11mo ago
  At the time when the Slashdot effect was at its peak, the limiting factors were probably RAM or bandwidth (third-party services was not as much a thing, and CPUs were generally able to push 10 Mbit/sec without much of a problem -- but 10 Mbit/sec Internet lines would be a luxury). Like, you could reasonably host Apache on 16 MB of RAM on a 2 Mbit/sec line for your hobby site with a couple of photos of your car or whatever, and then ten thousand people would come along at once and find it interesting, and your site would be dead for a while.
metajack 11mo ago
Many blogs are completely dynamic and grab content out of a database on every request. The static site generation style fell out of early fashion when wordpress took over from moveable type, and didn't really return much until jekyll. Even today I think most blogs you see are mostly dynamic with the platforms using caching to avoid hitting the database every time. Most people don't do performance tuning of their blogs because the average traffic is miniscule. This means that any configuration issues rear their heads when load happens. For example, having your max connections to the web server be far more than the database can support. Perhaps the machine could handle the load if properly configured, but no one did so because there was no need.
- xp84 11mo ago
  This is very true. A lot of corpsites of SaaS and similar companies run wordpress - i think because it's easy to find a "web developer" who can build very impressive custom themes that you wouldn't even know are blogs. My last company was paying about $500 a month to various "Professional Wordpress Hosting" companies to host their .com -- and they all had spotty downtime records because they were serving everything dynamically. My strategy to fix these has always been to install a "static site generation" plugin, stick the wordpress instance itself on a cheap host behind an IP restriction, and have the site generator plugin publish the site (on every change, which is never even that often) to some static hosting (S3, etc). Saves 99% of the cost and makes the site 50x faster and able to handle infinite traffic. But most companies don't seem to do this, and individual blogs are even less likely to.
- mbreese 11mo ago
  Don’t forget - at the time, even if you were pulling a static file, you were pulling it from disk. And disk was much slower… IDE was still common. If you had a good server, it could have been SCSI, but a white label Linux box could be quite slow. So, unless you had a good caching setup, you could still have issues sending even static data. Also, when slashdotting was common, networks weren’t always the most robust. It wasn’t uncommon in the late 90’s for companies to have a single a T1 line for access to the internet. So, unless you had a good, well peered network provider, that was another potential bottleneck. We worked with what we had. Thankfully, everything is more robust now.
jayd16 11mo ago
Hell, if the connection timeouts are long enough and you're waiting for some pooled DB connection, you could run out of ports even if your throughput was high enough. There are actually a lot of little things to tune if you can't just auto-scale out of a bottleneck.
0xbadcafebee 11mo ago
That's what's possible today. It wasn't back then (see: https://en.wikipedia.org/wiki/C10k_problem https://en.wikipedia.org/wiki/C10k_problem) The other reason is, not every web server, web application, database, etc is optimized for large number of connections. Sometimes they take up too much CPU and memory. Sometimes each page request triggers a connection to a database, so the database connection limit is hit. Sometimes the queries are inefficient and slow the database to a crawl. Sometimes the sheer number of packets causes so many interrupts that it sucks CPU away from the application/database. And sometimes the network connection of the webserver was just slow. Most people hit by the /. effect were not just a simple web server serving static content. Often it was a php website with a mysql database, on a server less powerful than a 2015 smartphone.
- yobbo 11mo ago
  A 2015 smartphone could be eight 2 GHz cores with 2-4 GB memory and 128 GB SSD. By far faster and more powerful than anything but the heaviest 2000-era servers if that.
locknitpicker 11mo ago
> Can someone with some more experience explain why this happens? The most basic webserver, like a TCP socket served from a C program, serving some files etc. can take tens of thousands of requests per second, per core, on reasonably recent hardware. You're grossly overstating the capacity of a server to meet peak load demands, specially when hit with unexpected load spikes. The slashdot effect was a phenomenon observed two decades ago, in a time when the bulk of the web was served as Apache running CGI scripts. The rate of requests any random server could process was around ~300requests per second, and some of the links being hugged to death were literally served from a box under some dude's desk.
- phkahler 11mo ago
  The Apache web server was popular because it was one of the first MVP in its category. I think it was weird that the solution developed back then was to figure out load balancing among more servers, rather than optimizing the web server itself.
  - toast0 11mo ago
    People did both. And you want load balancing for operations/redundancy anyway. Or because once you get the max requests per sec out of a machine, you can often get a second machine to double rps for much cheaper than a machine that can do double the rps. In the 20-30 year old time frame, easy to obtain servers are one to two cores, often one core per socket. SMP scalability was not a given; you could certainly run on SMP and get benefits, but depending on your load you might not get anywhere close to 2x the throughput. A single core webserver serving static files isn't going to be much faster than 1 ms per file, so you're around 1k rps. And that's assuming you files are all cached. If you do dynamic pages, they're more likely to start closer to 10 ms, and can easily be much longer, so 100 rps is a good target. But, if your pages are sizable, you can easily run out of bandwidth and then everything gets slow and you can hit too much concurrency and things get worse. A T3 was a very good connection back then, and it was 45 mbps. 1000 rps of 50k images is going to fill up your T3. Load balancing, done right, is very scalable for typical http loads where client sends little and server sends lots. Direct Server Return means the load balancer only sees the traffic from the client, which significantly reduces the effort for the load balancer. Of course, you can't do a lot of fancy things with DSR, but that means your balancers aren't overloaded with fancy things. If your DSR load balancer fails, you can fail over to DNS round robin.
dakna 11mo ago
At the time, a standard LAMP stack setup wasn't prepared for the https://en.wikipedia.org/wiki/C10k_problem https://en.wikipedia.org/wiki/C10k_problem. Quite often you had no DB connection pool and every request for dynamic data opened/closed a DB connection. Having a fixed number of connections available also quickly hit the limits unless it was a dynamically configured pool. Which also filled up eventually. If you were lucky you could reduce load by caching the URLs taking up the most resources and generate a webpage copy in the filesystem, then add some URL rewrite logic to Apache to skip going through your application logic and bypass the DB. Then you discovered there is a limit of open file descriptors in Linux. After updating this and also shutting down all log files you ran out of things to change pretty quick and started pricing a beefier server. For next time.
hnlmorg 11mo ago
- slower, often bare metal hardware - lack of scalability (you can’t dynamically spin up a bare metal server if you don’t already have one in your rack) - slower memory models (CGI, and forking meant it took longer for the OS kernel to allocate system resources) - improvements to TCP and fork performances in the Linux kernel over the years. Such as changes to how TCP negotiation happens - improvements to HTTP transit such as improved compression performance, HTTP/3, etc. - CDNs weren’t common place, and were super expensive. - caching was in its infancy. Tools like varnish (I think it was called) were complicated and setting up memcached, varnish, etc all required additional hardware, that added to hosting costs. - HTTP servers weren’t as mature. Plus weren’t included in the language runtime (eg you’d run httpd that would fork an instance of PHP rather than run the HTTP server as part of the Python, et al) - serving static content of NFSs mounted SAN was slow (the POSIX overhead, plus NFS protocol, plus network roundtrip, plus disk access times) - mechanical drives were slow And so on and so forth. The biggest difference I think was a combination of affordable or free CDNs, reducing pressure on the backend, and the easy of scaling (VM, Docker, etc) meaning you don’t need to find a happy middle ground between costs and high availability.
dminvs 11mo ago
there used to be a lot more shared hosting in the world when Slashdot ruled the geek news roost it'd be fine if it's one site on a dedicated machine, but these shared webhosts would routinely cram 500-1000 hosting accounts onto something like a single-core, HT-less 2.4GHz Pentium 4 with two gigs of RAM, running mpm_prefork on Apache 2.0, with single-node MySQL 5.x (maybe even 4.x!) on the same box... which was not terribly efficient as others observed you carry about 20x the compute power of that machine in your pocket, even a Raspberry Pi 4 could triple the workload
baq 11mo ago
/. effect is basically a ddos, same principles apply. basically sockets need to be created and torn down faster than incoming requests and they just aren't; they get queued up for a while and/or kill the host if it isn't configured correctly and keeps trying serving traffic which it can't due to resource exhaustion. whether it's a 'modern stack' requiring hundreds of MB of ram per request or a C CGI program matters only quantitatively.
marcosdumay 11mo ago
At the time Slashdot was a thing, web servers could serve dozens, maybe a hundred requests per second. Near to the end of (real) Slashdot's life, project 10k was happening. People were rewriting even the Linux process management to make servers perform close to their theoretic capacities.
- esseph 11mo ago
  twitches Muscle memory still takes me there several times a day...
jjcm 11mo ago
You forget that the vast majority of the web in the 2005 era was blogs hosted on wordpress by either service providers or shared hosting providers, neither of which dedicated significant attention to performance. Many of them were hitting quota limits, but many still were truly just going down because of the oversubscribed hardware they were running on. Even an independent server that was properly set up though could hit issues. I had a few of my posts get slashdotted at that time, and the spike was real. Apache hosting wordpress couldn't quite keep up with the spike - I had to use varnish to set up a reverse proxy cache to properly handle the influx.
Johnny555 11mo ago
Oh I remember those days. In our case, it was because we had so much website data that we couldn't keep it all in cache, so when the site became busy enough, performance fell off a cliff as we became disk I/O bound. We had maxed out our main server with 512MB of RAM. We couldn't afford the new Sun Ultra 2's that would have given us a spacious 2GB of RAM, so we ended up coding a front-end to shard requests across the 2 web servers and used a Sparcstation 5 to run that front-end. Eventually we rearchitected the entire site to move the static HTML websites into an Oracle database. (by then we had upgraded to a Sun Ultra Enterprise 3000, which could handle up to 6GB of memory, more than we could ever use)
storyinmemo 11mo ago
2003: http://www.geology.smu.edu/~dpa-www/attention_span http://www.geology.smu.edu/~dpa-www/attention_span. 22 Requests / second was a big deal. Perhaps the best benchmark for the maximum potential traffic of the community at that time comes from a Slashdot post by Rob Malda (CmdrTaco) on September 14, 2001 (https://news.slashdot.org/story/01/09/13/154222/handling-the-loads https://news.slashdot.org/story/01/09/13/154222/handling-the...), detailing the site's traffic following the 9/11 attacks. This load on Slashdot itself represents the full attention of the userbase and was higher than the site normally experienced: Normal Load: 18–20 dynamic page views per second. Peak Load: 60–70 page views per second. Daily Volume: 3 million page views (up from a daily average of 1.4 million). To handle this higher load, Slashdot itself had to disable dynamic content and serve static HTML to survive this load. Their database could not handle the query volume. A typical "Slashdotting" pushed 5 to 10 Mbps of traffic. Since many sites were hosted on T1 lines (1.5 Mbps), the pipe was instantly clogged, resulting in 100% packet loss for legitimate users. I remember when Slashdot broke because they used an Unsigned Mediumint counter for a primary key and finally overflowed it with 16 million comments over the life of the site on Nov 6, 2006. It was a different time back then when we still measured our installed RAM in megabytes.
- iso1631 11mo ago
  I remember the Tuesday well - pretty much every website was down due to the load, only Slashdot remained (with static pages, downloading over my 4kbit/second modem connection)
nailer 11mo ago
> Can someone with some more experience explain why this happens? The most basic webserver, like a TCP socket served from a C program, serving some files etc. can take tens of thousands of requests per second, per core, on reasonably recent hardware. Back then it was Wordpress/PHP with no caching. But: you're right. Once upon a time named Maxim made a web page that pointed out that a Pentium III should be able to handle 10K simultaneous requests, then said Apache was shit, and that he had made his own webserver, and because he was Russian he filled the webpage with animated hammers and sickles. This was how NGINX was created.

virgil_disgr4ce 11mo ago

Kind of funny to find this on the front page of HN. Makes me wonder what percentage of today's HN readers didn't live through the Slashdot era. (I'm aware it's still around.)

SirFatty 11mo ago
yeah, it's around, but a ghost of its former glory.
- ricksunny 11mo ago
  I still check t regularly alongside HN. Need a counterpoint to HN submission coverage and crowd-editorial zeitgeist.
  - bartvk 11mo ago
    Yup, me too. What’s funny is that your slashdot account can make friends (or enemies!) with other accounts, and there’s a limit of 200. Sometimes I spot comments of friends I made like 20 years ago.
  - SirFatty 11mo ago
    Same.. however I've been using Alterslash as a story summary for years. https://alterslash.org/ https://alterslash.org/
  - jiehong 11mo ago
    Maybe lobste.rs?
    - esseph 11mo ago
      Mostly HN reposts and extremely limited ability to post there if you don't have Bay Area social circles.
- theideaofcoffee 11mo ago
  It's sad to see its decline. I started reading right around 2000 and even noticed that it started to decline even as early as 2008, after and around some of its redesigns. I know I quit regularly stopping around there probably around that time too.

josefritzishere 11mo ago

I have only ever heard this called the "Hug of Death" not specifically attributed to slashdot.

gadders 11mo ago
Back in the early 2000s there weren't that many sites that could direct that sort of traffic to a website.
rvbissell 11mo ago
The name changed when Slashdot lost significant mindshare among tech nerds.
- joshdavham 11mo ago
  Why did it lose this mindshare?
  - hnlmorg 11mo ago
    Because all social platforms and message boards are like living entities: They have their infancy and grow into something popular (relatively speaking) then your older members either get bored or jaded, often both. The culture of the platform starts to change while more people leave disliking the change than new people discover the platform. Resulting in the platform steadily shrinking in popularity. Every social network and forum has experienced the same phenomenon.
  - marcosdumay 11mo ago
    It was sold and the new owner didn't like the way it was.
  - tedunangst 11mo ago
    Stories were selected by site mods (editors) and that's censorship, man!
  - zajio1am 11mo ago
    IMHO people moved to Hacker News. I would guess one of the reasons is better comment system here. Algorithmic ordering of comments based on up/down is important for scaling to thousands of comments. Slashdot had scoring system, but comments were still ordered by time, which makes it less practical.

trumbitta2 11mo ago

"I was there... 3000 years ago."

trumbitta2 11mo ago
Jesus Christ you guys need a life

dbg31415 11mo ago

Better to show up on Slashdot, instead of Fucked Company.

schmuckonwheels 11mo ago

No one in 2025 should be allowed to criticize early 2000s web practices as we have come full circle in many ways. They were limited by tools and resources. Now you'll just see some async "API endpoint error -32768 timeout" and 8,000 spinning pinwheels on the page instead of it simply hanging. Because now you need to allocate 650MB RAM so you can output Hello World in some buggy "stack" with "memory safety". The smart ones were using Java 25 years ago and what needed some 80lb Solaris box could run off a mobile phone now.

andrewmcwatters 11mo ago
One of the most popular API frameworks is called “FastAPI” but in any meaningful benchmark it’s one of the slowest commercially used pieces of web software the industry has adopted. There is no hope for large segments of the industry. We actively choose to make bad decisions all the time, and then people rationalize them.
- babylon5 11mo ago
  Fast to develop, fast to run under modest to reasonable requirements. Few things need to scale further.
  - phyrog 11mo ago
    It may not have to scale, but being able to use the same hardware for more (different things) is nice nevertheless.
- georgeburdell 11mo ago
  FastAPI is the best combination of an easy and a flexible Python library for web servers, from my experience. If I need performance, I will rewrite it in Go.
  - amypetrik8 11mo ago
    the slowest API i ever seen is FastAPI

compsciphd 11mo ago

anyone else where have a 4 digit slashdot ID? (or heck, even less digits?)

iso1631 11mo ago
You must be new here
beej71 11mo ago
Not me. I'm 86,000-something.

JSR_FDED 11mo ago

Kuro5hin was also from that era. I miss it!