5 ms·

Either there is a massive number of HN users F5ing the homepage for the last few hours, HN is woefully unprepared for surges in traffic, or there is some kind o

Either there is a massive number of HN users F5ing the homepage for the last few hours, HN is woefully unprepared for surges in traffic, or there is some kind of outage with their infrastructure. hnstatus on twitter and hund.io show no outage news.

oneeyedpigeon 5y ago
> a massive number of HN users It'll be this, I'd bet money on it. Lots like me who don't visit every day are coming to HN for more info. This site probably only ever sees a tiny percentage of its registered users accessing at once; right now, it'll be a much larger percent.
- hinfaits 5y ago
  I am one of the many that only goes to HN when a major service appears down (usually AWS)
nicoburns 5y ago
IIRC HN runs on a single server. So they don't really put much effort into scaling to higher traffic levels. They don't really need to. Even now the site is available, if a little slow.
- prirai 5y ago
  Yep and that's great. Whole of Fediverse, Matrix, IRC, Telegram works just fine. Lesson: Never depend on a centralized server.
  - ufmace 5y ago
    Seems like the opposite lesson to me - Telegram is kind of iffy, not sure about the others. The only thing that's been really solid today is Google. So I guess depend on multiple enormous centralized services?
  - joecool1029 5y ago
    Telegram is buckling under this too. Only major comms for me not slowing down at the moment are Fastmail (email) and Matrix (which we have the homeserver on our own ASN)
    - BoxOfRain 5y ago
      Discord is working for me, at least the text is.
  - Dylan16807 5y ago
    The underlying problem is that HN is near the limit of its hardware all the time, which could happen on one server or on 200 servers. And the reason it's near the limit is because it's single-threaded, I believe. Fix it to use multiple threads, and one centralized server will handle load spikes like this just fine.
    - klibertp 5y ago
      Nginx is single-threaded. At least was when I last looked. Node.js similarly is single-threaded. Both boast pretty impressive performance. Being single-threaded is not a detriment to performance, rather, it most often results in faster software, per core, due to lack of the need for synchronization and avoiding a memory allocation that happens for each thread (can be a problem, see C10K problem). Being single-threaded does not mean you can't do things concurrently. As long as you have a set of async primitives, starting with the venerable select(2), and a bit of language syntactic sugar (preferably, or just setjmp(3)) you can interleave handling of incoming and outgoing data in what is called "cooperative multitasking". There are many higher level frameworks built on top of the principle, which is that - hidden or visible - context switches are voluntary only. As HN seems to be written in Arc, which shares the implementation with Racket, then the feature most likely to be used for this is either call/cc (or one of the higher-level abstractions built upon it, like generators) or it would, going past the "cooperative" territory, simply use threads: https://docs.racket-lang.org/reference/threads.html https://docs.racket-lang.org/reference/threads.html These threads are not "real" threads, only one instruction among all threads ever executes at once, but the points of context switching are not specified in the code being executed (leading to preemptive multitasking but without parallelism). If HN uses those, then it can be called both "single-threaded" (the VM) and "multi-threaded" (the HN code) at the same time. Being single-threaded also doesn't prevent you from using multiple cores to max out your processing power. In age old tradition you can just spawn off a few worker processes and call it a day. You might need some sort of Inter-Process Communication, but there are many mechanisms to choose from. Nowadays on Linux you can use the listening socket as a simple round-robin scheduler for processes, free of charge, no need for a separate proxy. The processes might be managed by external supervisor, like systemd, or they might be managed by the language runtime itself, and Racket provides Places for this IIRC: https://docs.racket-lang.org/reference/places.html https://docs.racket-lang.org/reference/places.html Note: I'm just guessing based on what I've heard about HN implementation and my knowledge of Racket which may, or may not, be relevant at this point. Anyway: rewriting the HN backend to "use multiple OS-level, preemptively scheduled threads" (which you wanted to write but contracted to "multiple threads"), would not result in per-core performance increase, it's actually more probable to cause the exact opposite. I'd guess that the backend is already optimized in terms of algorithms and data storage, so the only recourse now would be to rewrite the whole thing in (or compile down to) a language that tends to perform better than what Racket VM can offer. Given that HN is already a weekend project of a hacker (I mean, written in a Lisp? A Lisp implemented on (former) Scheme...), I'd probably give Nginx + LuaJIT a whirl, or maybe one of the BEAM languages (there's a Lisp over there, too, so another fun weekend for porting Arc to LFE) if manual control over distribution over multiple nodes is desirable (but it would be a total time-sink, while things like Kubernetes (I'm told) mostly work well enough... though K8s is a time-sink in its own right...) Dang mentioned in another comment the GC being too slow. In that case, replacing the GC implementation could help, if possible. It's also possible to write code that avoids as many allocations as possible, but it's hard, and the GC can still happen at bad moments. In some cases, simply disabling the GC altogether and restarting the process periodically can also give better - or at least more predictable - performance. This is indeed a troubling problem to have, but it's completely orthogonal to single- or multi-thrededness. (Other than the multi-threding tends to allocate more memory for the same workload) So yeah, that was a tangent, but your statement was very wrong, yet very common, so I thought I'd do my duty as a good techie and shed some light on the matter.

oneeyedpigeon 5y ago

> a massive number of HN users It'll be this, I'd bet money on it. Lots like me who don't visit every day are coming to HN for more info. This site probably only ever sees a tiny percentage of its registered users accessing at once; right now, it'll be a much larger percent.

hinfaits 5y ago
I am one of the many that only goes to HN when a major service appears down (usually AWS)

nicoburns 5y ago

IIRC HN runs on a single server. So they don't really put much effort into scaling to higher traffic levels. They don't really need to. Even now the site is available, if a little slow.

prirai 5y ago
Yep and that's great. Whole of Fediverse, Matrix, IRC, Telegram works just fine. Lesson: Never depend on a centralized server.
- ufmace 5y ago
  Seems like the opposite lesson to me - Telegram is kind of iffy, not sure about the others. The only thing that's been really solid today is Google. So I guess depend on multiple enormous centralized services?
- joecool1029 5y ago
  Telegram is buckling under this too. Only major comms for me not slowing down at the moment are Fastmail (email) and Matrix (which we have the homeserver on our own ASN)
  - BoxOfRain 5y ago
    Discord is working for me, at least the text is.
- Dylan16807 5y ago
  The underlying problem is that HN is near the limit of its hardware all the time, which could happen on one server or on 200 servers. And the reason it's near the limit is because it's single-threaded, I believe. Fix it to use multiple threads, and one centralized server will handle load spikes like this just fine.
  - klibertp 5y ago
    Nginx is single-threaded. At least was when I last looked. Node.js similarly is single-threaded. Both boast pretty impressive performance. Being single-threaded is not a detriment to performance, rather, it most often results in faster software, per core, due to lack of the need for synchronization and avoiding a memory allocation that happens for each thread (can be a problem, see C10K problem). Being single-threaded does not mean you can't do things concurrently. As long as you have a set of async primitives, starting with the venerable select(2), and a bit of language syntactic sugar (preferably, or just setjmp(3)) you can interleave handling of incoming and outgoing data in what is called "cooperative multitasking". There are many higher level frameworks built on top of the principle, which is that - hidden or visible - context switches are voluntary only. As HN seems to be written in Arc, which shares the implementation with Racket, then the feature most likely to be used for this is either call/cc (or one of the higher-level abstractions built upon it, like generators) or it would, going past the "cooperative" territory, simply use threads: https://docs.racket-lang.org/reference/threads.html https://docs.racket-lang.org/reference/threads.html These threads are not "real" threads, only one instruction among all threads ever executes at once, but the points of context switching are not specified in the code being executed (leading to preemptive multitasking but without parallelism). If HN uses those, then it can be called both "single-threaded" (the VM) and "multi-threaded" (the HN code) at the same time. Being single-threaded also doesn't prevent you from using multiple cores to max out your processing power. In age old tradition you can just spawn off a few worker processes and call it a day. You might need some sort of Inter-Process Communication, but there are many mechanisms to choose from. Nowadays on Linux you can use the listening socket as a simple round-robin scheduler for processes, free of charge, no need for a separate proxy. The processes might be managed by external supervisor, like systemd, or they might be managed by the language runtime itself, and Racket provides Places for this IIRC: https://docs.racket-lang.org/reference/places.html https://docs.racket-lang.org/reference/places.html Note: I'm just guessing based on what I've heard about HN implementation and my knowledge of Racket which may, or may not, be relevant at this point. Anyway: rewriting the HN backend to "use multiple OS-level, preemptively scheduled threads" (which you wanted to write but contracted to "multiple threads"), would not result in per-core performance increase, it's actually more probable to cause the exact opposite. I'd guess that the backend is already optimized in terms of algorithms and data storage, so the only recourse now would be to rewrite the whole thing in (or compile down to) a language that tends to perform better than what Racket VM can offer. Given that HN is already a weekend project of a hacker (I mean, written in a Lisp? A Lisp implemented on (former) Scheme...), I'd probably give Nginx + LuaJIT a whirl, or maybe one of the BEAM languages (there's a Lisp over there, too, so another fun weekend for porting Arc to LFE) if manual control over distribution over multiple nodes is desirable (but it would be a total time-sink, while things like Kubernetes (I'm told) mostly work well enough... though K8s is a time-sink in its own right...) Dang mentioned in another comment the GC being too slow. In that case, replacing the GC implementation could help, if possible. It's also possible to write code that avoids as many allocations as possible, but it's hard, and the GC can still happen at bad moments. In some cases, simply disabling the GC altogether and restarting the process periodically can also give better - or at least more predictable - performance. This is indeed a troubling problem to have, but it's completely orthogonal to single- or multi-thrededness. (Other than the multi-threding tends to allocate more memory for the same workload) So yeah, that was a tangent, but your statement was very wrong, yet very common, so I thought I'd do my duty as a good techie and shed some light on the matter.