5 ms·

asciinema.org is currently being hugged to death! The btop stream is from one of the servers, currently reporting 95% CPU usage: https://asciinema.org/s/olesiD0

by ku1ik 1y ago

asciinema.org is currently being hugged to death! The btop stream is from one of the servers, currently reporting 95% CPU usage: https://asciinema.org/s/olesiD03BIFH6Yz1 https://asciinema.org/s/olesiD03BIFH6Yz1 But it's chugging along! asciinema server is Elixir/Phoenix, runs on BEAM, and even with lots of connections and high CPU usage it serves request fine. Power of the BEAM. Phew. Currently it runs on just 2 VMs, 2 GB of RAM each, so I will probably need to scale it up very soon anyway :)

Yoric 1y ago
Long live the BEAM!
darkamaul 1y ago
In an age where everything is about clouds, it’s pretty amazing to see a high-profile service like this running smoothly on just a couple of 2 GB VMs. That’s less RAM than a mid-range laptop, and those machines struggle to run a modern browser.
- imiric 1y ago
  Modern hardware in conjunction with smart software choices can be very capable, but it's pretty negligent to run a service people depend on with such limited resources. Sure, not every site needs to run on a highly-available elastic cluster of nodes, but the alternative mindset of actively rejecting this architecture is arguably even more harmful. A healthy deployment should be able to scale up and down easily, if not automatically. I wouldn't want to be woken up at 3am because prod1 needs a CPU/RAM/disk increase and a reboot, and the users wouldn't like that either.
  - phaer 1y ago
    Sounds preferable compared to be woken up at 5 am by your cloud providers billing alert.
  - arccy 1y ago
    > service people depend on why wouldn't you cap a free service? imo it's more irresponsible to demand other people keep a thing you don't pay for running just because you happen to depend on it.
    - imiric 1y ago
      Just because a service doesn't require direct payment doesn't mean that its users shouldn't have any expectations of availability. There should be fair usage limits, of course, even if the service is commercial, but being caught by surprise by a traffic surge and boasting that it runs on underpowered hardware when the project is 10+ years old is irresponsible IMO. Also, any service can have a financial purpose, even if it's "free" for users. This very forum is "free", yet the cost of hosting it is indirectly paid for by being an ad for the company that runs it. HN also famously runs on underpowered hardware that routinely fails to meet demand, but that is beside the point.
      - ku1ik 1y ago
        Feel free to NOT use asciinema.org if it doesn't meet your expectations for a reliable service. You can always self-host the server in your ultra-reliable, super-scalable K8S cluster.
  - ku1ik 1y ago
    There’s few reasons for this. First, I don’t have infinite budget for this, in fact the budget is close to 0 (sponsorships and donations help). Second, the servers are sponsored by Brightbox, and I’ve been running asciinema.org there for over a decade because BB is reliable and they’re great chaps, but I don’t have infinite resources available there. Finally, I don’t want devops complexity in my side projects.
- imtringued 1y ago
  Webservers require an insignificant amount of CPU time in comparison to any graphical interface.
lostmsu 1y ago
Thanks for the great service! Is there a way to mirror the streams? Like a Wikipedia dump?
imiric 1y ago
Tip: you might want to decrease the btop refresh rate. It's likely that at 100ms a considerable amount of CPU time is taken up by btop itself.
- ku1ik 1y ago
  The CPU usage was single digit (%) with no viewers (but other requests being served) with btop at 100ms, so that's not adding a lot. But thanks!
mayli 1y ago
That btop stream is lagging hard now, and jumping between full cpu and idle 0%. Is the service crashing and restarting?
- ku1ik 1y ago
  I was adding more VMs and restarting load balancer. Should be good now.
ku1ik 1y ago
Just scaled it up, seems pretty stable and responsive again.

by ku1ik 1y ago

Yoric 1y ago

Long live the BEAM!

darkamaul 1y ago

In an age where everything is about clouds, it’s pretty amazing to see a high-profile service like this running smoothly on just a couple of 2 GB VMs. That’s less RAM than a mid-range laptop, and those machines struggle to run a modern browser.

imiric 1y ago
Modern hardware in conjunction with smart software choices can be very capable, but it's pretty negligent to run a service people depend on with such limited resources. Sure, not every site needs to run on a highly-available elastic cluster of nodes, but the alternative mindset of actively rejecting this architecture is arguably even more harmful. A healthy deployment should be able to scale up and down easily, if not automatically. I wouldn't want to be woken up at 3am because prod1 needs a CPU/RAM/disk increase and a reboot, and the users wouldn't like that either.
- phaer 1y ago
  Sounds preferable compared to be woken up at 5 am by your cloud providers billing alert.
- arccy 1y ago
  > service people depend on why wouldn't you cap a free service? imo it's more irresponsible to demand other people keep a thing you don't pay for running just because you happen to depend on it.
  - imiric 1y ago
    Just because a service doesn't require direct payment doesn't mean that its users shouldn't have any expectations of availability. There should be fair usage limits, of course, even if the service is commercial, but being caught by surprise by a traffic surge and boasting that it runs on underpowered hardware when the project is 10+ years old is irresponsible IMO. Also, any service can have a financial purpose, even if it's "free" for users. This very forum is "free", yet the cost of hosting it is indirectly paid for by being an ad for the company that runs it. HN also famously runs on underpowered hardware that routinely fails to meet demand, but that is beside the point.
    - ku1ik 1y ago
      Feel free to NOT use asciinema.org if it doesn't meet your expectations for a reliable service. You can always self-host the server in your ultra-reliable, super-scalable K8S cluster.
- ku1ik 1y ago
  There’s few reasons for this. First, I don’t have infinite budget for this, in fact the budget is close to 0 (sponsorships and donations help). Second, the servers are sponsored by Brightbox, and I’ve been running asciinema.org there for over a decade because BB is reliable and they’re great chaps, but I don’t have infinite resources available there. Finally, I don’t want devops complexity in my side projects.
imtringued 1y ago
Webservers require an insignificant amount of CPU time in comparison to any graphical interface.

lostmsu 1y ago

Thanks for the great service! Is there a way to mirror the streams? Like a Wikipedia dump?

imiric 1y ago

Tip: you might want to decrease the btop refresh rate. It's likely that at 100ms a considerable amount of CPU time is taken up by btop itself.

ku1ik 1y ago
The CPU usage was single digit (%) with no viewers (but other requests being served) with btop at 100ms, so that's not adding a lot. But thanks!

mayli 1y ago

That btop stream is lagging hard now, and jumping between full cpu and idle 0%. Is the service crashing and restarting?

ku1ik 1y ago
I was adding more VMs and restarting load balancer. Should be good now.

ku1ik 1y ago

Just scaled it up, seems pretty stable and responsive again.