6 ms·

When HHVM doesn't work quite right

87 points by martin_ 11y ago

ck2 11y ago
3.8 and 3.9-dev are way more stable than any previous version. We used to have to restart hhvm weekly to prevent white screens of death but no longer with 3.8+ 3.8 is also 10% faster than previous versions: http://hhvm.com/blog/9803/hhvm-3-8-0 http://hhvm.com/blog/9803/hhvm-3-8-0
- dsp1234 11y ago
  I just started a new Laravel project and decided to try hhvm. Unfortunately, on 3.8.0 it segfaults under load just serving the default Laravel page. It's really unfortunate since I'd love to give hhvm a shot.
  - debacle 11y ago
    hhvm has tested compatibility with Laravel. Are you sure you were doing everything right?
    - raziel2p 11y ago
      A PHPUnit test suite passing and being stable under high load are two entirely unrelated things.
      - girvo 11y ago
        And having a local project segfaulting with zero extra code is something else altogether.
  - ck2 11y ago
    Are you sure you built it right? Maybe try 3.9-dev (trunk)to see if it was a bug.
  - martin_ 11y ago
    When did you try it? HHVM 3.8 was only released officially a week ago, perhaps you were on the unstable branch?
    - dsp1234 11y ago
      I just started the project 2 days ago, and it was installed from the official HHVM repository: sudo apt-get install software-properties-common sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0x5a16e7281be7a449 sudo add-apt-repository "deb http://dl.hhvm.com/ubuntu http://dl.hhvm.com/ubuntu $(lsb_release -sc) main" sudo apt-get update sudo apt-get install hhvm $hhvm --version HipHop VM 3.8.0 (rel) Compiler: tags/HHVM-3.8.0-0-gbb32f8d645b28561d8491870c44730b9b3f08d35 Repo schema: 74bee5d310450724cfd49245fd6c2e1e4323b10a
      - martin_ 11y ago
        if you can run it under gdb, repro, and get a "thread apply all bt" that'd be very helpful. The HHVM team is very responsive on GitHub issues!
ubercore 11y ago
You might check out http://circus.readthedocs.org/en/latest/ http://circus.readthedocs.org/en/latest/ instead of supervisord; you can restart managed processes on a schedule, and it also publishes events to zmq and statsd for integration with any other tooling you might want to use for health checks.
- martin_ 11y ago
  Sweet thanks! I had to make a workaround because supervisord doesn't have a way to serially restart "groups". Will give this a shot!
- viraptor 11y ago
  Even that is not necessary. If the original service was running under systemd, why not just use service instances? (see description in https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files https://www.digitalocean.com/community/tutorials/understandi... for example)
scragg 11y ago
Sweet, I am going to borrow your health check script. I can't wait until official packaged support for the ZMQ extension.
Navarr 11y ago
Who goes "this software has stability problems" and decides the correct solution is to go with the even more unstable branch?
- fapjacks 11y ago
  People that have identified leaks affecting them which are fixed on the nightlies?
  - Twirrim 11y ago
    People trading some known bugs for a whole heap of new ones, ones likely to change all the time? Doesn't sound like the smartest operations move. Given they've gone and written stuff to automatically restart the processes under certain conditions anyway, might as well just catch the memory leak and restart, rather than gamble with "not for production use" software in production environments.
    - martin_ 11y ago
      The build we're using was considered pretty safe and was performing better for us than 3.7.x - of course we aren't grabbing the new nightly constantly and plan to move to a "stable" branch soon :)
- coldtea 11y ago
  Pragmatists, who know that "stable" and "unstable" branch are just tags, and what's important is the actual stability of the software in each as it relates to their use case.
- Intermernet 11y ago
  Even worse, I've been known to cherry-pick commits from different unstable branches to get a semblance of "what I need". It's not pretty, but we do what we need to do.
TOGoS 11y ago
That's cool I guess, but this article is more about setting up a load balancer than it is about HHVM. I was hoping for some juicy bit about it completely hosing up on some obscure PHP function.
smrtinsert 11y ago
in the time you read this article, facebook developers have added another 1500 lines of php and pushed most of it to production.
daurnimator 11y ago
> First, I disabled HHVM via systemd and substituted in supervisord. Why? what does supervisord have that systemd doesn't provide?
- martin_ 11y ago
  I'm not aware of a method to manage pools with systemd, also it requires processes to be daemonized which would make configuration a little messier. Also I'm simply more familiar with supervisord as systemd only recently became the default init system in debian.
  - ianlevesque 11y ago
    Not sure about the other merits, but systemd definitely does not require daemonized processes. http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/systemd-house-of-horror/daemonize.html http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/sys...
  - daurnimator 11y ago
    > I'm not aware of a method to manage pools with systemd I'm not sure what you mean by manage pools. Maybe you're asking how to make multiple processes? Check out unit templates, they're the ones with an '@' in them. see `man systemd.unit` for more info. > also it requires processes to be daemonized which would make configuration a little messier. It does not. infact, it prefers you don't. Check out the various service unit types. see `man systemd.service` or http://www.freedesktop.org/software/systemd/man/systemd.service.html#Type= http://www.freedesktop.org/software/systemd/man/systemd.serv...
WillyNourson 11y ago
"nginx does provide health_check in its commercial builds, but at this time I'm not too inclined to pay $1500/server/year for the pleasure." But Varnish does it for free out of the box ! Quick example from the documentation (https://www.varnish-cache.org/trac/wiki/BackendPolling https://www.varnish-cache.org/trac/wiki/BackendPolling) : backend b0 { .host = "phk.freebsd.dk"; .probe = { .url = "/probe.cgi"; .timeout = 34 ms; .interval = 1s; .window = 10; .threshold = 8; } }

87 points by martin_ 11y ago

ck2 11y ago

3.8 and 3.9-dev are way more stable than any previous version. We used to have to restart hhvm weekly to prevent white screens of death but no longer with 3.8+ 3.8 is also 10% faster than previous versions: http://hhvm.com/blog/9803/hhvm-3-8-0 http://hhvm.com/blog/9803/hhvm-3-8-0

dsp1234 11y ago
I just started a new Laravel project and decided to try hhvm. Unfortunately, on 3.8.0 it segfaults under load just serving the default Laravel page. It's really unfortunate since I'd love to give hhvm a shot.
- debacle 11y ago
  hhvm has tested compatibility with Laravel. Are you sure you were doing everything right?
  - raziel2p 11y ago
    A PHPUnit test suite passing and being stable under high load are two entirely unrelated things.
    - girvo 11y ago
      And having a local project segfaulting with zero extra code is something else altogether.
- ck2 11y ago
  Are you sure you built it right? Maybe try 3.9-dev (trunk)to see if it was a bug.
- martin_ 11y ago
  When did you try it? HHVM 3.8 was only released officially a week ago, perhaps you were on the unstable branch?
  - dsp1234 11y ago
    I just started the project 2 days ago, and it was installed from the official HHVM repository: sudo apt-get install software-properties-common sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0x5a16e7281be7a449 sudo add-apt-repository "deb http://dl.hhvm.com/ubuntu http://dl.hhvm.com/ubuntu $(lsb_release -sc) main" sudo apt-get update sudo apt-get install hhvm $hhvm --version HipHop VM 3.8.0 (rel) Compiler: tags/HHVM-3.8.0-0-gbb32f8d645b28561d8491870c44730b9b3f08d35 Repo schema: 74bee5d310450724cfd49245fd6c2e1e4323b10a
    - martin_ 11y ago
      if you can run it under gdb, repro, and get a "thread apply all bt" that'd be very helpful. The HHVM team is very responsive on GitHub issues!

ubercore 11y ago

You might check out http://circus.readthedocs.org/en/latest/ http://circus.readthedocs.org/en/latest/ instead of supervisord; you can restart managed processes on a schedule, and it also publishes events to zmq and statsd for integration with any other tooling you might want to use for health checks.

martin_ 11y ago
Sweet thanks! I had to make a workaround because supervisord doesn't have a way to serially restart "groups". Will give this a shot!
viraptor 11y ago
Even that is not necessary. If the original service was running under systemd, why not just use service instances? (see description in https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files https://www.digitalocean.com/community/tutorials/understandi... for example)

scragg 11y ago

Sweet, I am going to borrow your health check script. I can't wait until official packaged support for the ZMQ extension.

Navarr 11y ago

Who goes "this software has stability problems" and decides the correct solution is to go with the even more unstable branch?

fapjacks 11y ago
People that have identified leaks affecting them which are fixed on the nightlies?
- Twirrim 11y ago
  People trading some known bugs for a whole heap of new ones, ones likely to change all the time? Doesn't sound like the smartest operations move. Given they've gone and written stuff to automatically restart the processes under certain conditions anyway, might as well just catch the memory leak and restart, rather than gamble with "not for production use" software in production environments.
  - martin_ 11y ago
    The build we're using was considered pretty safe and was performing better for us than 3.7.x - of course we aren't grabbing the new nightly constantly and plan to move to a "stable" branch soon :)
coldtea 11y ago
Pragmatists, who know that "stable" and "unstable" branch are just tags, and what's important is the actual stability of the software in each as it relates to their use case.
Intermernet 11y ago
Even worse, I've been known to cherry-pick commits from different unstable branches to get a semblance of "what I need". It's not pretty, but we do what we need to do.

TOGoS 11y ago

That's cool I guess, but this article is more about setting up a load balancer than it is about HHVM. I was hoping for some juicy bit about it completely hosing up on some obscure PHP function.

smrtinsert 11y ago

in the time you read this article, facebook developers have added another 1500 lines of php and pushed most of it to production.

daurnimator 11y ago

> First, I disabled HHVM via systemd and substituted in supervisord. Why? what does supervisord have that systemd doesn't provide?

martin_ 11y ago
I'm not aware of a method to manage pools with systemd, also it requires processes to be daemonized which would make configuration a little messier. Also I'm simply more familiar with supervisord as systemd only recently became the default init system in debian.
- ianlevesque 11y ago
  Not sure about the other merits, but systemd definitely does not require daemonized processes. http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/systemd-house-of-horror/daemonize.html http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/sys...
- daurnimator 11y ago
  > I'm not aware of a method to manage pools with systemd I'm not sure what you mean by manage pools. Maybe you're asking how to make multiple processes? Check out unit templates, they're the ones with an '@' in them. see `man systemd.unit` for more info. > also it requires processes to be daemonized which would make configuration a little messier. It does not. infact, it prefers you don't. Check out the various service unit types. see `man systemd.service` or http://www.freedesktop.org/software/systemd/man/systemd.service.html#Type= http://www.freedesktop.org/software/systemd/man/systemd.serv...

WillyNourson 11y ago

"nginx does provide health_check in its commercial builds, but at this time I'm not too inclined to pay $1500/server/year for the pleasure." But Varnish does it for free out of the box ! Quick example from the documentation (https://www.varnish-cache.org/trac/wiki/BackendPolling https://www.varnish-cache.org/trac/wiki/BackendPolling) : backend b0 { .host = "phk.freebsd.dk"; .probe = { .url = "/probe.cgi"; .timeout = 34 ms; .interval = 1s; .window = 10; .threshold = 8; } }