4 ms·

BERT and Ernie: Scaling your Ruby site with Erlang

17 points by fogus 17y ago

mrduncan 17y ago
There are some other really interesting looking talks at RubyConf this year also: http://rubyconf.org/pages/program http://rubyconf.org/pages/program. I only wish that it hadn't sold out so quickly.
oomkiller 17y ago
Why would you choose an approach like this instead of using tmm1-amqp + rabbitmq? A message queue seems like it would work well, and is the logical step up from delayed_job.
- defunkt 17y ago
  You could use BERT with amqp, as it's just a data format (like JSON). Ernie and BERT-RPC are libraries which build on that.
  - oomkiller 17y ago
    Ok, but why did you decide to build something rather than use an approach like I described? Is it because you needed synchronous requests, not all of them were async?
    - defunkt 17y ago
      Definitely needed fast synchronous requests. I know you can fake this in RabbitMQ but am not sure why we didn't go with it... guess you'll have to wait for Tom's presentation :)
charlesju 17y ago
I think the problem with solutions like these it usually comes down throwing down money at more development (Erlang is hard) or throwing down money on more servers (delayed_job, Amazon SQS, etc.). I usually prefer money on servers since it's faster to develop for my team. But I guess for the functionally gifted, this is a great solution.
- defunkt 17y ago
  We were constrained by network our file system and could not add more machines. We needed an RPC solution to shard the data and move away from our network file system. delayed_job, Amazon SQS, etc are not RPC systems. You can't block waiting on a response like you can with messaging systems. For instance, delayed_job makes an RPC call that asks "how much disk space is this repository using?" It waits until it gets a response from an Ernie handler then continues with its background job.
- deleted 17y ago
  [deleted]
- oomkiller 17y ago
  I wouldn't say Erlang is a "hard" language, especially once you understand it. Sure, there aren't a whole hell of a lot of people that are proficient in it, but I think that's just because it's a little obscure. To someone with a little experience with functional programming, Erlang should make a lot of sense.
  - silentbicycle 17y ago
    Agreed. I'm learning it now, and while there are some new ideas in the concurrency model, it doesn't strike me as hard to learn or use the way (say) Haskell and C++ are. There's a fairly small number of core elements, and the language design as a whole strongly encourages you to make things loudly crash when anything is wrong, rather than passing on corrupt data. That helps immensely with debugging... Also, I found that a lot of concepts from Prolog carry over nicely. Erlang was initially implemented in Prolog, and that's where it gets pattern matching and immutable variables from. (Along with several stylistic / syntactic quirks.)
- charlesju 17y ago
  Yeah I'm sure everyone has their own opinions on how to solve this problem (scaling in general), this is just my approach. I'm sure your approach is just as, or more, valid.
4buot 17y ago
Not really understanding why a bizarre homebrew solution was better than Thrift...?

17 points by fogus 17y ago

There are some other really interesting looking talks at RubyConf this year also: http://rubyconf.org/pages/program http://rubyconf.org/pages/program. I only wish that it hadn't sold out so quickly.

oomkiller 17y ago

Why would you choose an approach like this instead of using tmm1-amqp + rabbitmq? A message queue seems like it would work well, and is the logical step up from delayed_job.

defunkt 17y ago
You could use BERT with amqp, as it's just a data format (like JSON). Ernie and BERT-RPC are libraries which build on that.
- oomkiller 17y ago
  Ok, but why did you decide to build something rather than use an approach like I described? Is it because you needed synchronous requests, not all of them were async?
  - defunkt 17y ago
    Definitely needed fast synchronous requests. I know you can fake this in RabbitMQ but am not sure why we didn't go with it... guess you'll have to wait for Tom's presentation :)

charlesju 17y ago

I think the problem with solutions like these it usually comes down throwing down money at more development (Erlang is hard) or throwing down money on more servers (delayed_job, Amazon SQS, etc.). I usually prefer money on servers since it's faster to develop for my team. But I guess for the functionally gifted, this is a great solution.

defunkt 17y ago
We were constrained by network our file system and could not add more machines. We needed an RPC solution to shard the data and move away from our network file system. delayed_job, Amazon SQS, etc are not RPC systems. You can't block waiting on a response like you can with messaging systems. For instance, delayed_job makes an RPC call that asks "how much disk space is this repository using?" It waits until it gets a response from an Ernie handler then continues with its background job.
deleted 17y ago
[deleted]
oomkiller 17y ago
I wouldn't say Erlang is a "hard" language, especially once you understand it. Sure, there aren't a whole hell of a lot of people that are proficient in it, but I think that's just because it's a little obscure. To someone with a little experience with functional programming, Erlang should make a lot of sense.
- silentbicycle 17y ago
  Agreed. I'm learning it now, and while there are some new ideas in the concurrency model, it doesn't strike me as hard to learn or use the way (say) Haskell and C++ are. There's a fairly small number of core elements, and the language design as a whole strongly encourages you to make things loudly crash when anything is wrong, rather than passing on corrupt data. That helps immensely with debugging... Also, I found that a lot of concepts from Prolog carry over nicely. Erlang was initially implemented in Prolog, and that's where it gets pattern matching and immutable variables from. (Along with several stylistic / syntactic quirks.)
charlesju 17y ago
Yeah I'm sure everyone has their own opinions on how to solve this problem (scaling in general), this is just my approach. I'm sure your approach is just as, or more, valid.

4buot 17y ago

Not really understanding why a bizarre homebrew solution was better than Thrift...?