4 ms·

Great article, thanks. [from TFA] > This however does come with a cost: > - We cannot use Clojure’s concurrency primitives (futures/promises/agents).

by arohner 12y ago

Great article, thanks. [from TFA] > This however does come with a cost: > - We cannot use Clojure’s concurrency primitives (futures/promises/agents). Running my own large-ish Clojure service (CircleCI), I'm pretty convinced that futures are an anti-pattern in many situations. If your future is for side-effects, rather than for computation, the future call should probably be going on a durable queue being processed by a cluster of machines.

yayitswei 12y ago
What are you guys using for durable queues? I'm currently evaluating Redis and RabbitMQ.
- coolsunglasses 12y ago
  I wouldn't use either of those if you need durability.
  - arjie 12y ago
    What would you use? And why not RabbitMQ durable queues?
    - tensor 12y ago
      Probably referring to: http://aphyr.com/posts/315-call-me-maybe-rabbitmq http://aphyr.com/posts/315-call-me-maybe-rabbitmq But all systems like this have weak points, just look at the other systems aphyr has analyzed.
    - erichmond 12y ago
      Kafka may be worth a look. It's the best message delivery system I've seen in the wild.
      - hkarthik 12y ago
        Anyone using Kafka with AWS? I'm concerned that it won't be partition tolerant enough to use it there. http://aphyr.com/posts/293-call-me-maybe-kafka http://aphyr.com/posts/293-call-me-maybe-kafka We chose RabbitMQ, but we're achieving durability by storing messages in a backup database (CouchBase) that appears to handle partitions better. Still not ideal, and more complicated than we'd like.
  - imperialWicket 12y ago
    This isn't helpful. It's a perfectly valid question that even provides insight into some possible candidates. The candidates might not be the best options, but they commonly serve this purpose. What is it about RabbitMQ or Redis durability that you find lacking? What alternative do you recommend?
  - tensor 12y ago
    I've looked and have not found any alternatives that are better.
    - coolsunglasses 12y ago
      Did you rule out Kafka? If so, on what basis?
- lgierth 12y ago
  Pairs of RabbitMQ brokers, used in a Publish-One-Subscribe-Many fashion for availability. Scaling RabbitMQ at SoundCloud: http://vimeo.com/69960105 http://vimeo.com/69960105 Edit: usually with durable queues on SSDs Edit: text version of the talk: http://blog.pivotal.io/pivotal/case-studies-2/scaling-with-rabbitmq-soundcloud http://blog.pivotal.io/pivotal/case-studies-2/scaling-with-r...
- Scuds 12y ago
  and here I am, staring down the barrel of Erlang. again
  - therockhead 12y ago
    What type of issues are you having with Erlang?
smrtinsert 12y ago
That's a very interesting way of looking at it.
efuquen 12y ago
I don't quite get what you mean. The author's example is a GET request on some web service, are you arguing we should always be using queues instead of a request/response models when talking to external services?
- arohner 12y ago
  No. My stereotypical use case is say, responding to an HTTP request that involves side-effects: (post "/user/foo" [request] (update-db) (future (send-user-email)) {:status 200 :body (render-page)}) What happens if email delivery fails? What happens if this box powers off? etc. Queues don't solve all your failure scenarios here, but they're a great first step.
  - rybosome 12y ago
    I'm not sure that I understand. The future implementations that I'm aware of allow you to take different actions if a future succeeds or fails; in the above, couldn't you render a 500 if the email didn't send or the db update errored out? Furthermore, futures and work queues are not mutually exclusive. A future transformation could easily be the result of submitting work to a queue asynchronously; since you're talking about a separate service (possibly non-local), this submission itself isn't guaranteed to succeed.
    - Confusion 12y ago
      I think arohner's point is that as soon as you use a separate work queue for asynchronous actions that are only executed for their side effect, then you don't need explicit futures anymore. Futures are building material for libraries and not something to be used directly in your code, because they need adornments that need abstraction. As soon as you want to make the use of futures robust, you will basically start reimplementing parts of a work queue library. E.g. futures assume they will be handled within the lifetime of the current process: they do not survive catastrophic process failure. If you want to overcome that assumption with persistence, you can either reinvent a wheel, or use a work queuing library that has already implemented this for you. E.g. you can reinvent your own generic on-fail-retry or you can submit the work to a queue managed by library code that has already implemented retrying for you.

by arohner 12y ago

yayitswei 12y ago

What are you guys using for durable queues? I'm currently evaluating Redis and RabbitMQ.

coolsunglasses 12y ago
I wouldn't use either of those if you need durability.
- arjie 12y ago
  What would you use? And why not RabbitMQ durable queues?
  - tensor 12y ago
    Probably referring to: http://aphyr.com/posts/315-call-me-maybe-rabbitmq http://aphyr.com/posts/315-call-me-maybe-rabbitmq But all systems like this have weak points, just look at the other systems aphyr has analyzed.
  - erichmond 12y ago
    Kafka may be worth a look. It's the best message delivery system I've seen in the wild.
    - hkarthik 12y ago
      Anyone using Kafka with AWS? I'm concerned that it won't be partition tolerant enough to use it there. http://aphyr.com/posts/293-call-me-maybe-kafka http://aphyr.com/posts/293-call-me-maybe-kafka We chose RabbitMQ, but we're achieving durability by storing messages in a backup database (CouchBase) that appears to handle partitions better. Still not ideal, and more complicated than we'd like.
- imperialWicket 12y ago
  This isn't helpful. It's a perfectly valid question that even provides insight into some possible candidates. The candidates might not be the best options, but they commonly serve this purpose. What is it about RabbitMQ or Redis durability that you find lacking? What alternative do you recommend?
- tensor 12y ago
  I've looked and have not found any alternatives that are better.
  - coolsunglasses 12y ago
    Did you rule out Kafka? If so, on what basis?
lgierth 12y ago
Pairs of RabbitMQ brokers, used in a Publish-One-Subscribe-Many fashion for availability. Scaling RabbitMQ at SoundCloud: http://vimeo.com/69960105 http://vimeo.com/69960105 Edit: usually with durable queues on SSDs Edit: text version of the talk: http://blog.pivotal.io/pivotal/case-studies-2/scaling-with-rabbitmq-soundcloud http://blog.pivotal.io/pivotal/case-studies-2/scaling-with-r...
Scuds 12y ago
and here I am, staring down the barrel of Erlang. again
- therockhead 12y ago
  What type of issues are you having with Erlang?

smrtinsert 12y ago

That's a very interesting way of looking at it.

efuquen 12y ago

I don't quite get what you mean. The author's example is a GET request on some web service, are you arguing we should always be using queues instead of a request/response models when talking to external services?

arohner 12y ago
No. My stereotypical use case is say, responding to an HTTP request that involves side-effects: (post "/user/foo" [request] (update-db) (future (send-user-email)) {:status 200 :body (render-page)}) What happens if email delivery fails? What happens if this box powers off? etc. Queues don't solve all your failure scenarios here, but they're a great first step.
- rybosome 12y ago
  I'm not sure that I understand. The future implementations that I'm aware of allow you to take different actions if a future succeeds or fails; in the above, couldn't you render a 500 if the email didn't send or the db update errored out? Furthermore, futures and work queues are not mutually exclusive. A future transformation could easily be the result of submitting work to a queue asynchronously; since you're talking about a separate service (possibly non-local), this submission itself isn't guaranteed to succeed.
  - Confusion 12y ago
    I think arohner's point is that as soon as you use a separate work queue for asynchronous actions that are only executed for their side effect, then you don't need explicit futures anymore. Futures are building material for libraries and not something to be used directly in your code, because they need adornments that need abstraction. As soon as you want to make the use of futures robust, you will basically start reimplementing parts of a work queue library. E.g. futures assume they will be handled within the lifetime of the current process: they do not survive catastrophic process failure. If you want to overcome that assumption with persistence, you can either reinvent a wheel, or use a work queuing library that has already implemented this for you. E.g. you can reinvent your own generic on-fail-retry or you can submit the work to a queue managed by library code that has already implemented retrying for you.