6 ms·

> In-memory only for now, disk backed durability is on the roadmap. > Job payload & results are limited to 1 MiB each. > Workq servers are standalone and do n

> In-memory only for now, disk backed durability is on the roadmap. > Job payload & results are limited to 1 MiB each. > Workq servers are standalone and do not speak to each other. i.e. don't use it in production. This is a nice proof of concept, but let's not pretend that it is a professional grade product at the moment.

saghul 10y ago
I'm not the author, but I don't think anyone here pretended that it's a "professional grade product". From the actual README: "Workq is in alpha status and not yet stable."
- bryanlarsen 10y ago
  Sure, but it looks like he intends it to eventually be a professional grade product. Disk-backed durability can be added, but adding distributed/failover capability should be designed in from the beginning. And nobody's ever going to trust your distributed capability if you implement RAFT yourself. You need to build upon something trusted or convince Aphyr to run Jepsen on your implementation. etcd is a common thing to build upon, and even it is only partially trusted.
  - eitland 10y ago
    > And nobody's ever going to trust your ... I guess that could be said for a number of things that people successfully do? Edit: for one great example of someone who didn't get discouraged by naysayers, check caddyserver.
    - bryanlarsen 10y ago
      Great example. Nobody's going to use caddyserver in production for a major site for a couple years for that reason. Right now it's widely used on 'hobby' sites. A couple of years of good track record on hobby sites will let it be trusted enough to run on more mission critical sites. And caddyserver isn't a distributed service, so the level of trust required is much lower. Distributed services are difficult to get right for a wide variety of reasons, as shown by Aphyr's Jepsen tests.
      - eitland 10y ago
        It is however already starting to get developer traction and mindshare, even a few donations if I am representative for the userbase, something it would never do now if sat waiting for the perfect timing to release a perfect product. Also I think the README of workq didn't mention anything about raft, which is fine, you can go very far without it. My point is: encourage people to write code! Don't infect people with paralysis-by-analysis.
  - iamduo 10y ago
    This is the same reason why Workq is intended to be standalone. It takes so much expertise and time to implement this in a distributed nature. I spent 80% of time on Workq just writing tests for it and there is still so much to account for even as a standalone system. On the bright side, projects like etcd & consul (I believe the author of Raft is helping out there) are getting better and better and can be embedded.
    - bryanlarsen 10y ago
      Cool. If you think you can embed etcd or consul that'd be awesome. Please post again if/when that happens!
  - jalfresi 10y ago
    Not everyone needs a distributed solution. Sometimes, "beefing up the box" is enough for most systems, not everyone needs Facebook grade scaling tools. As they say, scaling is a nice problem to have.
    - bryanlarsen 10y ago
      True, I don't need scaling. I do need robustness to hardware failure, though.

saghul 10y ago

I'm not the author, but I don't think anyone here pretended that it's a "professional grade product". From the actual README: "Workq is in alpha status and not yet stable."

bryanlarsen 10y ago
Sure, but it looks like he intends it to eventually be a professional grade product. Disk-backed durability can be added, but adding distributed/failover capability should be designed in from the beginning. And nobody's ever going to trust your distributed capability if you implement RAFT yourself. You need to build upon something trusted or convince Aphyr to run Jepsen on your implementation. etcd is a common thing to build upon, and even it is only partially trusted.
- eitland 10y ago
  > And nobody's ever going to trust your ... I guess that could be said for a number of things that people successfully do? Edit: for one great example of someone who didn't get discouraged by naysayers, check caddyserver.
  - bryanlarsen 10y ago
    Great example. Nobody's going to use caddyserver in production for a major site for a couple years for that reason. Right now it's widely used on 'hobby' sites. A couple of years of good track record on hobby sites will let it be trusted enough to run on more mission critical sites. And caddyserver isn't a distributed service, so the level of trust required is much lower. Distributed services are difficult to get right for a wide variety of reasons, as shown by Aphyr's Jepsen tests.
    - eitland 10y ago
      It is however already starting to get developer traction and mindshare, even a few donations if I am representative for the userbase, something it would never do now if sat waiting for the perfect timing to release a perfect product. Also I think the README of workq didn't mention anything about raft, which is fine, you can go very far without it. My point is: encourage people to write code! Don't infect people with paralysis-by-analysis.
- iamduo 10y ago
  This is the same reason why Workq is intended to be standalone. It takes so much expertise and time to implement this in a distributed nature. I spent 80% of time on Workq just writing tests for it and there is still so much to account for even as a standalone system. On the bright side, projects like etcd & consul (I believe the author of Raft is helping out there) are getting better and better and can be embedded.
  - bryanlarsen 10y ago
    Cool. If you think you can embed etcd or consul that'd be awesome. Please post again if/when that happens!
- jalfresi 10y ago
  Not everyone needs a distributed solution. Sometimes, "beefing up the box" is enough for most systems, not everyone needs Facebook grade scaling tools. As they say, scaling is a nice problem to have.
  - bryanlarsen 10y ago
    True, I don't need scaling. I do need robustness to hardware failure, though.