5 ms·

Orchestra: An open-source system to orchestrate teams of experts and machines

121 points by marcua 11y ago

jrbotros 11y ago
Hi, Hacker News! Today we’re excited to be making our first contribution back to the tech community by open sourcing Orchestra, a system for coordinating project teams of experts and machines. It can be used to reimagine and implement all sorts of workflows, from professional services (like legal, design, and data analysis) to new product experiences (like M from Facebook). Orchestra is based upon years of research into machine-mediated expert teams. We hope you’ll come up with new and creative ways of using it! At Unlimited Labs, we’re excited about building a brighter future of work and are currently building our first product on top of Orchestra. We’d love to answer any questions you have on here, or feel free to email the team at hello@unlimitedlabs.com!
marcua 11y ago
(Orchestra dev here) If you're curious how it can be used, Daniel Haas put together a wonderful example of how Orchestra could be used in a newsroom: http://orchestra.readthedocs.org/en/latest/example_use.html http://orchestra.readthedocs.org/en/latest/example_use.html Happy to take questions!
CoreSet 11y ago
This might seem a dumb question, but can you talk about how this improves on existing productivity workflows? Does it integrate with tasking systems like Jira or is this meant to be a complete a replacement? It seems high-level (and great) I'm just curious what implementing it "in the field" would look like.
- dhaas 11y ago
  Products like Jira help organize tasks within workflows, but don't automate any that organization. Orchestra takes care of things like "which reporter should I assign to a new story?" without requiring humans to make those decisions. Research from the Flash teams project that I mentioned in another comment (http://stanfordhci.github.io/flash-teams/index.html http://stanfordhci.github.io/flash-teams/index.html) has shown that this type of automation can decrease end-to-end workflow time by as much as 2x!
usman-m 11y ago
Great stuff! For now it seems like this leaves out the job of finding crowd workers. Could it be possible to do something like designing microtasks in MTurk and "stitching" the workflow together using Orchestra?
- dhaas 11y ago
  (Orchestra dev here too!) That's a great question. We're hoping that emerging online expert communities like Upwork or Dribbble will make it easier and easier to find experts suited to the work, but it's definitely not a solved problem (or tech recruiting would be so much easier!). Re: MTurk, there's nothing stopping you from stitching microtasks together with Orchestra, but I think that undersells Orchestra's strengths (e.g., collaboration among workers) a bit.
tilek 11y ago
This is very exciting!
zallarak 11y ago
This seems very cool - are there case studies of adopting this type of workflow for complex projects?
- dhaas 11y ago
  I don't know of any formal case studies, but Adam Marcus and myself (two devs on the project) worked on a large-scale workflow at GoDaddy where we automatically organized data extraction experts into hierarchies of review. That work actually got published at a Databases conference: see Adam's nice summary here: http://blog.marcua.net/post/128120315677 http://blog.marcua.net/post/128120315677 On the research side of things, the flash teams work by Daniela Retelny at Stanford (http://stanfordhci.github.io/flash-teams/ http://stanfordhci.github.io/flash-teams/) has done great proof-of-concept things like build an app in a day with distributed teams.
ggiacaglia 11y ago
This seems very cool!!!
johndoe7 11y ago
Are musical instruments provided?
- dhaas 11y ago
  They don't come in the box, but you're free to hire musicians for your workflows!
SamBam 11y ago
Does this tie into products like PivotalTracker or JIRA to make it useful for agile software development? It would be nice to be able to create tasks from stories in Pivotal, assign a reviewer to them, and, when a reviewer thumbs-up, have it automatically built into a staging server. Anything like that in the works?
- dhaas 11y ago
  We haven't built that integration yet, but it's totally possible! Really all it should take is a Machine Step that calls out from Python to, say, Pivotal's APIs (https://pypi.python.org/pypi/pivotaltracker https://pypi.python.org/pypi/pivotaltracker).
dharma1 11y ago
i think this looks potentially great - would definitely benefit from more real world examples of workflows
keredhaas 11y ago
This looks really exciting, and I can't wait to try it. I think it will be very helpful for my business.
IanCal 11y ago
This looks absolutely wonderful. A few random thoughts: 1. I'm going to play with this as a way of documenting processes. Being able to answer "how do you develop features" or "what's best practice for X" with a usable process is great. 1 a) I'm particularly interested in using this for documenting how to perform experiments, and tracking experiments as they go. 2. I think this is amazing for progressively automating things. Split the task up into manageable, clear chunks to be performed by people. Each step can then (if it seems worth it) be replaced by code. 3. I don't think this is present at the moment (and it probably shouldn't be for a first release) but I'd like to see process hierarchies and importing others. I'd like to say 'Use github's "GITFLOW V2" process for managing feature additions and use "MY_COMPANY_QA V3" for reviewing features before full release'. Anyway, this is great, and I look forward to playing around with it. edit - Even without these, I can already see myself using it just for me. edit 2 - It'd be good to work on the getting started, so that there's just a "run this script" or "clone this repo". There's a lot of "download this file and copy it here" that feels like it should be scripted.
- dhaas 11y ago
  Thanks for the feedback! Your comment about embedding external QA workflows is super interesting. Right now Orchestra does support what we call 'Review Policies', which allow you to specify that a given workflow step be reviewed k times, or with probability X, or by specific expert Z. But it would be interesting to think of whole workflows for QA. Re: getting started, yeah, we're working on a script--turns out cross-platform support is hard!
  - IanCal 11y ago
    > Right now Orchestra does support what we call 'Review Policies', which allow you to specify that a given workflow step be reviewed k times, or with probability X, or by specific expert Z. That's handy, I can think of a bunch of things that fit well with that (particularly review a subset of x%) > Re: getting started, yeah, we're working on a script--turns out cross-platform support is hard! Hah, yeah it is! I've got a version setup now, but am struggling a bit to setup workflows & figure out quite what goes where, but I think I've found the right bit of documentation: http://orchestra.readthedocs.org/en/latest/example_use.html http://orchestra.readthedocs.org/en/latest/example_use.html As a suggestion, if you add a new example, try doing it step by step of what you need to add & do. It's more work, but I find it really helps show what someone will actually experience while creating / updating workflows.

121 points by marcua 11y ago

Hi, Hacker News! Today we’re excited to be making our first contribution back to the tech community by open sourcing Orchestra, a system for coordinating project teams of experts and machines. It can be used to reimagine and implement all sorts of workflows, from professional services (like legal, design, and data analysis) to new product experiences (like M from Facebook). Orchestra is based upon years of research into machine-mediated expert teams. We hope you’ll come up with new and creative ways of using it! At Unlimited Labs, we’re excited about building a brighter future of work and are currently building our first product on top of Orchestra. We’d love to answer any questions you have on here, or feel free to email the team at hello@unlimitedlabs.com!

marcua 11y ago

(Orchestra dev here) If you're curious how it can be used, Daniel Haas put together a wonderful example of how Orchestra could be used in a newsroom: http://orchestra.readthedocs.org/en/latest/example_use.html http://orchestra.readthedocs.org/en/latest/example_use.html Happy to take questions!

CoreSet 11y ago

This might seem a dumb question, but can you talk about how this improves on existing productivity workflows? Does it integrate with tasking systems like Jira or is this meant to be a complete a replacement? It seems high-level (and great) I'm just curious what implementing it "in the field" would look like.

dhaas 11y ago
Products like Jira help organize tasks within workflows, but don't automate any that organization. Orchestra takes care of things like "which reporter should I assign to a new story?" without requiring humans to make those decisions. Research from the Flash teams project that I mentioned in another comment (http://stanfordhci.github.io/flash-teams/index.html http://stanfordhci.github.io/flash-teams/index.html) has shown that this type of automation can decrease end-to-end workflow time by as much as 2x!

usman-m 11y ago

Great stuff! For now it seems like this leaves out the job of finding crowd workers. Could it be possible to do something like designing microtasks in MTurk and "stitching" the workflow together using Orchestra?

dhaas 11y ago
(Orchestra dev here too!) That's a great question. We're hoping that emerging online expert communities like Upwork or Dribbble will make it easier and easier to find experts suited to the work, but it's definitely not a solved problem (or tech recruiting would be so much easier!). Re: MTurk, there's nothing stopping you from stitching microtasks together with Orchestra, but I think that undersells Orchestra's strengths (e.g., collaboration among workers) a bit.

tilek 11y ago

This is very exciting!

zallarak 11y ago

This seems very cool - are there case studies of adopting this type of workflow for complex projects?

dhaas 11y ago
I don't know of any formal case studies, but Adam Marcus and myself (two devs on the project) worked on a large-scale workflow at GoDaddy where we automatically organized data extraction experts into hierarchies of review. That work actually got published at a Databases conference: see Adam's nice summary here: http://blog.marcua.net/post/128120315677 http://blog.marcua.net/post/128120315677 On the research side of things, the flash teams work by Daniela Retelny at Stanford (http://stanfordhci.github.io/flash-teams/ http://stanfordhci.github.io/flash-teams/) has done great proof-of-concept things like build an app in a day with distributed teams.

ggiacaglia 11y ago

This seems very cool!!!

johndoe7 11y ago

Are musical instruments provided?

dhaas 11y ago
They don't come in the box, but you're free to hire musicians for your workflows!

SamBam 11y ago

Does this tie into products like PivotalTracker or JIRA to make it useful for agile software development? It would be nice to be able to create tasks from stories in Pivotal, assign a reviewer to them, and, when a reviewer thumbs-up, have it automatically built into a staging server. Anything like that in the works?

dhaas 11y ago
We haven't built that integration yet, but it's totally possible! Really all it should take is a Machine Step that calls out from Python to, say, Pivotal's APIs (https://pypi.python.org/pypi/pivotaltracker https://pypi.python.org/pypi/pivotaltracker).

dharma1 11y ago

i think this looks potentially great - would definitely benefit from more real world examples of workflows

keredhaas 11y ago

This looks really exciting, and I can't wait to try it. I think it will be very helpful for my business.

IanCal 11y ago

This looks absolutely wonderful. A few random thoughts: 1. I'm going to play with this as a way of documenting processes. Being able to answer "how do you develop features" or "what's best practice for X" with a usable process is great. 1 a) I'm particularly interested in using this for documenting how to perform experiments, and tracking experiments as they go. 2. I think this is amazing for progressively automating things. Split the task up into manageable, clear chunks to be performed by people. Each step can then (if it seems worth it) be replaced by code. 3. I don't think this is present at the moment (and it probably shouldn't be for a first release) but I'd like to see process hierarchies and importing others. I'd like to say 'Use github's "GITFLOW V2" process for managing feature additions and use "MY_COMPANY_QA V3" for reviewing features before full release'. Anyway, this is great, and I look forward to playing around with it. edit - Even without these, I can already see myself using it just for me. edit 2 - It'd be good to work on the getting started, so that there's just a "run this script" or "clone this repo". There's a lot of "download this file and copy it here" that feels like it should be scripted.

dhaas 11y ago
Thanks for the feedback! Your comment about embedding external QA workflows is super interesting. Right now Orchestra does support what we call 'Review Policies', which allow you to specify that a given workflow step be reviewed k times, or with probability X, or by specific expert Z. But it would be interesting to think of whole workflows for QA. Re: getting started, yeah, we're working on a script--turns out cross-platform support is hard!
- IanCal 11y ago
  > Right now Orchestra does support what we call 'Review Policies', which allow you to specify that a given workflow step be reviewed k times, or with probability X, or by specific expert Z. That's handy, I can think of a bunch of things that fit well with that (particularly review a subset of x%) > Re: getting started, yeah, we're working on a script--turns out cross-platform support is hard! Hah, yeah it is! I've got a version setup now, but am struggling a bit to setup workflows & figure out quite what goes where, but I think I've found the right bit of documentation: http://orchestra.readthedocs.org/en/latest/example_use.html http://orchestra.readthedocs.org/en/latest/example_use.html As a suggestion, if you add a new example, try doing it step by step of what you need to add & do. It's more work, but I find it really helps show what someone will actually experience while creating / updating workflows.