4 ms·

Very interesting, thank you for making this available! At OpenAdapt (https://github.com/OpenAdaptAI/OpenAdapt https://github.com/OpenAdaptAI/OpenAdapt) we are

Very interesting, thank you for making this available! At OpenAdapt (https://github.com/OpenAdaptAI/OpenAdapt https://github.com/OpenAdaptAI/OpenAdapt) we are looking into using pm4py (https://github.com/pm4py https://github.com/pm4py) to extract a process graph from a recording of user actions. I will look into this more closely. In the meantime, could the authors share their perspective on whether Memary could be useful here?

oulipo 2y ago
Very cool project! I think one of the main way (a bit orthogonal to what you do now) to adapt to GUI / CLI would be to develop an open-source version of something like Aqua Voice https://withaqua.com/ https://withaqua.com/ Perhaps it could make sense to add this to your effort?
- abrichr 2y ago
  Thanks! OpenAdapt already supports audio recording during demonstration (https://github.com/OpenAdaptAI/OpenAdapt/pull/346 https://github.com/OpenAdaptAI/OpenAdapt/pull/346). Perhaps I misunderstood — can you please clarify your suggestion?
  - oulipo 2y ago
    It's a kind of text input which mixes text and edition instructions, look at the demo
dbish 2y ago
What’s the goal of creating a graph from the actions? Do you have any related papers that talk about that? We also capture and learn from actions but haven’t found value in adding structure beyond representing them semantically in a list with the context around them of what happened.
- abrichr 2y ago
  The goal is to have a deterministic representation of a process that can be traversed in order to accomplish a task. There's a lot of literature around process mining, e.g.: - https://en.wikipedia.org/wiki/Process_mining https://en.wikipedia.org/wiki/Process_mining - https://www.sciencedirect.com/science/article/pii/S2665963823000933 https://www.sciencedirect.com/science/article/pii/S266596382... - https://arxiv.org/abs/2404.06035 https://arxiv.org/abs/2404.06035
  - dbish 2y ago
    Awesome, thanks. Yes we are also on the process mining and RPA space and use image reco + ocr + click tracking for part of it. My (poorly worded probably) question was why do knolwdge graphs matter for you, since traditional rpa doesn’t use them for all the cases I’ve seen at least, unless I misunderstood what you’re saying here about trying out graphs. I’ll read the LLM RPA paper you have linked here too, maybe that explains the use of graphs, and I haven’t read this one so thank you. We basically used the same technique some of the UIPath folks have used which is representing everything as a sequence of actions, which "branching" being represented by different linear sequences that the model can ingest and make decisions over which sequence to follow, which is kind of a graph i guess but not how we represent it.
    - abrichr 2y ago
      > traditional rpa doesn’t use them for all the cases I’ve seen That's because traditional RPA relies on humans to create the automations. Our goal is to create the automation automatically by observing human demonstrations.

oulipo 2y ago

Very cool project! I think one of the main way (a bit orthogonal to what you do now) to adapt to GUI / CLI would be to develop an open-source version of something like Aqua Voice https://withaqua.com/ https://withaqua.com/ Perhaps it could make sense to add this to your effort?

abrichr 2y ago
Thanks! OpenAdapt already supports audio recording during demonstration (https://github.com/OpenAdaptAI/OpenAdapt/pull/346 https://github.com/OpenAdaptAI/OpenAdapt/pull/346). Perhaps I misunderstood — can you please clarify your suggestion?
- oulipo 2y ago
  It's a kind of text input which mixes text and edition instructions, look at the demo

dbish 2y ago

What’s the goal of creating a graph from the actions? Do you have any related papers that talk about that? We also capture and learn from actions but haven’t found value in adding structure beyond representing them semantically in a list with the context around them of what happened.

abrichr 2y ago
The goal is to have a deterministic representation of a process that can be traversed in order to accomplish a task. There's a lot of literature around process mining, e.g.: - https://en.wikipedia.org/wiki/Process_mining https://en.wikipedia.org/wiki/Process_mining - https://www.sciencedirect.com/science/article/pii/S2665963823000933 https://www.sciencedirect.com/science/article/pii/S266596382... - https://arxiv.org/abs/2404.06035 https://arxiv.org/abs/2404.06035
- dbish 2y ago
  Awesome, thanks. Yes we are also on the process mining and RPA space and use image reco + ocr + click tracking for part of it. My (poorly worded probably) question was why do knolwdge graphs matter for you, since traditional rpa doesn’t use them for all the cases I’ve seen at least, unless I misunderstood what you’re saying here about trying out graphs. I’ll read the LLM RPA paper you have linked here too, maybe that explains the use of graphs, and I haven’t read this one so thank you. We basically used the same technique some of the UIPath folks have used which is representing everything as a sequence of actions, which "branching" being represented by different linear sequences that the model can ingest and make decisions over which sequence to follow, which is kind of a graph i guess but not how we represent it.
  - abrichr 2y ago
    > traditional rpa doesn’t use them for all the cases I’ve seen That's because traditional RPA relies on humans to create the automations. Our goal is to create the automation automatically by observing human demonstrations.