4 ms·

Great question. I guess for me as a technologist, I’m coming at it from the other side and asking what failure rate is possible, and can that rate be reduced wi

Great question. I guess for me as a technologist, I’m coming at it from the other side and asking what failure rate is possible, and can that rate be reduced with further prompt fiddling and other machinations? Then I will try to gauge whether that rate is acceptable to users. I guess since it’s all so new, people wouldn’t know what would be useful or usable until they tried it. So we are looking for users to participate and help us find if there’s a viable product involving GPT-3 summaries.

ativzzz 6y ago
Makes sense. For our use case, having "incorrect" gpt output isn't a viable option so unfortunately it looks like the tech isn't there quite yet as we would need to manually verify the output, or we would need access to the model to train it on our own data, but not sure if that would improve the results.
- stochastimus 6y ago
  Here’s a question for you - for a typical L2 support person or front line MSP NOC monitor, or junior SRE, or similar - what do you think would be the error rate? I wonder if there isn’t sort of a front-end role here that can be subsumed at least in part by automatic summarization, for the purpose of streamlining the escalation and routing process overall, even though errors would occur - since I imagine they already do. What do you think?
  - ativzzz 6y ago
    I've only worked at (very) small companies so I've never had to think about different tiers of support. The automatic summarization may be useful for better determining which specialist to route the request to, but in any case a wrong classification will get picked up by a human regardless and then re-routed appropriately. I imagine the error rate is already pretty high, so anything to automate/reduce that can go a long way. I work in education tech where our product goes directly to students/teachers, so tolerance for "incorrect" answers generated by AI is much lower, or nonexistent.

ativzzz 6y ago

Makes sense. For our use case, having "incorrect" gpt output isn't a viable option so unfortunately it looks like the tech isn't there quite yet as we would need to manually verify the output, or we would need access to the model to train it on our own data, but not sure if that would improve the results.

stochastimus 6y ago
Here’s a question for you - for a typical L2 support person or front line MSP NOC monitor, or junior SRE, or similar - what do you think would be the error rate? I wonder if there isn’t sort of a front-end role here that can be subsumed at least in part by automatic summarization, for the purpose of streamlining the escalation and routing process overall, even though errors would occur - since I imagine they already do. What do you think?
- ativzzz 6y ago
  I've only worked at (very) small companies so I've never had to think about different tiers of support. The automatic summarization may be useful for better determining which specialist to route the request to, but in any case a wrong classification will get picked up by a human regardless and then re-routed appropriately. I imagine the error rate is already pretty high, so anything to automate/reduce that can go a long way. I work in education tech where our product goes directly to students/teachers, so tolerance for "incorrect" answers generated by AI is much lower, or nonexistent.