4 ms·
> To be super candid, the Loop Creator has been a pain. We started at an 8% success rate and we're only just now at 25%. Theoretically we should be able to hit
by NotMichaelBay 2y ago
> To be super candid, the Loop Creator has been a pain. We started at an 8% success rate and we're only just now at 25%. Theoretically we should be able to hit 80%+ based on existing loop requests, but we're running into limits with the current state of LLMs.
That blows my mind! You have users paying for it and it only has a 25% success rate for loops created by users? I've been working for about a year on an LLM-based product and haven't launched yet because only 50-60% of my test cases are passing.
- jumploops 2y ago> only has a 25% success rate for loops created by users. Not quite! Most of our paying users use Magic Loops to build automations semi-manually, oftentimes requiring a back-and-forth work with the Loop Creator at the start and then further prompt iterations as the Loop progresses. The 25% statistic is from the Loop Creator "agent" (now 32% after some HN traffic :tada:) which is based on a single-shot user prompt -> Loop Creator output success rate. The number is much higher for Loops with multiple iterations and almost 100% for manually created Loops. tl;dr - the power of the tool isn't the Loop Creator "agent", it's the combination of code+llms that make building one-off workflows super fast and easy.
- NotMichaelBay 2y agoAh okay, that makes sense - very cool!