4 ms·
Well the website seems to be down so I can't actually see any information about what LLM you are using, but I seriously hope you are not just sending the data t
by nerdjon 2y ago
Well the website seems to be down so I can't actually see any information about what LLM you are using, but I seriously hope you are not just sending the data to OpenAI API or something like that and are forcing the use of a private (ideally self hosted) service.
I would not want any data about my infrastructure sent to a public LLM, regardless of how sanitized things are.
Otherwise, on paper it seems cool. But I worry about getting complicit with this tech. It is going to fail, that is just the reality. We know LLM's will hallucinate and there is not much we can do about it, it is the nature of the tech.
SO it might work most of the time, but when it doesn't and you're bashing your head against the wall trying to figure out what is broken. This system is telling you that all of these things are fine, but one of them actually isn't. But it worked enough times that you trust it, so you don't bother double checking.
That is before we even talk about having this thing running code for automatic remediation, which I hope no one seriously considers ever doing that.
- wilson090 2y agoHmm we're not seeing any issues with the website on our end -- tryparity.com is down for you? The data security point with LLMs is definitely relevant. There's a broader conversation ongoing right now about how teams will securely use LLMs, but from our conversations so far teams have been willing to adopt the tech. We've been working with startups up to this point, so we'll likely need to offer support self-hosted LLMs if we were to support enterprise or bring-your-own-keys for larger startups. The hallucination point is interesting. I think a lot of products will need to solve this problem of having so much trust with the user that they'll blindly follow the outputs, but occasionally failing due to hallucination. Our approach has been to 1) only focus on investigation/root cause and 2) make sure it's easy to audit the results by sharing all of the results + supporting evidence