I work at a company which performs analytics on medical data. To test our software, and to allow our clients to test the kinds of analyses they perform, we hav

I work at a company which performs analytics on medical data. To test our software, and to allow our clients to test the kinds of analyses they perform, we have "synthetic" datasets available to our software suite. These datasets statistically look like real medical data, but they're all fake. This means we can use them and get realistic results provided we're not concerned about accuracy (testing a new feature, for example), and we don't have to worry about things like HIPAA compliance because no actual medical data is being used. To me that's where things like LLMs come in. They produce statistically plausible, fake data. They're not reliable as a ground source of truth, even for summarizing, because accuracy isn't their job. Statistical plausibility with respect to their model is. And note that unless I'm running the model on my own, on a PC much beefier than any I actually own, I control neither the corpus used to train the model nor what counts as "statistical plausibility" -- someone at Microsoft does that. Which means I can't trust ChatGPT's output to produce an accurate summary or handle information I provide it in a responsible way. It may well decide to fool me with garbage because that's what it's designed to do: produce convincing output, not accurate output. Where it might come in handy is providing a starting point for like a fictional story, or an essay or something. Fake data can be profoundly useful in certain contexts, but it's still fake.