Yeah, there is always the high-level view of these things and the low-level. The way LLMs make stuff up is oft-remarked upon and is unclear whether it's a deal

Yeah, there is always the high-level view of these things and the low-level. The way LLMs make stuff up is oft-remarked upon and is unclear whether it's a dealbreaker or not but it might be for any/all of this to come to pass. Here's an Werner/AWS-specific LLM anecdote along those lines: I went onto AWS's LLM Q for the first time today, a little chat icon nicely/helpfully linked from their documentation pages (when logged into my AWS account) and asked it a question which I was unable to find an answer in their docs or via Google/Bing/StackOverflow after 5-10 minutes of searching. What Redshift SQL query can I write to tell whether the instance I am connected to is a Redshift Serverless or a Redshift Provisioned cluster? (I have migrated Redshift clusters in Dev/Nonprod to Serverless to reduce cost, but featureset is not identical so some of our app code is sensitive to the differences and needs to detect which type of environment is present before executing. The release number, but not the instance type, can be found in Redshift SELECT VERSION();.) AWS Q LLM replies: To check if an Amazon Redshift cluster is provisioned or serverless, you can run the following SQL query: SELECT cluster_type FROM svv_cluster_info; The cluster_type will return either "provisioned" or "serverless". You can only connect to and query Serverless clusters using the Amazon Redshift Query Editor v2. Provisioned clusters can be queried using either Query Editor v1 or v2. [...+more text on the differences and a bulleted list of 3 weblinks/sources provided...] Wow! Awesome! This answered my question perfectly! The response and code is formatted beautifully just like real documentation. There are a lot of SVV_ system views and I guess I just missed this one. I am at this point very impressed. I go to actually try the above query on my serverless and provisioned clusters... Serverless: SQL Error [42P01]: ERROR: relation "svv_cluster_info" does not exist Provisioned: SQL Error [42P01]: ERROR: relation "svv_cluster_info" does not exist Aww, man! Another attempt answered similarly affirmatively, this time mentioning an imaginary "SVV_CLUSTER" view (which also didn't work.) So I don't think this is just some "unreleased future feature" leaked to the LLM via its training set of internal documentation (wouldn't that be nice! err, kinda...), since even the SVV_ view name isn't the same when you ask a similar question at different times. Even as a chatbot, the most obvious use case, this is is not really meeting the mark. I know it's brand new and rushed out of the door for re:Invent two weeks ago and I know it'll get better... but it was a bit hard to read Werner's article a few hours later and take it seriously. My first Bayesian prior is now a "err, no".