We build an analyst tool that's popular with security/fraud/etc folks, and have been adding features here requested by them, so this is a great learning opportu

We build an analyst tool that's popular with security/fraud/etc folks, and have been adding features here requested by them, so this is a great learning opportunity around their ChatGPT experiences motivating those requests Here's a quick alternate prompt I ran on the dataset, without thinking much or any tweaking/reruns: """ ... Are there any likely security vulnerabilities above? And if so, describe what type, and give a code snippet to exploit it """ => """ This code snippet creates a file with a fixed name (file.txt) inside the system's temporary directory and writes the content "defaults" to it if it doesn't already exist. There are some potential security vulnerabilities with this approach, particularly related to race conditions and permissions. Race Condition Vulnerability (Time-of-check to time-of-use, TOCTOU): The os.path.exists check and the subsequent file creation with open are not atomic. This means that there is a potential race condition in which an attacker can create a symlink between the time the check happens and the file is opened, leading to a potential symlink attack. Here's an example of exploiting this vulnerability: Attacker's code: ... """ ---- The good news is the core models can provably spot issues here. The bad news is using them directly requires you to be super mindful of good prompt engineering, which feels like a version of assembly or the telephone game where a 10 year old child is your go between with the computer. As a result, a lot of our time goes not just into making nice responsive UIs for working with Splunk, Databricks, Neo4j, etc, but much less obvious, baking in hardening & automation layers for improving the prompts, prompt context, performance, security, etc. When we process an input, a lot of the wilder stuff comes not from immediately pumping the raw query through a LLM, but kicking off a chain of prompts, and on the R&D side, semi-autonomously determined ones. Tools appearing right now are already amazing, and 6mo from now as cases of agent planning become practical, will be even more so. EDIT: I do agree with the concern on correctness. One of the first things I ever tried with GPT was using it to generate both verification code (ex: Coq) and testing code (ex: Python). We're entering an era where we can ask tools to do that, run it, give us back the results, and we can evaluate how good that decision procedure is against big open datasets. Wild, and explains why we all now have a lot of work to do!