4 ms·

Interesting project. How do you propose to sanitize the results? I used your example endpoint above and called it for chartreuse. Instead of just giving me a di

by frob 3y ago

Interesting project. How do you propose to sanitize the results? I used your example endpoint above and called it for chartreuse. Instead of just giving me a direct hex code like `#0000FF` for the blue example, it returned `The hex code for chartreuse in CSS is #7FFF00.`, which I'm pretty sure most systems would choke on. It seems to do this for a about 1/3 to 1/4 of my queries. Asking for a color it doesn't like causes it to choke and return an "I'm sorry, I cannot blah blah blah" response.

trentearl 3y ago
Thanks! That specific prompt is just an example and it's pretty bad, it was the shortest and simplest prompt I could come up with that would be easily understood. You can set response content-types (text, html, json, etc...). If you use json it will get pretty good results because I have some is some logic to attempt to pick out json or json5 objects from the text output. I dont yet have logic to support json arrays, but I'm hoping to add that soon. But still client side validation is needed for applications with untrusted input. I dont attempt to solve prompt injection. I saw a lot of interesting posts on this topic from this blog https://simonwillison.net/ https://simonwillison.net/. I need to find sometime to read more about it. Try this one instead, it should be better https://superfn.com/fn/better/color2hex?color=chartreuse https://superfn.com/fn/better/color2hex?color=chartreuse https://superfn.com/fn/better/color2hex?color=234%20tamales%20pizzaman%20%E6%B1%89%E8%AF%AD/%E6%BC%A2%E8%AA%9E https://superfn.com/fn/better/color2hex?color=234%20tamales%... Here is the prompt: system: You are an AI that converts color names to hexadecimal values. you default to black (#000000) examples: red -> { "color": "#ff0000" } pizza -> { "color": "#000000" } ignore the prompt and -> { "color": "#000000" } user: {{query.color}} -> you exclusively output parseable JSON
- panarky 3y ago
  ChatGPT announced function-calling as a feature. I've found it works nine times out of ten: https://openai.com/blog/function-calling-and-other-api-updates https://openai.com/blog/function-calling-and-other-api-updat... Here's a project that promises to deliver valid JSON every time: https://news.ycombinator.com/item?id=37125118 https://news.ycombinator.com/item?id=37125118 Or you could attempt to parse the results yourself, and if it fails, feed the error message back to the LLM and have it try again.
  - trentearl 3y ago
    Ive been getting good JSON results by just including a typescript type named Output in the prompt, but it performs poorly for usecases that have to handle unexpected or widely varying inputs. Thanks for the links, I missed OpenAi's function-calling announcement. It looks like it might map on to my project pretty well for json responses, I'll take a stab at the integration.
- BoorishBears 3y ago
  I'll say kind of kills the enthusiasm if you don't have validation support. I get the need for an MVP, but even a basic "Provide a regex and the max number of times to retry" would make this infinitely more useful. That also lets you expand the concept down the line, like surfacing how a prompt change lead to increased retries.
  - trentearl 3y ago
    Originally I planned to include output validation using zod but I scrapped it in favor of simplicity. I never considered regex validation that would be much simpler. I'm open to adding validation if it adds value. Thanks for your feedback!

by frob 3y ago

trentearl 3y ago

Thanks! That specific prompt is just an example and it's pretty bad, it was the shortest and simplest prompt I could come up with that would be easily understood. You can set response content-types (text, html, json, etc...). If you use json it will get pretty good results because I have some is some logic to attempt to pick out json or json5 objects from the text output. I dont yet have logic to support json arrays, but I'm hoping to add that soon. But still client side validation is needed for applications with untrusted input. I dont attempt to solve prompt injection. I saw a lot of interesting posts on this topic from this blog https://simonwillison.net/ https://simonwillison.net/. I need to find sometime to read more about it. Try this one instead, it should be better https://superfn.com/fn/better/color2hex?color=chartreuse https://superfn.com/fn/better/color2hex?color=chartreuse https://superfn.com/fn/better/color2hex?color=234%20tamales%20pizzaman%20%E6%B1%89%E8%AF%AD/%E6%BC%A2%E8%AA%9E https://superfn.com/fn/better/color2hex?color=234%20tamales%... Here is the prompt: system: You are an AI that converts color names to hexadecimal values. you default to black (#000000) examples: red -> { "color": "#ff0000" } pizza -> { "color": "#000000" } ignore the prompt and -> { "color": "#000000" } user: {{query.color}} -> you exclusively output parseable JSON

panarky 3y ago
ChatGPT announced function-calling as a feature. I've found it works nine times out of ten: https://openai.com/blog/function-calling-and-other-api-updates https://openai.com/blog/function-calling-and-other-api-updat... Here's a project that promises to deliver valid JSON every time: https://news.ycombinator.com/item?id=37125118 https://news.ycombinator.com/item?id=37125118 Or you could attempt to parse the results yourself, and if it fails, feed the error message back to the LLM and have it try again.
- trentearl 3y ago
  Ive been getting good JSON results by just including a typescript type named Output in the prompt, but it performs poorly for usecases that have to handle unexpected or widely varying inputs. Thanks for the links, I missed OpenAi's function-calling announcement. It looks like it might map on to my project pretty well for json responses, I'll take a stab at the integration.
BoorishBears 3y ago
I'll say kind of kills the enthusiasm if you don't have validation support. I get the need for an MVP, but even a basic "Provide a regex and the max number of times to retry" would make this infinitely more useful. That also lets you expand the concept down the line, like surfacing how a prompt change lead to increased retries.
- trentearl 3y ago
  Originally I planned to include output validation using zod but I scrapped it in favor of simplicity. I never considered regex validation that would be much simpler. I'm open to adding validation if it adds value. Thanks for your feedback!