3 ms·

Simply holding temp to 0 or making the selection deterministic isn't an adequate solution unless the process is always run with the same set of inputs (at which

by lsy 3y ago

Simply holding temp to 0 or making the selection deterministic isn't an adequate solution unless the process is always run with the same set of inputs (at which point why not run the model once on all inputs and create a map?). Ultimately with LLMs it's not possible (or desirable) to keep inputs separate from the rest of the prompt, so changing "give me the top X" to "give me the top Y" has the potential for a wildly different result. With traditional code we can achieve reliability because we sanitize and apply bounds to inputs, which then transit through the logic in a way we can reason about. The strength and weakness of an LLM is that it mashes up the input with the rest of its data and prompts, meaning we cannot predict the output for an arbitrary set of inputs.

dwallin 3y ago
Correct me if I misunderstand, but your point is that even if the textual content can be made deterministic the (form/shape/type?) of the output is not deterministic? If you are expecting a specific format you can just check whether the LLM outputs the correct format, and return null or an error in that case. Given the input is arbitrary text, and assuming a non-trivial transformation, traditional code would need a way of handling failure cases anyways. This means your function either way would look something like: Item from Universal Set -> Value of Type | Null You need to reject the entire set of invalid inputs. However the sets of all valid and invalid inputs are often both infinite themselves and it’s also not guaranteed that this set is actually computable. Alternatively in these cases (and most commonly) you construct a calculable subset of inputs and reject the rest. However this means you are still rejecting an infinite number of valid inputs. On the other hand an LLM always returns a value. Your job as a programmer using an LLM is instead to validate and narrow down the result type as much as possible. However the way they work means that for many cases they can output a valid output for a much wider range of inputs than you could do with traditional code in a reasonable amount of code complexity. For many tasks this is transformational.

by lsy 3y ago

dwallin 3y ago

Correct me if I misunderstand, but your point is that even if the textual content can be made deterministic the (form/shape/type?) of the output is not deterministic? If you are expecting a specific format you can just check whether the LLM outputs the correct format, and return null or an error in that case. Given the input is arbitrary text, and assuming a non-trivial transformation, traditional code would need a way of handling failure cases anyways. This means your function either way would look something like: Item from Universal Set -> Value of Type | Null You need to reject the entire set of invalid inputs. However the sets of all valid and invalid inputs are often both infinite themselves and it’s also not guaranteed that this set is actually computable. Alternatively in these cases (and most commonly) you construct a calculable subset of inputs and reject the rest. However this means you are still rejecting an infinite number of valid inputs. On the other hand an LLM always returns a value. Your job as a programmer using an LLM is instead to validate and narrow down the result type as much as possible. However the way they work means that for many cases they can output a valid output for a much wider range of inputs than you could do with traditional code in a reasonable amount of code complexity. For many tasks this is transformational.