3 ms·

I bring this up often at work. There is more ROI in assuming models will continue to improve, and planning/engineering with that future in mind, rather than usi

by extr 2y ago

I bring this up often at work. There is more ROI in assuming models will continue to improve, and planning/engineering with that future in mind, rather than using a worse model and spending a lot of dev time shoring up it's weaknesses, prompt engineering, etc. The best models today will be cheaper tomorrow. The worst models today will literally cease to exist. You want to lean into this - have the AI handle as much as it possibly can. Eg: We were using Flash 1.5 for awhile. Spent a lot of time prompt engineering to get it to do exactly what we wanted and be more reliable. Probably should have just done multi-shot and said "take best of 3", because as soon as Flash 2.0 came out, all the problems evaporated.

ankit219 2y ago
Thats the core of the argument. We are switching from a 100% deterministic and controlled worldview (in software terms) to a scenario where it's probabilistic, and we haven't updated ourselves accordingly. Best of n (with parallelization) is probably the simplest fix instead of such rigorous prompt engineering. Still many teams do want a deterministic output and spend a lot of time on prompts (as opposed to evals to choose the best output).
jncfhnb 2y ago
Honestly I feel the takeaways are the opposite. There’s no point in building something non functional now simply because it will be replaceable with something functional later. You should either do it without AI or not do it at all. You’re not actually adding value with a placeholder for “future AI”.
- typewithrhythm 2y ago
  Its more that if you don't know for sure if it's possible; and usually you don't. Then adding your expertise onto an ai system is never going to pay off compared to building out the ai compute infrastructure and training data. This has the best chance of being functional in the long term, in the face of uncertainty. If you already know it can work, then you can improve with specific expertise, but it's a fixed solution at that point.
- extr 2y ago
  Eh, so in reality there are a lot of AI products people are trying to build and it's very unclear at the outset "if it's possible", where "possible" is a business question that includes factors like: - How hard is the task? Can it be completed with cheaper/faster models or does it require heavyweight SOTA tier models? - What's your cost envelope for AI compute? - How are you going to test/refine the exact prompt and examples you give the AI? - How much scaffolding (aka, dev time = $$$) do you need to set up to integrate the AI with other systems? - Is the result reliable enough to productize and show to users? What you realize when designing these systems is there is a sliding scale where the more scaffolding and domain expertise you put into the system as a whole, the less you need to rely on the AI, but the more expensive it is in terms of man-hours it is to develop and maintain. It looks more and more just like a traditional system. And vice versa, perhaps with the most powerful SOTA models you can just dump 20K tokens of context and get an answer that is highly reliable and accurate with almost no extra work on your end (but costs more to run). It's very individualized and task-dependent. But we do know from recent history, you can generally assume models are going to get faster/smarter/cheaper pretty quickly. So you try to figure out how close to the latter scenario you can get away with for now, knowing that in 6 months the equation could have completely changed in favor of "let the AI do most of the work". As an addendum, I think it's completely crazy right now to be in the business of training your own models unless you have HIGHLY specialized needs or like to light money on fire. You are never going to achieve the performance/$ of the big AI labs, and they/their investors are doing all your R&D for FREE. It's like if Ford was releasing a new car every 6 months made out of ever more efficient and stronger carbon nanotubes or whatever, because the carbon nanotube companies were all competing for market share and wanted to win the "carbon nanotube race". It's crazy, never seen anything like it.

by extr 2y ago

ankit219 2y ago

Thats the core of the argument. We are switching from a 100% deterministic and controlled worldview (in software terms) to a scenario where it's probabilistic, and we haven't updated ourselves accordingly. Best of n (with parallelization) is probably the simplest fix instead of such rigorous prompt engineering. Still many teams do want a deterministic output and spend a lot of time on prompts (as opposed to evals to choose the best output).

jncfhnb 2y ago

Honestly I feel the takeaways are the opposite. There’s no point in building something non functional now simply because it will be replaceable with something functional later. You should either do it without AI or not do it at all. You’re not actually adding value with a placeholder for “future AI”.

typewithrhythm 2y ago
Its more that if you don't know for sure if it's possible; and usually you don't. Then adding your expertise onto an ai system is never going to pay off compared to building out the ai compute infrastructure and training data. This has the best chance of being functional in the long term, in the face of uncertainty. If you already know it can work, then you can improve with specific expertise, but it's a fixed solution at that point.
extr 2y ago
Eh, so in reality there are a lot of AI products people are trying to build and it's very unclear at the outset "if it's possible", where "possible" is a business question that includes factors like: - How hard is the task? Can it be completed with cheaper/faster models or does it require heavyweight SOTA tier models? - What's your cost envelope for AI compute? - How are you going to test/refine the exact prompt and examples you give the AI? - How much scaffolding (aka, dev time = $$$) do you need to set up to integrate the AI with other systems? - Is the result reliable enough to productize and show to users? What you realize when designing these systems is there is a sliding scale where the more scaffolding and domain expertise you put into the system as a whole, the less you need to rely on the AI, but the more expensive it is in terms of man-hours it is to develop and maintain. It looks more and more just like a traditional system. And vice versa, perhaps with the most powerful SOTA models you can just dump 20K tokens of context and get an answer that is highly reliable and accurate with almost no extra work on your end (but costs more to run). It's very individualized and task-dependent. But we do know from recent history, you can generally assume models are going to get faster/smarter/cheaper pretty quickly. So you try to figure out how close to the latter scenario you can get away with for now, knowing that in 6 months the equation could have completely changed in favor of "let the AI do most of the work". As an addendum, I think it's completely crazy right now to be in the business of training your own models unless you have HIGHLY specialized needs or like to light money on fire. You are never going to achieve the performance/$ of the big AI labs, and they/their investors are doing all your R&D for FREE. It's like if Ford was releasing a new car every 6 months made out of ever more efficient and stronger carbon nanotubes or whatever, because the carbon nanotube companies were all competing for market share and wanted to win the "carbon nanotube race". It's crazy, never seen anything like it.