3 ms·

New OpenAI Feature: Predicted Outputs

• user_james92 2y ago
  [dead]
• AIFounder 2y ago
  [dead]
• cbhl 2y ago
  If you use the Cursor IDE: the folks that wrote it talked about their use of speculative decoding to make "Apply" faster on the Lex Friedman podcast last month. Here it is on YouTube, although you can also find it on Spotify and other podcast platforms: https://youtu.be/oFfVt3S51T4?t=1206 https://youtu.be/oFfVt3S51T4?t=1206
  • afro88 2y ago
    For those that prefer text: it seems they use a weaker but faster model for the "predicted output" / speculation. Pretty smart.
  • pandada8 2y ago
    https://fireworks.ai/blog/cursor https://fireworks.ai/blog/cursor Fireworks AI has a blog about it.
• deleted 2y ago
  [deleted]
• 768DataSeeker 2y ago
  [flagged]
• creativenolo 2y ago
  I found the OpenAI page to be more interesting https://platform.openai.com/docs/guides/latency-optimization#use-predicted-outputs https://platform.openai.com/docs/guides/latency-optimization...
  • msp26 2y ago
    It's incredibly well written. I can see this being very helpful for newcomers. As for the Predicted Outputs feature, it looks incredibly useful in a few of my pipelines. Can't wait to test it out.
• nunez 2y ago
  This is like the likely() and unlikely() macros in the Linux kernel! Huge speedup if you're right; small penalty if you're not.
  • digdugdirk 2y ago
    Any recommendations for high level overview/learning resources about this? It seems interesting, but like most Linux internals, things get real technical, real quick.