6 ms·

It's refreshing to read something about ML and inference and have it not be anything related to a transformer architecture sending up fruit growing from a huge

by sfink 3y ago

It's refreshing to read something about ML and inference and have it not be anything related to a transformer architecture sending up fruit growing from a huge heap of rotten, unknown, mostly irrelevant data. With traditional ML, it's useful to talk about the sources of bias and error, and even measure some of them. You can do things that improve them without starting over on everything else. With LLMs, it's more like you buy a large pancake machine that you dump all of your compost into (and you suspect the installers might have hooked up to your sewage line as input too). It triples your electricity bill, it makes bizarre screeching noises as it runs, you haven't seen your cat in a week, but at the end out come some damn fine pancakes. I apologize. I'm talking about the thing that I was saying was a relief to be not talking about.

nikhilsimha 3y ago
I agree with you - about the sentiment around the GenAI megaphone. FWIW, Chronon does serve context within prompts to personalize LLM responses. It is also used to time-travel new prompts for evaluation.
- cactusplant7374 3y ago
  > time-travel new prompts for evaluation What does this mean?
  - nikhilsimha 3y ago
    Imagine you are building a customer support bot for a food delivery app. The user might say - I need a refund. The bot needs to know contextual information - order details, delivery tracking details etc. Now you have written a prompt template that needs to be rendered with contextual information. This rendered prompt is what the model will use to decide whether to issue a refund or not. Before you deploy this prompt to prod, you want to evaluate its performance - instances where it correctly decided to issue or decline a refund. To evaluate, you can “replay” historical refund requests. The issue is that the information in the context changes with time. You want to instead simulate the value of the context at a historical point in time - or time-travel.
    - jamesblonde 3y ago
      Time-travel evals, nice.
    - jamesblonde 3y ago
      Are you using function calling for the context info?
    - uoaei 3y ago
      In what world is it appropriate or even legal to decide on refunds via LLM? Can you give an example that's not ripe for abuse? This really doesn't sell LLMs as anything useful except insulation from the consequences of bad decisions.
      - nikhilsimha 3y ago
        "Imagine" is the operative word :-)
      - throwaway2037 3y ago
        Don't think of LLM as completely replacing the support agent here; rather augmenting. A lot of customer service is setting/finding context: customer name, account, order, item, etc. If an LLM chatbot can do all of that, then handoff to a human support agent, there is real cost savings to be had, without reducing the quality of service.
        uoaei 3y ago
        I'd love for others to think that way. I am a very vocal (in my own bubble) advocate for human-in-the-loop ML.
      - amenhotep 3y ago
        Have you requested a refund off Amazon lately? They have an automated system where, iirc, a wizard will ask you a few questions and then process it, presumably inspecting your customer history and so on. If the system thinks your request looks genuine and it's within whatever parameters they've set, it'll accept instantly, refund you, sometimes without even asking you to send the item back. If it's less sure, it will pass the request on to a human agent to be dealt with like it would have been in the Before Times. I can see no reason why it would be illegal or inappropriate to use an LLM as part of the initial flow there. In fact I see no reason why it would be illegal for Amazon to simply flip a coin to decide whether to immediately accept your refund. (Appropriateness is another matter!) I guess you're assuming the LLM would be the only point of contact with no recourse if it rejects you? Which strikes me as very pessimistic, unless you live in a very poorly regulated country.
haolez 3y ago
What do you think of the approach in DSPy[0]? It seems to give a more traditional ML feel to LLM optimization. [0] https://dspy.ai/ https://dspy.ai/

by sfink 3y ago

nikhilsimha 3y ago

I agree with you - about the sentiment around the GenAI megaphone. FWIW, Chronon does serve context within prompts to personalize LLM responses. It is also used to time-travel new prompts for evaluation.

cactusplant7374 3y ago
> time-travel new prompts for evaluation What does this mean?
- nikhilsimha 3y ago
  Imagine you are building a customer support bot for a food delivery app. The user might say - I need a refund. The bot needs to know contextual information - order details, delivery tracking details etc. Now you have written a prompt template that needs to be rendered with contextual information. This rendered prompt is what the model will use to decide whether to issue a refund or not. Before you deploy this prompt to prod, you want to evaluate its performance - instances where it correctly decided to issue or decline a refund. To evaluate, you can “replay” historical refund requests. The issue is that the information in the context changes with time. You want to instead simulate the value of the context at a historical point in time - or time-travel.
  - jamesblonde 3y ago
    Time-travel evals, nice.
  - jamesblonde 3y ago
    Are you using function calling for the context info?
  - uoaei 3y ago
    In what world is it appropriate or even legal to decide on refunds via LLM? Can you give an example that's not ripe for abuse? This really doesn't sell LLMs as anything useful except insulation from the consequences of bad decisions.
    - nikhilsimha 3y ago
      "Imagine" is the operative word :-)
    - throwaway2037 3y ago
      Don't think of LLM as completely replacing the support agent here; rather augmenting. A lot of customer service is setting/finding context: customer name, account, order, item, etc. If an LLM chatbot can do all of that, then handoff to a human support agent, there is real cost savings to be had, without reducing the quality of service.
      - uoaei 3y ago
        I'd love for others to think that way. I am a very vocal (in my own bubble) advocate for human-in-the-loop ML.
    - amenhotep 3y ago
      Have you requested a refund off Amazon lately? They have an automated system where, iirc, a wizard will ask you a few questions and then process it, presumably inspecting your customer history and so on. If the system thinks your request looks genuine and it's within whatever parameters they've set, it'll accept instantly, refund you, sometimes without even asking you to send the item back. If it's less sure, it will pass the request on to a human agent to be dealt with like it would have been in the Before Times. I can see no reason why it would be illegal or inappropriate to use an LLM as part of the initial flow there. In fact I see no reason why it would be illegal for Amazon to simply flip a coin to decide whether to immediately accept your refund. (Appropriateness is another matter!) I guess you're assuming the LLM would be the only point of contact with no recourse if it rejects you? Which strikes me as very pessimistic, unless you live in a very poorly regulated country.

haolez 3y ago

What do you think of the approach in DSPy[0]? It seems to give a more traditional ML feel to LLM optimization. [0] https://dspy.ai/ https://dspy.ai/