4 ms·

I think taking a slow path with AI around financial services is probably wise. Unfortunately I don’t think other countries and multinationals will take the sam

I think taking a slow path with AI around financial services is probably wise. Unfortunately I don’t think other countries and multinationals will take the same approach. So how do we avoid another arms race? This seems like a good public position, but ignoring AI isn’t the right private position if you care about your financial system.

threeseed 2y ago
Nobody is ignoring AI. There has been unprecedented budget spend allocated for AI in the enterprise. The results so far have been poor where apart from customer service there hasn't been any game changing use cases to justify the hype. There just isn't that much you can do with an inauditable black box that is consistently wrong 5-10% of the time and depends on high quality input data. And most enterprises have been doing ML/DS for years and so have already have tackled low hanging fruit using NLP etc.
- baxtr 2y ago
  Even for customer service I’m not aware of good use cases. But probably I’m ignorant. Can anyone share examples? I’m mean well working proven ones, not marketing BS.
  - raffraffraff 2y ago
    I would guess that by fine tuning an LLM on product manuals, installation guides, FAQs and vetted and customer support cases, one could create a competent support chat bot. Using RAG you could provide it with the output of your real-time status page for the product, and use a prompt that had the ability to forward issues to a human if the customer seemed unhappy. I'm not sure if there's a concrete example of this in reality, but why wouldn't it work, greed and incompetence aside?
    - portaouflop 2y ago
      At work we use a chatbot trained on our docs, it’s pretty good and sees lots of usage. Of course people can also just read the docs, but many prefer the bot.
      - aenis 2y ago
        Same here, but adoptoion is a challenge. People tend to stop using it after the first instance of a bullshit answer - they lose trust completely. Which is, seemingly, inevitable.
        falcor84 2y ago
        Why is it inevitable? I think that the proper way to model it is as a multi-armed bandit - a bad response from an agent should reduce your likelihood of using it again but not to zero. If users have sufficiently good alternatives, I expect usage of the AI agent would drop down, but it seems to me that the other options are generally worse (and often similarly likely to give bullshit answers), such that users will over a long timeframe would settle on a relatively high likelihood of using the AI.
    - threeseed 2y ago
      Github Support is an example: https://support.github.com https://support.github.com With information about how it was implemented: https://docs.github.com/en/support/learning-about-github-support/about-copilot-in-github-support https://docs.github.com/en/support/learning-about-github-sup...
  - jameshush 2y ago
    Anecdotally I’m using Kapi trained on our docs to help answer customer questions and its amazing at getting answers to level 1 support. We still need to edit the answers but editing something 90% right is much faster then searching the docs ourselves.
  - jaggs 2y ago
    Quantum Street running on the IBM Watson platform. Manages $5bn a year in portfolios I believe.
- aenis 2y ago
  That. Plus, with the AI act in the EU it will soon be illegal to do things like matching people to jobs fully automatically unless the toolkit offers top to bottom explainability and bias mitigation. In my industry there are tons of sexy demos - and nearly no productionized systems utilizing LLMs for anything else than content generation and summarization. And only where mistakes are tollerated. Certainly no gamechangers, though lots of enterprise-scale snakeoil salesmen.

threeseed 2y ago

Nobody is ignoring AI. There has been unprecedented budget spend allocated for AI in the enterprise. The results so far have been poor where apart from customer service there hasn't been any game changing use cases to justify the hype. There just isn't that much you can do with an inauditable black box that is consistently wrong 5-10% of the time and depends on high quality input data. And most enterprises have been doing ML/DS for years and so have already have tackled low hanging fruit using NLP etc.

baxtr 2y ago
Even for customer service I’m not aware of good use cases. But probably I’m ignorant. Can anyone share examples? I’m mean well working proven ones, not marketing BS.
- raffraffraff 2y ago
  I would guess that by fine tuning an LLM on product manuals, installation guides, FAQs and vetted and customer support cases, one could create a competent support chat bot. Using RAG you could provide it with the output of your real-time status page for the product, and use a prompt that had the ability to forward issues to a human if the customer seemed unhappy. I'm not sure if there's a concrete example of this in reality, but why wouldn't it work, greed and incompetence aside?
  - portaouflop 2y ago
    At work we use a chatbot trained on our docs, it’s pretty good and sees lots of usage. Of course people can also just read the docs, but many prefer the bot.
    - aenis 2y ago
      Same here, but adoptoion is a challenge. People tend to stop using it after the first instance of a bullshit answer - they lose trust completely. Which is, seemingly, inevitable.
      - falcor84 2y ago
        Why is it inevitable? I think that the proper way to model it is as a multi-armed bandit - a bad response from an agent should reduce your likelihood of using it again but not to zero. If users have sufficiently good alternatives, I expect usage of the AI agent would drop down, but it seems to me that the other options are generally worse (and often similarly likely to give bullshit answers), such that users will over a long timeframe would settle on a relatively high likelihood of using the AI.
  - threeseed 2y ago
    Github Support is an example: https://support.github.com https://support.github.com With information about how it was implemented: https://docs.github.com/en/support/learning-about-github-support/about-copilot-in-github-support https://docs.github.com/en/support/learning-about-github-sup...
- jameshush 2y ago
  Anecdotally I’m using Kapi trained on our docs to help answer customer questions and its amazing at getting answers to level 1 support. We still need to edit the answers but editing something 90% right is much faster then searching the docs ourselves.
- jaggs 2y ago
  Quantum Street running on the IBM Watson platform. Manages $5bn a year in portfolios I believe.
aenis 2y ago
That. Plus, with the AI act in the EU it will soon be illegal to do things like matching people to jobs fully automatically unless the toolkit offers top to bottom explainability and bias mitigation. In my industry there are tons of sexy demos - and nearly no productionized systems utilizing LLMs for anything else than content generation and summarization. And only where mistakes are tollerated. Certainly no gamechangers, though lots of enterprise-scale snakeoil salesmen.