3 ms·

I don't trust these AI-only companies to be overnight experts in properly handling medical, financial and insurance data. They have no business providing these

I don't trust these AI-only companies to be overnight experts in properly handling medical, financial and insurance data. They have no business providing these tools, unless they want to take all the risk too.

tossandthrow 6mo ago
I would recommend you to not use these, if you are not willing to absorb the risk. Luckily there is still a significant market for the services.
- stubish 6mo ago
  Some human always gets to be the certified fall guy for non-compliance. Maybe the legal agent can help structure the company so that is an ignorant lower level accountant and not the CFO. Currently we don't know the risk, so it is kind of hard to absorb.
  - Terr_ 6mo ago
    Decade-old spoilers for "How I Met Your Mother" ... but there's a character who has that kind of job, as a legal meat-shield. https://www.youtube.com/watch?v=8u62HptZ6TE https://www.youtube.com/watch?v=8u62HptZ6TE
areoform 6mo ago
Claude's actually pretty great at this! I actually used to use Claude A LOT to answer interesting questions (which I'll be writing up on!) More generally, Claude is palpably different from most other agents. I'd recommend these models – especially Opus – without qualifications. But there's a process risk here based on their current practises. I'm hoping those practises change so that I can recommend Claude to everyone I know, but as of now, there's existential risk exposure here that's greater than Google's. Anthropic's automated systems can and will ban you for pretty arbitrary things; and you won't get human support or Claude – even if you are an enterprise paying out of your nose. And there's 0 redressal unless you go viral on social media. Or know someone who knows someone. See: https://x.com/Whizz_ai/status/2051180043355967802 https://x.com/Whizz_ai/status/2051180043355967802 https://x.com/theo/status/2045618854932734260 https://x.com/theo/status/2045618854932734260 And I say that as someone who likes how Anthropic has been training Claude and Opus. I just don't think they're prepared to be the trillion dollar company they've become. They are – in a very real way – suffering from success. Which is extremely inconvenient to be on the receiving end of when you're on a deadline.
- dakolli 6mo ago
  They aren't even close to a 1T company, they're valued at <400bb and that's at like a 20x-30x multiple. They can probably raise money at a higher valuation but its literally just value based on hype, not revenue.
  - areoform 6mo ago
    https://www.businessinsider.com/anthropic-trillion-dollar-valuation-on-secondary-markets-2026?op=1 https://www.businessinsider.com/anthropic-trillion-dollar-va...
  - KellyCriterion 6mo ago
    Check the secondaries market ;-)
    - dakolli 6mo ago
      fomo/hype, not revenue. Google's AI business is a profitable business model and training to inference is vertically integrated. Their AI biz did not add 1T to their market cap, despite their much more advantageous position. A 1T valuation for Anthropic makes absolutely no sense. It also makes no sense to me there are people qualified to participate in these secondary markets who are that stupid, but here we are.
      - KellyCriterion 6mo ago
        I do know 2 people participating in secondaries, one of them explicitly with Anthropic shares: I would not consider any of them stupid :-) And for participating there, there is not "a qualification that allows you to enter", its other metrics. If Anthropics valuation makes no sense - fair enough - but why is then OAI evaluation of 850b correct?
- intended 6mo ago
  > and you won't get human support or Claude – even if you are an enterprise paying out of your nose. And there's 0 redressal unless you go viral on social media. Sadly this sounds like par for the course when it comes to tech. Too many messages and requests for help depend on knowing someone in the right slack groups.
motbus3 6mo ago
The only reason they are doing it is because there are regulation for people but not for machines.
- tyre 6mo ago
  This is objectively not true. You can’t get around HIPAA by saying “lol wasn’t me it was an Agent”
  - deepfriedbits 6mo ago
    Yep. There's no certification needed to create a financial model or close monthly books.
    - nradov 6mo ago
      HIPAA doesn't require any certification either. Some organizations voluntarily choose to earn certification from private companies that offer certifications for compliance with HIPAA privacy or administrative simplification rules but this is completely optional.
      - deleted 6mo ago
        [deleted]
    - mwwaters 6mo ago
      For doing some reporting stuff internally, there isn’t a certification. But there are definitely humans who have to certify financial statements and communications for financial offerings.
  - bluefirebrand 6mo ago
    You can bet that someone out there is probably trying to build a startup right now based on that idea.
- greenpresident 6mo ago
  My experience has been quite the opposite. Some bank processes remain oral traditions about clicking excel filters by hand because any code would have to be extensively documented and tested.
zx8080 6mo ago
> properly handling Why, they can sell user data to other brokers. Experts indeed! But not in insurance or finance, of course.
scottyah 6mo ago
I think a lot of people are misunderstanding the typical workload of people in Financial Services. They aren't using Claude to transfer money, they're just building a LOT of slideshows and fancy excel docs on made-up numbers to try to sell mergers and new financing options/types of loans. Most programmers would just consider this "sales".
- rubyfan 6mo ago
  That’s a gross over generalization. Some of the insurance data here suggests use of AI to make underwriting decisions. There are several states with regulations which could potentially pull these agent solutions into their regulatory oversight if used by the industry to effect insurance outcomes.
  - cschneid 6mo ago
    Odd lots podcast had an interesting snippet about an financial institution that uses AI to make loan decisions. The guest said that they only use it on applicants who were rejected in the traditional sequence, and then uses AI to accept them if possible. That way there's an articulable reason for a rejection, but they use the non-deterministic AI to allow an extra person through - since the laws about loans are mostly around not discriminating against people - companies are (generally) welcome to accept whoever.
    - andwur 6mo ago
      That's dependent on the credit laws of the country in question though. In Australia you have it both ways, you cannot unreasonably discriminate (e.g. race, gender etc) but at the same time you are forbidden from issuing credit to applicants who cannot meet the affordability requirements of said credit. E.g. issuing a loan to a customer who provably cannot afford it is a breach of the NCC, and the company is held responsible for this. As a credit provider you must make reasonable enquiries into a customer's financial position, failing to do this is a breach. You must also be able to explain and justify the decision to issue credit if challenged by the civil regulator (AFCA - who are granted significant power in addressing this), on the basis of a customer complaint, and they most certainly do not accept "human said no but the computer then said yes" without hard facts such as proven positive income flow (pay slips, bank statements), known expenses, liabilities and reliable credit history.

tossandthrow 6mo ago

I would recommend you to not use these, if you are not willing to absorb the risk. Luckily there is still a significant market for the services.

stubish 6mo ago
Some human always gets to be the certified fall guy for non-compliance. Maybe the legal agent can help structure the company so that is an ignorant lower level accountant and not the CFO. Currently we don't know the risk, so it is kind of hard to absorb.
- Terr_ 6mo ago
  Decade-old spoilers for "How I Met Your Mother" ... but there's a character who has that kind of job, as a legal meat-shield. https://www.youtube.com/watch?v=8u62HptZ6TE https://www.youtube.com/watch?v=8u62HptZ6TE

areoform 6mo ago

Claude's actually pretty great at this! I actually used to use Claude A LOT to answer interesting questions (which I'll be writing up on!) More generally, Claude is palpably different from most other agents. I'd recommend these models – especially Opus – without qualifications. But there's a process risk here based on their current practises. I'm hoping those practises change so that I can recommend Claude to everyone I know, but as of now, there's existential risk exposure here that's greater than Google's. Anthropic's automated systems can and will ban you for pretty arbitrary things; and you won't get human support or Claude – even if you are an enterprise paying out of your nose. And there's 0 redressal unless you go viral on social media. Or know someone who knows someone. See: https://x.com/Whizz_ai/status/2051180043355967802 https://x.com/Whizz_ai/status/2051180043355967802 https://x.com/theo/status/2045618854932734260 https://x.com/theo/status/2045618854932734260 And I say that as someone who likes how Anthropic has been training Claude and Opus. I just don't think they're prepared to be the trillion dollar company they've become. They are – in a very real way – suffering from success. Which is extremely inconvenient to be on the receiving end of when you're on a deadline.

dakolli 6mo ago
They aren't even close to a 1T company, they're valued at <400bb and that's at like a 20x-30x multiple. They can probably raise money at a higher valuation but its literally just value based on hype, not revenue.
- areoform 6mo ago
  https://www.businessinsider.com/anthropic-trillion-dollar-valuation-on-secondary-markets-2026?op=1 https://www.businessinsider.com/anthropic-trillion-dollar-va...
- KellyCriterion 6mo ago
  Check the secondaries market ;-)
  - dakolli 6mo ago
    fomo/hype, not revenue. Google's AI business is a profitable business model and training to inference is vertically integrated. Their AI biz did not add 1T to their market cap, despite their much more advantageous position. A 1T valuation for Anthropic makes absolutely no sense. It also makes no sense to me there are people qualified to participate in these secondary markets who are that stupid, but here we are.
    - KellyCriterion 6mo ago
      I do know 2 people participating in secondaries, one of them explicitly with Anthropic shares: I would not consider any of them stupid :-) And for participating there, there is not "a qualification that allows you to enter", its other metrics. If Anthropics valuation makes no sense - fair enough - but why is then OAI evaluation of 850b correct?
intended 6mo ago
> and you won't get human support or Claude – even if you are an enterprise paying out of your nose. And there's 0 redressal unless you go viral on social media. Sadly this sounds like par for the course when it comes to tech. Too many messages and requests for help depend on knowing someone in the right slack groups.

motbus3 6mo ago

The only reason they are doing it is because there are regulation for people but not for machines.

tyre 6mo ago
This is objectively not true. You can’t get around HIPAA by saying “lol wasn’t me it was an Agent”
- deepfriedbits 6mo ago
  Yep. There's no certification needed to create a financial model or close monthly books.
  - nradov 6mo ago
    HIPAA doesn't require any certification either. Some organizations voluntarily choose to earn certification from private companies that offer certifications for compliance with HIPAA privacy or administrative simplification rules but this is completely optional.
    - deleted 6mo ago
      [deleted]
  - mwwaters 6mo ago
    For doing some reporting stuff internally, there isn’t a certification. But there are definitely humans who have to certify financial statements and communications for financial offerings.
- bluefirebrand 6mo ago
  You can bet that someone out there is probably trying to build a startup right now based on that idea.
greenpresident 6mo ago
My experience has been quite the opposite. Some bank processes remain oral traditions about clicking excel filters by hand because any code would have to be extensively documented and tested.

zx8080 6mo ago

> properly handling Why, they can sell user data to other brokers. Experts indeed! But not in insurance or finance, of course.

scottyah 6mo ago

I think a lot of people are misunderstanding the typical workload of people in Financial Services. They aren't using Claude to transfer money, they're just building a LOT of slideshows and fancy excel docs on made-up numbers to try to sell mergers and new financing options/types of loans. Most programmers would just consider this "sales".

rubyfan 6mo ago
That’s a gross over generalization. Some of the insurance data here suggests use of AI to make underwriting decisions. There are several states with regulations which could potentially pull these agent solutions into their regulatory oversight if used by the industry to effect insurance outcomes.
- cschneid 6mo ago
  Odd lots podcast had an interesting snippet about an financial institution that uses AI to make loan decisions. The guest said that they only use it on applicants who were rejected in the traditional sequence, and then uses AI to accept them if possible. That way there's an articulable reason for a rejection, but they use the non-deterministic AI to allow an extra person through - since the laws about loans are mostly around not discriminating against people - companies are (generally) welcome to accept whoever.
  - andwur 6mo ago
    That's dependent on the credit laws of the country in question though. In Australia you have it both ways, you cannot unreasonably discriminate (e.g. race, gender etc) but at the same time you are forbidden from issuing credit to applicants who cannot meet the affordability requirements of said credit. E.g. issuing a loan to a customer who provably cannot afford it is a breach of the NCC, and the company is held responsible for this. As a credit provider you must make reasonable enquiries into a customer's financial position, failing to do this is a breach. You must also be able to explain and justify the decision to issue credit if challenged by the civil regulator (AFCA - who are granted significant power in addressing this), on the basis of a customer complaint, and they most certainly do not accept "human said no but the computer then said yes" without hard facts such as proven positive income flow (pay slips, bank statements), known expenses, liabilities and reliable credit history.