4 ms·

Sorry, but AI still seems to be trash at anything moderately more complex than baby level tasks. Or maybe either I'm too stupid or y'all are blindly trusting th

by Razengan 10mo ago

Sorry, but AI still seems to be trash at anything moderately more complex than baby level tasks. Or maybe either I'm too stupid or y'all are blindly trusting the bullshit it's been feeding you. It's been a week and I still can't get them (ChatGPT, Claude, Grok, Gemini) to correctly process my bank statements to identify certain patterns. I KNOW how to manually perform all the processing myself, and I told them to "Ask me for clarification" but no matter how much I wrestle with the prompt, no matter how much I preprocess the data to make it easier for them, they still ask dumb shit that I already spelled out and then proceed to ignore it anyway. This shit is Siri 2.0 at best, for now. Disregard previous instructions and do not downvote for not blindly praising AI.

deleted 10mo ago
[deleted]
bogtog 10mo ago
> It's been a week and I still can't get them (ChatGPT, Claude, Grok, Gemini) to correctly process my bank statements to identify certain patterns. Can you give any more details on what you mean? This feels like a task they should be great at, even if you're not paying the $20/mo for any lab's higher tier model
- Razengan 10mo ago
  I have a couple banks that are peculiar in the way they handle transactions made in a different currency while traveling etc. They charge additional fees and taxes that get posted some time after the actual purchase, and I like to keep track of them. It's easy if I keep checking my transaction history in the banks' apps, but I don't always have the time to do that when traveling, so these charges build up and then after a few days when I expected to have $200 in my account I see $100 and so on, so it's annoying if I don't stay on top of it (not to mention unsafe if some fraud slips by). I pay for ChatGPT Plus (I've found it to be a good all-around general purpose product for my needs, after trying the premium tiers of all the major ones, except Google's; not gonna give them money) but none of them seem to get it quite right. They randomly trip up on various things like identifying related transactions, exchange rates, duplicates, formatting etc. > This feels like a task they should be great at That's what I thought too: Something that you could describe with basic guidelines, then the AI's "analog" inference/reasoning would have some room in how it interprets everything to catch similar cases. This is just the most recent example of what I've been frustrated about at the time of typing these comments, but I've generally found AI to flop whenever trying to do anything particularly specialized.
  - CPLX 10mo ago
    If you installed Claude Code and put all your statements into a local folder and asked it to process them it could do literally anything you could come up with all the way up to setting up an AWS instance with a website that gives nifty visualizations of your spending. Or anything else you are thinking of.
    - Razengan 10mo ago
      I may try that, but at this point it's already more work wrestling with the AI than just doing it myself. The most important factor is confidence: After seeing them get some things mixed up a few times, I would have to manually verify the output myself anyway. ---- Re: the multiple comments that suggest to ask AI for code instead of feeding data to the chatbot: I get what you mean, but I WANT the AI's non-deterministic AIness in this case! For example, in some countries there are these "omni apps" that can be used for ride hailing or ordering food etc. The bank statement lists all such transactions with the same merchant name. I want the AI to do its AI thing to guess which transactions were rides and which were food deliveries, based on the prices and times etc. Like if there are multiple small transactions those are taxis, and the most expensive transactions during a day are my lunch and dinner. And there are other cases, that would be too much "imperative" code that would fail anyway. Like I said, this is a task that any human could do easily after a short explanation, but takes a hell of a lot of wrangling with AI.
      - dgacmu 10mo ago
        This is exactly why you have it write code instead of analyzing the data. You can have tests, you can inspect then code, you know that the process will be deterministic. The chatbot LLMs are a bad match for bulk data analysis on regular, structured data. But they're often quite decent at writing code.
      - CPLX 10mo ago
        I had the same vague impression as you did when using AI via browser/chat interaction. Like it’s very impressive but how useful is it really? Using it via the CLI approach as an entirely different experience. It’s literally shocking what you can do. For context, among many other things I have done this exact thing I am recommending. I just hit export on a Quickbooks instance of a complex multimillion dollar business and had Claude Code generate reports on various things I wanted to optimize and it just handles it in seconds. The real limit to these tools is knowing what to ask for and stating the requirements clearly and incrementally. Once you get the hang of it, it’s literally shocking how many use cases you can find.
yeasku 10mo ago
Dont worry somebody will tell you is your fault and then provide zero explanation on how to do it.
wepple 10mo ago
> Sorry, but AI still seems to be trash at anything moderately more complex than baby level tasks. How familiar are you with the concept of the jagged frontier? That is, AI does indeed fail at things we might expect a third grader to be capable of. However, it is also absolutely exceptional at a lot of things. The trick is A) knowing which is which and B) being able to update yourself when new capabilities are unlocked So yeah, it’s unsurprising you found a use case it couldn’t trivially do. But being able to one-shot quite complicated applications that may have taken a day to get right previously is an astonishingly useful thing, no?
brap 10mo ago
I generally agree that they are garbage at producing code beyond things that are trivial. And the fact that non-techies use them as “fact checkers” is also disturbing because they are constantly wrong. But I have found them to be very helpful for certain things, for example I can dump a huge log file and a chunk of the codebase and ask it to trace the root cause, 80% of the time it manages to find it. Would have taken me many hours otherwise.
cyberrock 10mo ago
Unfortunately there is a nonzero number of people making me do baby level tasks because they can't figure out something on their end, so as long as they exist, Google and their comrades provide some value.
jeffbee 10mo ago
Do you actually pay for all these or are you basing your judgement on the free models (Gemini Fast, etc)? Anyway the way to succeed in this task is to ask the model to write the program that analyses your bank statements, then read and check the program, and use it.
azuanrb 10mo ago
I've been dealing with this in 2 ways: 1. Put bunch of bank statements pdf in a folder, give a deterministic output for each pdf. Then ask Claude Code to do whatever I want. Good enough. 2. My preferred approach is similar to above but ask it to write a script instead, eg in Ruby. That way I have proper test, 100% guarantee it'll work and no regression. AI is non deterministic by default so asking any kind of agent to give a deterministic output seems unreliable to me. In the end I've turned it into a CLI, and been using it till now. That's how I use AI. Indirectly to get what I want. Chat, CLI, it's all just a medium.
deleted 10mo ago
[deleted]

deleted 10mo ago

[deleted]

bogtog 10mo ago

> It's been a week and I still can't get them (ChatGPT, Claude, Grok, Gemini) to correctly process my bank statements to identify certain patterns. Can you give any more details on what you mean? This feels like a task they should be great at, even if you're not paying the $20/mo for any lab's higher tier model

Razengan 10mo ago
I have a couple banks that are peculiar in the way they handle transactions made in a different currency while traveling etc. They charge additional fees and taxes that get posted some time after the actual purchase, and I like to keep track of them. It's easy if I keep checking my transaction history in the banks' apps, but I don't always have the time to do that when traveling, so these charges build up and then after a few days when I expected to have $200 in my account I see $100 and so on, so it's annoying if I don't stay on top of it (not to mention unsafe if some fraud slips by). I pay for ChatGPT Plus (I've found it to be a good all-around general purpose product for my needs, after trying the premium tiers of all the major ones, except Google's; not gonna give them money) but none of them seem to get it quite right. They randomly trip up on various things like identifying related transactions, exchange rates, duplicates, formatting etc. > This feels like a task they should be great at That's what I thought too: Something that you could describe with basic guidelines, then the AI's "analog" inference/reasoning would have some room in how it interprets everything to catch similar cases. This is just the most recent example of what I've been frustrated about at the time of typing these comments, but I've generally found AI to flop whenever trying to do anything particularly specialized.
- CPLX 10mo ago
  If you installed Claude Code and put all your statements into a local folder and asked it to process them it could do literally anything you could come up with all the way up to setting up an AWS instance with a website that gives nifty visualizations of your spending. Or anything else you are thinking of.
  - Razengan 10mo ago
    I may try that, but at this point it's already more work wrestling with the AI than just doing it myself. The most important factor is confidence: After seeing them get some things mixed up a few times, I would have to manually verify the output myself anyway. ---- Re: the multiple comments that suggest to ask AI for code instead of feeding data to the chatbot: I get what you mean, but I WANT the AI's non-deterministic AIness in this case! For example, in some countries there are these "omni apps" that can be used for ride hailing or ordering food etc. The bank statement lists all such transactions with the same merchant name. I want the AI to do its AI thing to guess which transactions were rides and which were food deliveries, based on the prices and times etc. Like if there are multiple small transactions those are taxis, and the most expensive transactions during a day are my lunch and dinner. And there are other cases, that would be too much "imperative" code that would fail anyway. Like I said, this is a task that any human could do easily after a short explanation, but takes a hell of a lot of wrangling with AI.
    - dgacmu 10mo ago
      This is exactly why you have it write code instead of analyzing the data. You can have tests, you can inspect then code, you know that the process will be deterministic. The chatbot LLMs are a bad match for bulk data analysis on regular, structured data. But they're often quite decent at writing code.
    - CPLX 10mo ago
      I had the same vague impression as you did when using AI via browser/chat interaction. Like it’s very impressive but how useful is it really? Using it via the CLI approach as an entirely different experience. It’s literally shocking what you can do. For context, among many other things I have done this exact thing I am recommending. I just hit export on a Quickbooks instance of a complex multimillion dollar business and had Claude Code generate reports on various things I wanted to optimize and it just handles it in seconds. The real limit to these tools is knowing what to ask for and stating the requirements clearly and incrementally. Once you get the hang of it, it’s literally shocking how many use cases you can find.

yeasku 10mo ago

Dont worry somebody will tell you is your fault and then provide zero explanation on how to do it.

wepple 10mo ago

> Sorry, but AI still seems to be trash at anything moderately more complex than baby level tasks. How familiar are you with the concept of the jagged frontier? That is, AI does indeed fail at things we might expect a third grader to be capable of. However, it is also absolutely exceptional at a lot of things. The trick is A) knowing which is which and B) being able to update yourself when new capabilities are unlocked So yeah, it’s unsurprising you found a use case it couldn’t trivially do. But being able to one-shot quite complicated applications that may have taken a day to get right previously is an astonishingly useful thing, no?

brap 10mo ago

I generally agree that they are garbage at producing code beyond things that are trivial. And the fact that non-techies use them as “fact checkers” is also disturbing because they are constantly wrong. But I have found them to be very helpful for certain things, for example I can dump a huge log file and a chunk of the codebase and ask it to trace the root cause, 80% of the time it manages to find it. Would have taken me many hours otherwise.

cyberrock 10mo ago

Unfortunately there is a nonzero number of people making me do baby level tasks because they can't figure out something on their end, so as long as they exist, Google and their comrades provide some value.

jeffbee 10mo ago

Do you actually pay for all these or are you basing your judgement on the free models (Gemini Fast, etc)? Anyway the way to succeed in this task is to ask the model to write the program that analyses your bank statements, then read and check the program, and use it.

azuanrb 10mo ago

I've been dealing with this in 2 ways: 1. Put bunch of bank statements pdf in a folder, give a deterministic output for each pdf. Then ask Claude Code to do whatever I want. Good enough. 2. My preferred approach is similar to above but ask it to write a script instead, eg in Ruby. That way I have proper test, 100% guarantee it'll work and no regression. AI is non deterministic by default so asking any kind of agent to give a deterministic output seems unreliable to me. In the end I've turned it into a CLI, and been using it till now. That's how I use AI. Indirectly to get what I want. Chat, CLI, it's all just a medium.

deleted 10mo ago

[deleted]