5 ms·

> this is shockingly close to replacing an entire role at companies with only a couple hours of effort. and > It seems like there’s almost no limit to how goo

> this is shockingly close to replacing an entire role at companies with only a couple hours of effort. and > It seems like there’s almost no limit to how good GPT could get at this. I don't see how that's a valid conclusion given the results. 2 simple things right, moderate to difficult things wrong? Hardly a ringing endorsement.

Eji1700 4y ago
Glad i'm not the only one who saw this. Feels extremely overblown. Very impressive, don't get me wrong, and is yet another bridge closer to letting business side do plain text queries. But that said, stuff like this has existed for awhile, and it's certainly not replacing analysts. For a start, this assumes a VERY clean model, which is of course what you strive for, but the moment you step out of FAANG you'll find that many places have a decent, but not ideal, model/warehouse/db which only makes things like this worse. Getting the simple examples right is interesting, but hardly replacing an analyst. A basic background around tech and some skill navigating w3schools would replace an analyst if this was the bar. The mildly complex queries being "almost" right is frankly horrifying. Others have mentioned how much of a "oh fuck" it is to send out bad data, and having some AI generate your answers an be SLIGHTLY off is basically worse than it being obviously off. You now have to check every query it generates for errors, because you won't be able to glance check if its sane or not. Finally, at least where I work, these probably wouldn't even qualify as medium complexity queries? I don't really know how the industry at large would qualify these, but "using 2 inner joins, a where, a group, and a order" all falls under beginner skills in my eyes? Its still fascinating, and I'm glad they showed HOW they did it, but this isn't replacing anyone, at least if you're paying taxes off the data.
- RandomBK 4y ago
  >the moment you step out of FAANG Even in FAANG, most data is going to be extremely messy. * There's usually very little incentive for good documentation, so you have to guess what the table contains based on column names and exploring patterns in the data. * Enforcing a single pattern/process for data across the entire org is a pipe dream. * Logging tables are written to by hundreds of engineers across dozens of teams, with no standardized naming scheme. * The product contains multiple edge cases/special overrides for very specific circumstances, so there's very few simple queries that don't come with 100 footnotes attached. FAANG is not immune to large-organization problems, and data quality is no exception.
  - Eji1700 4y ago
    Oh yeah im not surprised to hear that, Ive just known one or two people who've been in similar sized companies as analysts, and while the underlying table structure was a nightmare, the data model they dealt with was pretty clean. But with that in mind, thats because theres a major pipeline of people and processes to get the data to that point, and it meant that there could sometimes be significant delay on new KPI's as they had to be cleanly worked into the model, and it of course didn't represent everything.
  - foobazgt 4y ago
    Can't say speak for FAANG, but can confirm for a $100B+ business that analytics was a huge mess. There was constant investment on both the engineering and analytics functions to wrangle a coherent view on top of the underlying operational data model.
- LeftHandPath 4y ago
  Still, while it might not be useful yet, can’t it stand its own as an intriguing proof of concept? The first car was hardly ready to replace the horse. But look where we are now.
  - Eji1700 4y ago
    Sure, if that was the title of the article. Instead its saying its ready to replace a full time position, which its not even close to. And this is something people have been saying about things like AI and crypto for years now. They've both made some interesting developments, and they both face serious hurdles (some fundamental). AI obviously has a ton of real world use cases, but overhyping things like this doesnt actually help.
  - ResearchCode 4y ago
    Large language models are very intriguing as a technology to replace business people. Make a few PowerPoint slides, fill in an excel sheet. Generate a summary for a pointless meeting.
  - JohnBooty 4y ago
    A car is mostly an amalgamation of mechanical devices and we have a solid track record of steadily and incrementally improving those over dozens of millennia. Modern cars also have a lot of computing power in the form of fairly dumb systems that are good at sensing things and following rules e.g. "based on conditions X, Y, and Z the flow of the fuel injector nozzle should be N". Got a solid track record here too albiet one that's decades long and not millennia. A tool that replaces a human data analyst needs to do a lot of inference. When I'm translating stakeholder requests into code and reports, it's about 10% coding and 90% human inference. Remains to be seen how good machines can get there.
urbandw311er 4y ago
I came here to say the same! I was “shockingly close” to winning the lottery last week… …the guy who lives next door won.

Eji1700 4y ago

Glad i'm not the only one who saw this. Feels extremely overblown. Very impressive, don't get me wrong, and is yet another bridge closer to letting business side do plain text queries. But that said, stuff like this has existed for awhile, and it's certainly not replacing analysts. For a start, this assumes a VERY clean model, which is of course what you strive for, but the moment you step out of FAANG you'll find that many places have a decent, but not ideal, model/warehouse/db which only makes things like this worse. Getting the simple examples right is interesting, but hardly replacing an analyst. A basic background around tech and some skill navigating w3schools would replace an analyst if this was the bar. The mildly complex queries being "almost" right is frankly horrifying. Others have mentioned how much of a "oh fuck" it is to send out bad data, and having some AI generate your answers an be SLIGHTLY off is basically worse than it being obviously off. You now have to check every query it generates for errors, because you won't be able to glance check if its sane or not. Finally, at least where I work, these probably wouldn't even qualify as medium complexity queries? I don't really know how the industry at large would qualify these, but "using 2 inner joins, a where, a group, and a order" all falls under beginner skills in my eyes? Its still fascinating, and I'm glad they showed HOW they did it, but this isn't replacing anyone, at least if you're paying taxes off the data.

RandomBK 4y ago
>the moment you step out of FAANG Even in FAANG, most data is going to be extremely messy. * There's usually very little incentive for good documentation, so you have to guess what the table contains based on column names and exploring patterns in the data. * Enforcing a single pattern/process for data across the entire org is a pipe dream. * Logging tables are written to by hundreds of engineers across dozens of teams, with no standardized naming scheme. * The product contains multiple edge cases/special overrides for very specific circumstances, so there's very few simple queries that don't come with 100 footnotes attached. FAANG is not immune to large-organization problems, and data quality is no exception.
- Eji1700 4y ago
  Oh yeah im not surprised to hear that, Ive just known one or two people who've been in similar sized companies as analysts, and while the underlying table structure was a nightmare, the data model they dealt with was pretty clean. But with that in mind, thats because theres a major pipeline of people and processes to get the data to that point, and it meant that there could sometimes be significant delay on new KPI's as they had to be cleanly worked into the model, and it of course didn't represent everything.
- foobazgt 4y ago
  Can't say speak for FAANG, but can confirm for a $100B+ business that analytics was a huge mess. There was constant investment on both the engineering and analytics functions to wrangle a coherent view on top of the underlying operational data model.
LeftHandPath 4y ago
Still, while it might not be useful yet, can’t it stand its own as an intriguing proof of concept? The first car was hardly ready to replace the horse. But look where we are now.
- Eji1700 4y ago
  Sure, if that was the title of the article. Instead its saying its ready to replace a full time position, which its not even close to. And this is something people have been saying about things like AI and crypto for years now. They've both made some interesting developments, and they both face serious hurdles (some fundamental). AI obviously has a ton of real world use cases, but overhyping things like this doesnt actually help.
- ResearchCode 4y ago
  Large language models are very intriguing as a technology to replace business people. Make a few PowerPoint slides, fill in an excel sheet. Generate a summary for a pointless meeting.
- JohnBooty 4y ago
  A car is mostly an amalgamation of mechanical devices and we have a solid track record of steadily and incrementally improving those over dozens of millennia. Modern cars also have a lot of computing power in the form of fairly dumb systems that are good at sensing things and following rules e.g. "based on conditions X, Y, and Z the flow of the fuel injector nozzle should be N". Got a solid track record here too albiet one that's decades long and not millennia. A tool that replaces a human data analyst needs to do a lot of inference. When I'm translating stakeholder requests into code and reports, it's about 10% coding and 90% human inference. Remains to be seen how good machines can get there.

urbandw311er 4y ago

I came here to say the same! I was “shockingly close” to winning the lottery last week… …the guy who lives next door won.