4 ms·

Great! Maybe I can finally learn what I'm suppose to be using generative AI for to be more productive. I'll be tuning in and spinning up whatever models/tools

by hx8 2y ago

Great! Maybe I can finally learn what I'm suppose to be using generative AI for to be more productive. I'll be tuning in and spinning up whatever models/tools they suggest, but the longer this tech wave occurs the more confident I am that gen-ai is going to tally up to be an at-most 3% lift on global productivity.

onemoresoop 2y ago
Yeah, but finally everybody can now bulshit freely aided by their personal llm, not just the natural bulshitters; but I forsee that bullshitting may spike up a bit now then fall out of fashion for something even more fleeting as people’s attention span is getting more and more fractured.
azinman2 2y ago
Here’s something scary I recently learned: Cedar-Sinai in LA (major hospital) used to have 15 lawyers on contracts. Now they have 1 and an AI app reviewing contracts. Those are 14 lawyers gone. That’s more than 3% on “productivity”, but 14 people who lost their jobs. And that’s now with the current state of things.
- broken_clock 2y ago
  Can you share a source/article?
- linkregister 2y ago
  Do you have an article that has more information about this? I'd really like to learn more about what happened.
- NBJack 2y ago
  Going to be interesting to see the MTTL (mean time to lawsuit) on this. Sounds grossly negligent. I feel kinda sorry for the lonely lawyer.
  - DannyBee 2y ago
    Highly doubtful it is either negligence, or something they will get sued over. I don't even quite understand what you think they will get sued over. Most of the policing would happen by courts and bar associations.
  - schmookeeg 2y ago
    As someone who spends a lot of time battling mis-use/over-use of PII constantly, I am adopting MTTL as a term of art. :D
    - hx8 2y ago
      MTTL really applies more to a startup portfolio, has long does the average startup from an incubator operate before it receives it's first lawsuit. For your situation, you really want to measure how many PII records you handle per lawsuit. That way you can accurately measure the lawsuit cost per record and compare it to revenue per record to see if you're profitable.
- DannyBee 2y ago
  Lawyer here - there are fields, and law is definitely one of them, where labor is the major cost. That labor is not often used sanely. It is common to use lawyers costing hundreds per hour to do fairly basic document review and summarization. That is, to produce a fairly simple artifact. Not legal research, not opinionated briefing. But literal: Read these documents, produce a summary of what they say. While I can't say this is the same as what you are talking about ("contracts review" means many things to many people), i'm not even the slightest bit surprised that AI is starting to replace remarkably inefficient uses of labor in law. I will add: Lots of funding being thrown at AI legal startups around on products that do document review and summarization, but that's not the big fish, and will be commodity very quickly. So i expect there will be an ebb and flow of these sorts of products as the startups either move on to things that enable them to capture a meaningful market (document review ain't it), or die and leave these companies hanging :)
  - azinman2 2y ago
    But how do you know if AI is able to pick out the salient bits for summarization? Like some nasty point poison pill hidden in there? Wouldn’t you want an expert for such things?
    - chefandy 2y ago
      Your last statement hints at a big consideration: accountability. One lawyer on a formerly 15 lawyer staff is accountable for 15 lawyers worth of potential mistakes, and we know that “but the AI did it!” doesn’t hold water in law.
      - DannyBee 2y ago
        There's a bunch of assumptions here. The main one i think is probably wrong is that there was 15 lawyers worth of work being done before (when measured by some average lawyer standard). For example, it's possible there was only really 1 lawyer worth of work being split 15 ways, so each lawyer was really only responsible for 1/15th of an average lawyers amount of work :) In that scenario, they'd only be responsible for 1 average lawyers worth of mistakes now. Is that realistic? Who knows. I've definitely seen that level of "waste" (for lack of a better term) before in law firms :) Even in the scenario you are positing, it's not obvious it matters as much as you seem to think it does. If the per-lawyer mistake rate was low enough, it may be that 15x that rate simply does not matter. These kinds of contracts are fairly standardized, and so they are mostly looking at the differences from last time. Those differences are often not legal as much as factual. IE the table of costs changed, not the legal responsibility. So the main thing mistakes get you is maybe cost (if mistakes matter at all). This isn't like they are seeing brand new from scratch contracts constantly that require brand new analysis. Even if they were, like I said, the main issue with a mistake is cost. For all we know, the AI company also agreed to indemnify them for a certain rate of mistakes or something (which wouldn't be hard to get insurance for). I'm not actually a fan of AI taking necessary jobs, but I think the view here that this is sort of life or death is strange. I'd be much more worried about AI handling criminal defense in some semi-autonomous fashion than this.
- seunosewa 2y ago
  That's great. That means potentially slightly cheaper healthcare. A company shouldn't need so many lawyers unless it's a legal firm.
TheAceOfHearts 2y ago
I was pretty cynical trying earlier models, but with Gemini Flash 2.0 I felt like there was a pretty significant boost in usability and capabilities. In particular I've found that these tools make it a lot easier to explore or get started with unfamiliar domains. One of my big issues has often been decision paralysis, so having a tool to help me narrow down the list of resources and make it more approachable has been a huge win. My general experience has been that getting AI tools to directly do stuff for you tends to produce pretty bad results, but you can use it as a force multiplier for your own capabilities. If I'm confused or uncertain about how to do something, AI tools are usually pretty good at clarifying what needs to be done.
- chefandy 2y ago
  Maybe there’s a particular cognitive profile that benefits most from LLM chat bots? I’ve tried multiple times to realize this force multiplier in my life for everything from day-to-day stuff to picking up new things, using the latest paid bots with the best models, and I’ve persistently found them to be awkward, inaccurate, difficult to pin down into giving useful info rather than a bunch of non-committal pablum without hallucinations, etc etc etc
  - j7ake 2y ago
    Maybe you can give some specific examples
    - chefandy 2y ago
      For what — things that AI wasn’t helpful for when I gave it a shot?
  - anon373839 2y ago
    They’re useful for tasks that don’t require correctness: brainstorming, exploratory research, sketching out the vague shape of a solution to a problem. They’re mediocre to awful at consistently following instructions, unless you have the fortune of having a task and domain that are well represented in the post-training data. Yesterday I needed to generate ~500 filenames given a (human written) summary of each document’s contents. This seemed to be the perfect task to throw an LLM into a for loop. Yet it took three hours of prompt engineering to get a passable set of results - yes, I could’ve done it by hand in that time. Each iteration on the prompt revealed new blind spots, new ways in which the models would latch onto irrelevant details or gloss over the main point.
throw83288 2y ago
Cursor Composer is a game-changer for greenfield projects, it's definitely not 3% change.
DANmode 2y ago
Folks are busy optimizing their own building, not telling you how to optimize yours...
edanm 2y ago
> Maybe I can finally learn what I'm suppose to be using generative AI for to be more productive. So many things. It's a general-purpose "thing doer" in many situations where you otherwise wouldn't have one. Let me give a super-simple example - not a high-value one, but an example of obvious value IMO. Say for some reason, you have a screenshot of a bunch of text. Maybe you took a picture of a page from a book or something, idk. Now you want it in textual form. You can throw it in ChatGPT and ask it to give you the text, and a few seconds later you have the text. I'm not saying there are no other solutions for this - there are. You can look for some software to do OCR or something. But that's what makes ChatGPT or others general-purpose - they're a one-stop shop for a lot of different things, including small one-off tasks like this. I can name a dozen other one-off tasks that it helps me with. Again, not the most high-value things it helps me with (that'd be programming help), but an undeniable example of value, IMO. How would you do this without an LLM? (I personally would've just typed it up myself, probably.)
- hx8 2y ago
  > How would you do this without an LLM? (I personally would've just typed it up myself, probably.) This has been a feature of Apple Preview (default image program) for years and years. You can just highlight text and copy it from a jpeg or png.
  - edanm 2y ago
    OK, first of all, really cool. I don't think I knew this! Thank you. But how about a next step that I use LLMs for - taking this text from a document and reformatting it as, say, bullet points. E.g. I literally had this in a Jupyter notebook: [some_var_name, some_other_name, ...] and a bunch of those, and I wanted them redone as bullet points. It was a bit more complicated than that but I'm simplifying for the example. I literally took a screenshot, through it in ChatGPT and got back the correctly-formatted list. There are other ways to solve something like this of course (normally I'd put it in VSCode and use multiple cursors or macros), but I don't think there's anything that can go from a screenshot and a one-sentence description, to having finished the task, all in a single tool (that can also solve 100 other problems that come up in my work similarly easily).

by hx8 2y ago

onemoresoop 2y ago

Yeah, but finally everybody can now bulshit freely aided by their personal llm, not just the natural bulshitters; but I forsee that bullshitting may spike up a bit now then fall out of fashion for something even more fleeting as people’s attention span is getting more and more fractured.

azinman2 2y ago

Here’s something scary I recently learned: Cedar-Sinai in LA (major hospital) used to have 15 lawyers on contracts. Now they have 1 and an AI app reviewing contracts. Those are 14 lawyers gone. That’s more than 3% on “productivity”, but 14 people who lost their jobs. And that’s now with the current state of things.

broken_clock 2y ago
Can you share a source/article?
linkregister 2y ago
Do you have an article that has more information about this? I'd really like to learn more about what happened.
NBJack 2y ago
Going to be interesting to see the MTTL (mean time to lawsuit) on this. Sounds grossly negligent. I feel kinda sorry for the lonely lawyer.
- DannyBee 2y ago
  Highly doubtful it is either negligence, or something they will get sued over. I don't even quite understand what you think they will get sued over. Most of the policing would happen by courts and bar associations.
- schmookeeg 2y ago
  As someone who spends a lot of time battling mis-use/over-use of PII constantly, I am adopting MTTL as a term of art. :D
  - hx8 2y ago
    MTTL really applies more to a startup portfolio, has long does the average startup from an incubator operate before it receives it's first lawsuit. For your situation, you really want to measure how many PII records you handle per lawsuit. That way you can accurately measure the lawsuit cost per record and compare it to revenue per record to see if you're profitable.
DannyBee 2y ago
Lawyer here - there are fields, and law is definitely one of them, where labor is the major cost. That labor is not often used sanely. It is common to use lawyers costing hundreds per hour to do fairly basic document review and summarization. That is, to produce a fairly simple artifact. Not legal research, not opinionated briefing. But literal: Read these documents, produce a summary of what they say. While I can't say this is the same as what you are talking about ("contracts review" means many things to many people), i'm not even the slightest bit surprised that AI is starting to replace remarkably inefficient uses of labor in law. I will add: Lots of funding being thrown at AI legal startups around on products that do document review and summarization, but that's not the big fish, and will be commodity very quickly. So i expect there will be an ebb and flow of these sorts of products as the startups either move on to things that enable them to capture a meaningful market (document review ain't it), or die and leave these companies hanging :)
- azinman2 2y ago
  But how do you know if AI is able to pick out the salient bits for summarization? Like some nasty point poison pill hidden in there? Wouldn’t you want an expert for such things?
  - chefandy 2y ago
    Your last statement hints at a big consideration: accountability. One lawyer on a formerly 15 lawyer staff is accountable for 15 lawyers worth of potential mistakes, and we know that “but the AI did it!” doesn’t hold water in law.
    - DannyBee 2y ago
      There's a bunch of assumptions here. The main one i think is probably wrong is that there was 15 lawyers worth of work being done before (when measured by some average lawyer standard). For example, it's possible there was only really 1 lawyer worth of work being split 15 ways, so each lawyer was really only responsible for 1/15th of an average lawyers amount of work :) In that scenario, they'd only be responsible for 1 average lawyers worth of mistakes now. Is that realistic? Who knows. I've definitely seen that level of "waste" (for lack of a better term) before in law firms :) Even in the scenario you are positing, it's not obvious it matters as much as you seem to think it does. If the per-lawyer mistake rate was low enough, it may be that 15x that rate simply does not matter. These kinds of contracts are fairly standardized, and so they are mostly looking at the differences from last time. Those differences are often not legal as much as factual. IE the table of costs changed, not the legal responsibility. So the main thing mistakes get you is maybe cost (if mistakes matter at all). This isn't like they are seeing brand new from scratch contracts constantly that require brand new analysis. Even if they were, like I said, the main issue with a mistake is cost. For all we know, the AI company also agreed to indemnify them for a certain rate of mistakes or something (which wouldn't be hard to get insurance for). I'm not actually a fan of AI taking necessary jobs, but I think the view here that this is sort of life or death is strange. I'd be much more worried about AI handling criminal defense in some semi-autonomous fashion than this.
seunosewa 2y ago
That's great. That means potentially slightly cheaper healthcare. A company shouldn't need so many lawyers unless it's a legal firm.

TheAceOfHearts 2y ago

I was pretty cynical trying earlier models, but with Gemini Flash 2.0 I felt like there was a pretty significant boost in usability and capabilities. In particular I've found that these tools make it a lot easier to explore or get started with unfamiliar domains. One of my big issues has often been decision paralysis, so having a tool to help me narrow down the list of resources and make it more approachable has been a huge win. My general experience has been that getting AI tools to directly do stuff for you tends to produce pretty bad results, but you can use it as a force multiplier for your own capabilities. If I'm confused or uncertain about how to do something, AI tools are usually pretty good at clarifying what needs to be done.

chefandy 2y ago
Maybe there’s a particular cognitive profile that benefits most from LLM chat bots? I’ve tried multiple times to realize this force multiplier in my life for everything from day-to-day stuff to picking up new things, using the latest paid bots with the best models, and I’ve persistently found them to be awkward, inaccurate, difficult to pin down into giving useful info rather than a bunch of non-committal pablum without hallucinations, etc etc etc
- j7ake 2y ago
  Maybe you can give some specific examples
  - chefandy 2y ago
    For what — things that AI wasn’t helpful for when I gave it a shot?
- anon373839 2y ago
  They’re useful for tasks that don’t require correctness: brainstorming, exploratory research, sketching out the vague shape of a solution to a problem. They’re mediocre to awful at consistently following instructions, unless you have the fortune of having a task and domain that are well represented in the post-training data. Yesterday I needed to generate ~500 filenames given a (human written) summary of each document’s contents. This seemed to be the perfect task to throw an LLM into a for loop. Yet it took three hours of prompt engineering to get a passable set of results - yes, I could’ve done it by hand in that time. Each iteration on the prompt revealed new blind spots, new ways in which the models would latch onto irrelevant details or gloss over the main point.

throw83288 2y ago

Cursor Composer is a game-changer for greenfield projects, it's definitely not 3% change.

DANmode 2y ago

Folks are busy optimizing their own building, not telling you how to optimize yours...

edanm 2y ago

> Maybe I can finally learn what I'm suppose to be using generative AI for to be more productive. So many things. It's a general-purpose "thing doer" in many situations where you otherwise wouldn't have one. Let me give a super-simple example - not a high-value one, but an example of obvious value IMO. Say for some reason, you have a screenshot of a bunch of text. Maybe you took a picture of a page from a book or something, idk. Now you want it in textual form. You can throw it in ChatGPT and ask it to give you the text, and a few seconds later you have the text. I'm not saying there are no other solutions for this - there are. You can look for some software to do OCR or something. But that's what makes ChatGPT or others general-purpose - they're a one-stop shop for a lot of different things, including small one-off tasks like this. I can name a dozen other one-off tasks that it helps me with. Again, not the most high-value things it helps me with (that'd be programming help), but an undeniable example of value, IMO. How would you do this without an LLM? (I personally would've just typed it up myself, probably.)

hx8 2y ago
> How would you do this without an LLM? (I personally would've just typed it up myself, probably.) This has been a feature of Apple Preview (default image program) for years and years. You can just highlight text and copy it from a jpeg or png.
- edanm 2y ago
  OK, first of all, really cool. I don't think I knew this! Thank you. But how about a next step that I use LLMs for - taking this text from a document and reformatting it as, say, bullet points. E.g. I literally had this in a Jupyter notebook: [some_var_name, some_other_name, ...] and a bunch of those, and I wanted them redone as bullet points. It was a bit more complicated than that but I'm simplifying for the example. I literally took a screenshot, through it in ChatGPT and got back the correctly-formatted list. There are other ways to solve something like this of course (normally I'd put it in VSCode and use multiple cursors or macros), but I don't think there's anything that can go from a screenshot and a one-sentence description, to having finished the task, all in a single tool (that can also solve 100 other problems that come up in my work similarly easily).