4 ms·

As impressive as AI is and what it can do. I've felt the same way in my last job. It was almost there; it was almost good enough such that we could almost ship

As impressive as AI is and what it can do. I've felt the same way in my last job. It was almost there; it was almost good enough such that we could almost ship the project but we didn't. GPT3.5 was not good enough, GPT4 seemed like it was good enough in terms of accuracy but it was too slow and we were getting throttled like crazy. I lost my job over it. When I was worried about AI taking my job, I didn't picture it like this.

jongjong 3y ago
Our use case was that we needed to consider information from thousands of pages of documents to answer customer questions. I let GPT decide where to look based on content tables of the documents (thankfully each section was not more than ~500 words) and only fetched information from the relevant sections to let GPT answer the user's question. GPT3.5 was fast enough to be usable but often inaccurate; it would ignore some information or get confused about simple mathematical comparisons in the user's questions like 'greater than versus less than'. GPT4 was good and potentially fast enough but the API throttling was too restrictive and it was not possible to feed it less information and still get a good, nuanced answer every time. Actually the trick to get GPT to find the information based on table of contents worked really well in narrowing down the information for the second-stage summary (to answer the user's question) but our use case nevertheless required a lot of information (often from multiple sections) with a lot of grey areas and nuance and the limited amount of throughout provided by GPT4 didn't cut it. It was frustrating that, aside from accuracy issues (which depended on the complexity of the question), it often seemed to work very well and fast with GPT3.5 but the exact same code did not meet usability expectations when we switched over to GPT4 due to slow speed (though the accuracy was awesome). I only spent 1 month and a half on this project but management didn't have the patience to wait for Open AI to remove GPT4 throttling and because they couldn't see it working (at the speed they wanted), they had no reason to believe me that my solution was designed correctly... That's startups I guess. Anyway I still think there is a lot of hope for this technology and I'm confident that the problem can and will be solved. It was just unfortunate timing for me. Being an early adopter isn't always a good thing.
- ta988 3y ago
  Start your own company then, that's a product people would use. (I'm making my own and agree with you, it works)
- BoorishBears 3y ago
  From what you said your solution wasn't designed correctly? You should definitely have been using BM25 and SBERT for something like this, you definitely should have been asking 3.5 for structured output and doing any math yourself. If these were answers from a fixed set of documents there's also a ton of pre-processing you should have been doing. - I recently helped a friend with a problem of identifying a certain form of language in EDGAR sourced contracts. He had someone who tried feeding in entire documents into LLMs to find these and the result was some 20 minute search time. I took a few minutes sitting with him to come up with a demo that used synthetic data and SBERT to process documents in less than 30 seconds. 99% of that was stuff you could do 2 years ago, the part the LLM helped with was creating buckets of synthetic data quickly, and even that could be done procedurally if you had more time.
  - jongjong 3y ago
    >> you definitely should have been asking 3.5 for structured output and doing any math yourself. I suspect that this solution would have required a lot more time given my lack of experience in these technologies and there are a lot of unknowns for that particular use case. For example, a user could ask to compare legalese from 30 different providers to find all those which met a certain conditions; even if you can narrow down the input to just a few sentences per provider, you still need to feed it 30 times that amount or else the AI has no way to compare all the providers. With my solution, GPT only saw a tiny highly relevant fraction of all the documents. It's possible that the solutions you mentioned would have helped cut our input size further but it's not clear by how much given the complexity of the questions and the data.
- tacticalDonut 3y ago
  Why didn't you go the retrieval augmented generation route with a vector store?
thfuran 3y ago
>It was almost there; it was almost good enough such that we could almost ship I feel like that a lot but we've been shipping for years.
maxlamb 3y ago
What do you mean you lost your job over it? All because GPT4 was too slow for your project?
- jongjong 3y ago
  Yes. This is the reality in many startups. I should probably have refused to work on this project and suggested some other task; that's what my colleague did; he only worked on the front end part. I took a risk, I misjudged the tool and failed. I wasn't even upset (I got a decent severance), I'd been let go over far more trivial things when I was a junior years ago... Like when the boss fired me the first monday after he met my girlfriend at the office christmas party because he thought she was out of my league. That's why I don't bring her (now wife) to any office events nowadays.

jongjong 3y ago

Our use case was that we needed to consider information from thousands of pages of documents to answer customer questions. I let GPT decide where to look based on content tables of the documents (thankfully each section was not more than ~500 words) and only fetched information from the relevant sections to let GPT answer the user's question. GPT3.5 was fast enough to be usable but often inaccurate; it would ignore some information or get confused about simple mathematical comparisons in the user's questions like 'greater than versus less than'. GPT4 was good and potentially fast enough but the API throttling was too restrictive and it was not possible to feed it less information and still get a good, nuanced answer every time. Actually the trick to get GPT to find the information based on table of contents worked really well in narrowing down the information for the second-stage summary (to answer the user's question) but our use case nevertheless required a lot of information (often from multiple sections) with a lot of grey areas and nuance and the limited amount of throughout provided by GPT4 didn't cut it. It was frustrating that, aside from accuracy issues (which depended on the complexity of the question), it often seemed to work very well and fast with GPT3.5 but the exact same code did not meet usability expectations when we switched over to GPT4 due to slow speed (though the accuracy was awesome). I only spent 1 month and a half on this project but management didn't have the patience to wait for Open AI to remove GPT4 throttling and because they couldn't see it working (at the speed they wanted), they had no reason to believe me that my solution was designed correctly... That's startups I guess. Anyway I still think there is a lot of hope for this technology and I'm confident that the problem can and will be solved. It was just unfortunate timing for me. Being an early adopter isn't always a good thing.

ta988 3y ago
Start your own company then, that's a product people would use. (I'm making my own and agree with you, it works)
BoorishBears 3y ago
From what you said your solution wasn't designed correctly? You should definitely have been using BM25 and SBERT for something like this, you definitely should have been asking 3.5 for structured output and doing any math yourself. If these were answers from a fixed set of documents there's also a ton of pre-processing you should have been doing. - I recently helped a friend with a problem of identifying a certain form of language in EDGAR sourced contracts. He had someone who tried feeding in entire documents into LLMs to find these and the result was some 20 minute search time. I took a few minutes sitting with him to come up with a demo that used synthetic data and SBERT to process documents in less than 30 seconds. 99% of that was stuff you could do 2 years ago, the part the LLM helped with was creating buckets of synthetic data quickly, and even that could be done procedurally if you had more time.
- jongjong 3y ago
  >> you definitely should have been asking 3.5 for structured output and doing any math yourself. I suspect that this solution would have required a lot more time given my lack of experience in these technologies and there are a lot of unknowns for that particular use case. For example, a user could ask to compare legalese from 30 different providers to find all those which met a certain conditions; even if you can narrow down the input to just a few sentences per provider, you still need to feed it 30 times that amount or else the AI has no way to compare all the providers. With my solution, GPT only saw a tiny highly relevant fraction of all the documents. It's possible that the solutions you mentioned would have helped cut our input size further but it's not clear by how much given the complexity of the questions and the data.
tacticalDonut 3y ago
Why didn't you go the retrieval augmented generation route with a vector store?

thfuran 3y ago

>It was almost there; it was almost good enough such that we could almost ship I feel like that a lot but we've been shipping for years.

maxlamb 3y ago

What do you mean you lost your job over it? All because GPT4 was too slow for your project?

jongjong 3y ago
Yes. This is the reality in many startups. I should probably have refused to work on this project and suggested some other task; that's what my colleague did; he only worked on the front end part. I took a risk, I misjudged the tool and failed. I wasn't even upset (I got a decent severance), I'd been let go over far more trivial things when I was a junior years ago... Like when the boss fired me the first monday after he met my girlfriend at the office christmas party because he thought she was out of my league. That's why I don't bring her (now wife) to any office events nowadays.