3 ms·

I have this too, but on information extraction tasks, if the input document is too long or complex it will skip sections saying something along the lines of <

I have this too, but on information extraction tasks, if the input document is too long or complex it will skip sections saying something along the lines of < !-- List of products. Truncated for brevity -- > (instead of the list) < !-- Shipper details go here -- > (instead of the details) Very annoying when you want to extract everything and is hard to fix. Maybe they had some examples solved by abbreviation in the fine-tuning set and it damaged the model. It doesn't make sense to train this behaviour into GPT-4.

antonvs 3y ago
Using GPT models like that is unreliable anyway. I was talking to someone who was excited that he was able to use ChatGPT to convert a short tabular text listing of about 70 rows to a comma-separated list of the ids on each row, without having to e.g. import it into a spreadsheet or database. I pointed out that this couldn't be relied on, but he was skeptical. So I checked the output he pasted - there were only 69 ids. It had left out an id from the middle of the list, seemingly at random.
- phatskat 3y ago
  This boggles my mind. I know doing it by hand, or using a tool takes more time and effort, but if you’re going to automate it away _anyway_ you should be doing it in a reliable and retractable way. I had a vim session saved that had a command history with a handful of regex patterns to format copy/pasted bank statement pdfs converted to docx (don’t ask) that would format the data into a CSV. Load the session, paste the ugly data, and cycle through the command history. It was ugly but reliable, and after the initial time sink of writing it once, it became quite efficient and wasn’t made more so (say a VimScript function) because it worked well enough.
  - antonvs 3y ago
    Yeah, I know. The person in question is CTO of a well-funded Series B startup, and is highly technical. But he's very much a cowboy when it comes to coding practices etc. I think LLMs appeal to him because they hold out the promise of not having to deal with his own developers when he's trying to do something himself that he doesn't already know how to do.

antonvs 3y ago

Using GPT models like that is unreliable anyway. I was talking to someone who was excited that he was able to use ChatGPT to convert a short tabular text listing of about 70 rows to a comma-separated list of the ids on each row, without having to e.g. import it into a spreadsheet or database. I pointed out that this couldn't be relied on, but he was skeptical. So I checked the output he pasted - there were only 69 ids. It had left out an id from the middle of the list, seemingly at random.

phatskat 3y ago
This boggles my mind. I know doing it by hand, or using a tool takes more time and effort, but if you’re going to automate it away _anyway_ you should be doing it in a reliable and retractable way. I had a vim session saved that had a command history with a handful of regex patterns to format copy/pasted bank statement pdfs converted to docx (don’t ask) that would format the data into a CSV. Load the session, paste the ugly data, and cycle through the command history. It was ugly but reliable, and after the initial time sink of writing it once, it became quite efficient and wasn’t made more so (say a VimScript function) because it worked well enough.
- antonvs 3y ago
  Yeah, I know. The person in question is CTO of a well-funded Series B startup, and is highly technical. But he's very much a cowboy when it comes to coding practices etc. I think LLMs appeal to him because they hold out the promise of not having to deal with his own developers when he's trying to do something himself that he doesn't already know how to do.