4 ms·

Fwiw, GPT o1 helped me figure out how a fairly complex use case of epub.js, an open-source library with pretty opaque documentation and relatively few public sa

by nfw2 2y ago

Fwiw, GPT o1 helped me figure out how a fairly complex use case of epub.js, an open-source library with pretty opaque documentation and relatively few public samples. It took a few back-and-forths to get to a working solution, but it did get there. It makes me wonder if the AI successfully found and digested obscure sources on the internet or was just better at making sense of the esoteric documentation than me. If the latter, perhaps the need for public samples will diminish.

tpoacher 2y ago
Did you then share this insight back onto a public channel?
kachapopopow 2y ago
Experienced the same thing with a library that has no documentation and takes advantage of c++23(latest) features.
- MichaelZuo 2y ago
  Same, I’m pretty convinced it does in fact do genuinely original reasoning in at least a few areas, after enough prompts and with enough prodding. But it takes so long and so much prompting skill to get to that point that the use cases seem limited.
TaylorAlexander 2y ago
Well Gemini completely hallucinated command line switches on a recent question I asked it about the program “john the ripper”. We absolutely need public sources of truth at the very least until we can build systems that actually reason based on a combination of first principles and experience, and even then we need sources of truth for experience. You simply cannot create solutions to new problems if your data gets too old to encompass the new subject matter. We have so systems which can adequately determine fact from fiction, and new human experiences will always need to be documented for machines to understand them.
- mptest 2y ago
  In my experience o1 is not comparable to any other llm experience. I have had multiple phd friends test it - it's what has turned them from stochastic parrot campers to possible believers and to be clear - as a layman, (in almost every field) I've recognized that llm's weren't up to the challenge of disavowing that notion from my phd friends up until o1 and never even tried, even though I've 'believed' since gpt 2
  - Tostino 2y ago
    I haven't found really any use case that o1 was better than 4o or 3.5 Sonnet that related to actual work. Any time I tried some of the more complex prompts I was working through something with Sonnet or 4o, o1 would totally miss important points and ignore a lot of the instructions while going really deep trying to figure out some relatively basic portions of the prompt. Seems fine for reasonably simple prompts, but gets caught up when things get deeper.
    - mptest 2y ago
      Yeah, I generally agree with that. Why I said it only moved them from stochastic parrot campers to "possible" believers - to clarify, the few I've had test it have all pretty much said "this feels like it could lead to real reasoning/productivity/advances/intelligence".
ilrwbwrkhv 2y ago
Curious about your complex use case of epub.js. What were you trying to do with it?
- nfw2 2y ago
  I'm building an e-reader app where "enhancement content" such as illustrations, context-approprate summaries, and group chat can be integrated into the reading experience. The way I am connecting external content to the epub is through an anchoring system -- sequences of words can be hashed to form unique ids that are referenced by the enhancement. Doing this lets me index the enhancement content in such a way that is format-independent and doesn't require modifying the underlying epub. The specific task o1 helped me with was determining what the text is visible at any given point in time. This text is then turned into hashes to pull the relevant enhancement content. Getting the current words on the page doesn't seem all that complex, but the epub.js API is pretty confusing. There are abstractions like "Rendition", "Location", "Contents", "Range", and it's not always intuitive which of these will provide the appropriate methods. I'm sure I would have figured it out eventually by looking at the API docs, but GPT probably saved me and hour or two of trial-and-error.
  - CptFribble 2y ago
    > group chat hmm having a sort of mini-forum-like experience tied to particular pages in a book seems like a fascinating idea! being able to discuss plot twists and such only once you've already gotten to that point? wow this seems like an amazing idea actually! any names yet? I'd love to check it out once it's done!
  - ilrwbwrkhv 2y ago
    Interesting. How did you come across this idea. And how long have you been working on it?
    - nfw2 2y ago
      I'm not sure when I first had the idea. I read a lot of mystery, but I'm often frustrated trying to remember all the details. A virtual notebook seems like it could help a lot for me personally. My mom and grandma also read books with a physical notebook handy, and seems like modern technology should make that unnecessary. I was laid off as part of the massive layoffs in 2022 and made a first pass at the project. I didn't use epubjs and instead wasted a lot of time making a custom rendering engine -- this was dumb. Eventually I got another job and paused the project. I started again in earnest a month or so ago using epubjs as the base.
rmbyrro 2y ago
I'm having a similar experience with o1. It's the only model that can identify causes of bugs for me. Perhaps it's already clever enough to be used in generating synthetic data about coding to train and improve less capable models. Even synthetic StackOverflow Q&A format.
inthebin 2y ago
To be honest, many times GPT 4o helps me understand poorly written emails by colleagues. I often find myself asking it "Did he means this when he wrote this?"... I'm a bit on the spectrum so if someone asks me vague questions or hallucinates words for things that don't exist, I have to verify with chatGPT to reaffirm that they are in fact just stupid.
- adam_hn 2y ago
  same here, sometimes my understanding of a message doesn't make sense, but when gpt4 gives me the same understanding, it feels like we can't both reach the same conclusion by mistake. of course, that's not true, but in most cases, it's good enough.

by nfw2 2y ago

tpoacher 2y ago

Did you then share this insight back onto a public channel?

kachapopopow 2y ago

Experienced the same thing with a library that has no documentation and takes advantage of c++23(latest) features.

MichaelZuo 2y ago
Same, I’m pretty convinced it does in fact do genuinely original reasoning in at least a few areas, after enough prompts and with enough prodding. But it takes so long and so much prompting skill to get to that point that the use cases seem limited.

TaylorAlexander 2y ago

Well Gemini completely hallucinated command line switches on a recent question I asked it about the program “john the ripper”. We absolutely need public sources of truth at the very least until we can build systems that actually reason based on a combination of first principles and experience, and even then we need sources of truth for experience. You simply cannot create solutions to new problems if your data gets too old to encompass the new subject matter. We have so systems which can adequately determine fact from fiction, and new human experiences will always need to be documented for machines to understand them.

mptest 2y ago
In my experience o1 is not comparable to any other llm experience. I have had multiple phd friends test it - it's what has turned them from stochastic parrot campers to possible believers and to be clear - as a layman, (in almost every field) I've recognized that llm's weren't up to the challenge of disavowing that notion from my phd friends up until o1 and never even tried, even though I've 'believed' since gpt 2
- Tostino 2y ago
  I haven't found really any use case that o1 was better than 4o or 3.5 Sonnet that related to actual work. Any time I tried some of the more complex prompts I was working through something with Sonnet or 4o, o1 would totally miss important points and ignore a lot of the instructions while going really deep trying to figure out some relatively basic portions of the prompt. Seems fine for reasonably simple prompts, but gets caught up when things get deeper.
  - mptest 2y ago
    Yeah, I generally agree with that. Why I said it only moved them from stochastic parrot campers to "possible" believers - to clarify, the few I've had test it have all pretty much said "this feels like it could lead to real reasoning/productivity/advances/intelligence".

ilrwbwrkhv 2y ago

Curious about your complex use case of epub.js. What were you trying to do with it?

nfw2 2y ago
I'm building an e-reader app where "enhancement content" such as illustrations, context-approprate summaries, and group chat can be integrated into the reading experience. The way I am connecting external content to the epub is through an anchoring system -- sequences of words can be hashed to form unique ids that are referenced by the enhancement. Doing this lets me index the enhancement content in such a way that is format-independent and doesn't require modifying the underlying epub. The specific task o1 helped me with was determining what the text is visible at any given point in time. This text is then turned into hashes to pull the relevant enhancement content. Getting the current words on the page doesn't seem all that complex, but the epub.js API is pretty confusing. There are abstractions like "Rendition", "Location", "Contents", "Range", and it's not always intuitive which of these will provide the appropriate methods. I'm sure I would have figured it out eventually by looking at the API docs, but GPT probably saved me and hour or two of trial-and-error.
- CptFribble 2y ago
  > group chat hmm having a sort of mini-forum-like experience tied to particular pages in a book seems like a fascinating idea! being able to discuss plot twists and such only once you've already gotten to that point? wow this seems like an amazing idea actually! any names yet? I'd love to check it out once it's done!
- ilrwbwrkhv 2y ago
  Interesting. How did you come across this idea. And how long have you been working on it?
  - nfw2 2y ago
    I'm not sure when I first had the idea. I read a lot of mystery, but I'm often frustrated trying to remember all the details. A virtual notebook seems like it could help a lot for me personally. My mom and grandma also read books with a physical notebook handy, and seems like modern technology should make that unnecessary. I was laid off as part of the massive layoffs in 2022 and made a first pass at the project. I didn't use epubjs and instead wasted a lot of time making a custom rendering engine -- this was dumb. Eventually I got another job and paused the project. I started again in earnest a month or so ago using epubjs as the base.

rmbyrro 2y ago

I'm having a similar experience with o1. It's the only model that can identify causes of bugs for me. Perhaps it's already clever enough to be used in generating synthetic data about coding to train and improve less capable models. Even synthetic StackOverflow Q&A format.

inthebin 2y ago

To be honest, many times GPT 4o helps me understand poorly written emails by colleagues. I often find myself asking it "Did he means this when he wrote this?"... I'm a bit on the spectrum so if someone asks me vague questions or hallucinates words for things that don't exist, I have to verify with chatGPT to reaffirm that they are in fact just stupid.

adam_hn 2y ago
same here, sometimes my understanding of a message doesn't make sense, but when gpt4 gives me the same understanding, it feels like we can't both reach the same conclusion by mistake. of course, that's not true, but in most cases, it's good enough.