4 ms·

I work in AI for a living, my advice would be worry less about what's the latest and greatest and focus on solving your problem, and learning the things you're

I work in AI for a living, my advice would be worry less about what's the latest and greatest and focus on solving your problem, and learning the things you're interested in well. Precisely because "the rate of change in the AI space is super fast", there really isn't much of a point to keeping up, even if you're an academic researcher (for those you only need to keep up with your piece of the puzzle, and you likely know everyone in that space already). For example, I wasn't working in the NLP space for a few years. I kept an eye on what was getting mentions in various circles but basically ignored everything until I had a problem to solve. I work with LLMs everyday now, and honestly, even though I do understand them pretty deeply, there's no real need. Prior to the rise of LLMs I spent some time building LSTMs because I felt that I needed to understand them better. Lots of fun projects, but if I had skipped all that it wouldn't have really mattered at all. Even more dramatically, I was never particularly specialized in computer vision, but currently build things (for fun) with Staple Diffusion every night. I've spent a fair bit of time really understanding the underlying model well (still have much to learn) but, because the space moves so fast, it's not that big a deal that I didn't also spend nights building GANs. Even though Stable Diffusion is also perpetually changing, the community has largely stuck with 1.5 and is very focused on squeezing as much juice out of that as they can. Most important: the fundamentals have never changed. GPT4 still projects information into some highly non-linear latent space and samples from that. Diffusion models are probably the most novel thing happening now, but more so for their engineering (they're essentially 3 models all trained together with differentiable programming). If you really understand the fundamentals then catching up when you need to is fairly easy.

mbm 3y ago
Yes of course, but let’s assume the goal is to be conversational in what’s changing/new at the level of a Marc Andreesen, without actually having to be Marc Andreesen. I work in the space as well, but because I’m not in the Valley, find it more challenging to always be abreast of the changes. While it might not be essential for the actual product engineering to know what’s new, as a founder talking to customers it’s highly beneficial to be on top of things (depending of course on your vertical).
- PheonixPharts 3y ago
  > at the level of a Marc Andreesen [sic], without actually having to be Marc Andreesen [sic]. Oh, I guess we are in a AI bubble then. There are so many wonderful thinkers right now in the AI space to aspire to be like, but the only reason to aspire to be like Marc Andreessen is because you want AI to be the next crypto. Better find a good coat, since AI winter will be here sooner than I'd thought.
- predictabl3 3y ago
  I got at least one glance when the "ew what" came out audibly. What does this even mean, anyway, I've kept up with bleeding edge tech from a 3g connection miles from civilization. Do Marcs farts convey some AI-hype-prescience that filters into the SV air that you're missing out on? What a "producty" take too. Knowing the hype-y buzzwords is more important than your engineers knowing what is cutting edge? Idk, I'm not in sales, maybe I'm way off base.
JoshuaDavid 3y ago
Slightly off-topic, but I am just starting to learn about the field, and saw > GPT4 still projects information into some highly non-linear latent space and samples from that Do you have more information/know where I can find things to read about the "highly non-linear" part of that? I've been reading some stuff[1][2][3] about smaller models, and my impression has been that the latent space is shockingly linear for those ones. But counterexamples would be informative, especially if it's something that only happens with larger models. ----- [1] https://arxiv.org/abs/2209.15162 https://arxiv.org/abs/2209.15162 [2] https://www.neelnanda.io/mechanistic-interpretability/othello https://www.neelnanda.io/mechanistic-interpretability/othell... [3] https://arxiv.org/pdf/1610.01644.pdf https://arxiv.org/pdf/1610.01644.pdf
- jiggawatts 3y ago
  Hilariously, the best explanations for how AI models like GPT work I got from GPT-4 itself! There’s a lot of jargon and assumed shared knowledge in research papers that make them hard for a beginner to parse. I’ve found GPT can “translate” and explain in context. I know neither Python nor PyTorch, so I give it snippets and tell it to show me the Mathematica equivalent, which I can read.
- psyklic 3y ago
  The space must be non-linear, as a consequence of the non-linear activation functions. A neural net is just a big math equation, and deeply embedded throughout are non-linear transforms which necessarily make the entire transform non-linear. Just like the presence of 1/x makes an equation no longer linear (at least with rare exception!). Here is a deeper explanation/visualization: https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ The three references don't seem to conclude that the latent space is linear. The first seems to mention "linear" only since they add an additional linear layer to project an image encoding into a generative language model. So I'm not sure this applies here, since the encoding itself is already richly complex by the time it is mapped. The second and third are using a "linear probe" in an attempt to gain insights about each layer. This feeds the output of a given layer into a linear classifier that attempts to predict the correct output labels. This doesn't perform well in early layers, but it improves monotonically until the final layer is reached. The researchers conclude this happens entirely as a consequence of the final layer being a linear classifier. So, the features eventually must become linearly separable since that's what the network was trained to do. This doesn't conclude that each layer's feature space is linear. Instead, they are just using a linear projection to examine how "easily" the net at that layer can make correct predictions. Even if a layer's output is decently predictive in this way, the actual representation could still be richer and contain additional information.

mbm 3y ago

Yes of course, but let’s assume the goal is to be conversational in what’s changing/new at the level of a Marc Andreesen, without actually having to be Marc Andreesen. I work in the space as well, but because I’m not in the Valley, find it more challenging to always be abreast of the changes. While it might not be essential for the actual product engineering to know what’s new, as a founder talking to customers it’s highly beneficial to be on top of things (depending of course on your vertical).

PheonixPharts 3y ago
> at the level of a Marc Andreesen [sic], without actually having to be Marc Andreesen [sic]. Oh, I guess we are in a AI bubble then. There are so many wonderful thinkers right now in the AI space to aspire to be like, but the only reason to aspire to be like Marc Andreessen is because you want AI to be the next crypto. Better find a good coat, since AI winter will be here sooner than I'd thought.
predictabl3 3y ago
I got at least one glance when the "ew what" came out audibly. What does this even mean, anyway, I've kept up with bleeding edge tech from a 3g connection miles from civilization. Do Marcs farts convey some AI-hype-prescience that filters into the SV air that you're missing out on? What a "producty" take too. Knowing the hype-y buzzwords is more important than your engineers knowing what is cutting edge? Idk, I'm not in sales, maybe I'm way off base.

JoshuaDavid 3y ago

Slightly off-topic, but I am just starting to learn about the field, and saw > GPT4 still projects information into some highly non-linear latent space and samples from that Do you have more information/know where I can find things to read about the "highly non-linear" part of that? I've been reading some stuff[1][2][3] about smaller models, and my impression has been that the latent space is shockingly linear for those ones. But counterexamples would be informative, especially if it's something that only happens with larger models. ----- [1] https://arxiv.org/abs/2209.15162 https://arxiv.org/abs/2209.15162 [2] https://www.neelnanda.io/mechanistic-interpretability/othello https://www.neelnanda.io/mechanistic-interpretability/othell... [3] https://arxiv.org/pdf/1610.01644.pdf https://arxiv.org/pdf/1610.01644.pdf

jiggawatts 3y ago
Hilariously, the best explanations for how AI models like GPT work I got from GPT-4 itself! There’s a lot of jargon and assumed shared knowledge in research papers that make them hard for a beginner to parse. I’ve found GPT can “translate” and explain in context. I know neither Python nor PyTorch, so I give it snippets and tell it to show me the Mathematica equivalent, which I can read.
psyklic 3y ago
The space must be non-linear, as a consequence of the non-linear activation functions. A neural net is just a big math equation, and deeply embedded throughout are non-linear transforms which necessarily make the entire transform non-linear. Just like the presence of 1/x makes an equation no longer linear (at least with rare exception!). Here is a deeper explanation/visualization: https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ The three references don't seem to conclude that the latent space is linear. The first seems to mention "linear" only since they add an additional linear layer to project an image encoding into a generative language model. So I'm not sure this applies here, since the encoding itself is already richly complex by the time it is mapped. The second and third are using a "linear probe" in an attempt to gain insights about each layer. This feeds the output of a given layer into a linear classifier that attempts to predict the correct output labels. This doesn't perform well in early layers, but it improves monotonically until the final layer is reached. The researchers conclude this happens entirely as a consequence of the final layer being a linear classifier. So, the features eventually must become linearly separable since that's what the network was trained to do. This doesn't conclude that each layer's feature space is linear. Instead, they are just using a linear projection to examine how "easily" the net at that layer can make correct predictions. Even if a layer's output is decently predictive in this way, the actual representation could still be richer and contain additional information.