jph00

21 ms·

331.
by jph00 3y ago
I'm sure eventually it would, but we haven't gotten to that point yet in our training.
332.
by jph00 3y ago
I think our experiments actually don't show catastrophic forgetting! The accuracy does not decrease as loss gets worse -- it's simply getting over-confident. So I'm not even sure we're showing any problem to solve he
333.
by jph00 3y ago
Thank you for posting this to HN! :D I'm one of the authors of this post -- Johno & I found it really interesting looking into this curious issue of rapid memorization from LLMs. I've been working with neural nets for 30 years
334.
by jph00 3y ago
That's fine - you can just use SGDR (SGD with restarts). https://arxiv.org/abs/1608.03983
335.
by jph00 3y ago
I haven't seen any researcher that supports your contention. SGDR (SGD with restarts) has been shown to work well. https://arxiv.org/abs/1608.03983
336.
by jph00 3y ago
AFAIK re-warming it up and then gradually decreasing it again ought to work fine. Have you seen any research showing that it doesn't?
337.
by jph00 3y ago
I'm one of the recipients of the AI grants, to support my work at fast.ai. I'm extremely grateful to a16z for their support. Here's some additional details based on questions I see in the comments: My grant covered the purcha
338.
by jph00 3y ago
That'll be Joseph Spisak, Head of Generative AI Open Source at Meta AI.
339.
by jph00 3y ago
It feels more natural to you because of familiarity. However, if you've learned Iverson Bracket notation in math ( https://en.wikipedia.org/wiki/Iverson_bracket ) then the APL approach will probably feel more natura
340.
AI and Power: The Ethical Challenges of Automation, Centralization, and Scale(fast.ai)
2 points by jph00 3y ago|0 comments
341.
by jph00 3y ago
Joseph your work on word vectors was really the key foundation behind modern NLP. Ulmfit and everything else builds on top of that.
342.
by jph00 3y ago
That video shows examples of new models with one-shot performance better than old models with fine-tuning. It doesn't compare new models with fine-tuning to new models without fine-tuning.
343.
by jph00 3y ago
That tweet is about a specific type of fine tuning: instruction tuning and RLHF. It causes to get better at the thing that it's being fine tuned for, which is making users feel better about the answer. Whether that's actually desi
344.
by jph00 3y ago
I haven't seen any recent papers that show that fine-tuning is obsolete - I've only seen papers showing the opposite. I'd be very interested to see any papers that have demonstrated applications where fine-tuning is not effec
345.
by jph00 3y ago
> " The idea of fine-tuning has a strong research pedigree. The approach can be traced back to 2018, when two influential papers were published. " The article refers to the BERT and GPT papers as the source of the fine-tuning i
346.
by jph00 3y ago
The premise behind nbdev isn’t that DS software should be a different set of tools - it’s that we should all consider using a different set of tools, based on literate and exploratory programming principles. I build all my projects, inclu
347.
by jph00 3y ago
There isn’t a problem any more with notebooks and git because nbdev now provides a merge driver. So it’s fine to use notebooks with git now.
348.
by jph00 3y ago
No the answer was simply that git doesn’t come with a merge driver for json, only one for line oriented text. So nbdev provides one for ipynb json documents.
349.
by jph00 3y ago
Please read the paper. Or at least the blog post. It's really quite readable. They explain that they've done both retrieval and reconstruction, and have lots of pictures showing examples of each. https://medarc-ai.githu
350.
by jph00 3y ago
Yes there was. However this is a different paper, describing a different method, applied to a different dataset, with different results. As the abstract says, "In particular, MindEye can retrieve the exact original image even among hig
351.
by jph00 3y ago
I don't think "nothing more" is the right conclusion here. From https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8658706/ : "Long COVID-19 anosmia fatigue could result from damage to olfactory
352.
by jph00 3y ago
I think you might be missing the clear sarcasm in that HN comment...
353.
by jph00 3y ago
Yes much better.
354.
by jph00 3y ago
You're misunderstanding that paper. The statistical analysis simply shows that a confirmatory diagnostic test had no impact on the likelihood of having most Long COVID symptoms, suggesting that belief in having been infected is as accu
355.
by jph00 3y ago
The previous posts are to a twitter thread that's been taken down, and the preview of a post that requires a $1000 subscription. This post however is freely available (for now at least).
356.
by jph00 3y ago
No, a meme post from 2022 did not in fact reference papers posted in 2023. You must be thinking of some other post, or you're just making stuff up.
357.
by jph00 3y ago
Thank you for sharing this article here! I'm the author. Please let me know if anyone has questions about it.
358.
by jph00 3y ago
Yeah so many people shooting themselves in the foot by failing to take advantage of this powerful tool - as you say especially for stuff that you might have otherwise used an overly-complex bash script for.
359.
by jph00 3y ago
That might be true culturally, but it's not true technically. Once you know a bit of Perl (including some command line switches like -e, -n, -a, -l, -p, and -i) you will be able to do just as easily everything you used to do in awk and
360.
by jph00 3y ago
Yes. I created a course which uses implementing Stable Diffusion from scratch as the project, and goes through lots of architecture choices, hyperparam selection, and debugging. (But note that this isn't something that's fast or e

More ›