p1esk

45 ms·

811.
by p1esk 3y ago
What do you do during sprint planning meetings if you don’t know your objectives?
812.
by p1esk 3y ago
In practice, you have one or two IT guys in office, and the rest remote (could be in India if the work is not classified).
813.
by p1esk 3y ago
You do realize the first ever decent AI has appeared less than a year ago? It’s available to anyone today for $20/mo, and it will get better and better every year. What we have now was considered pure science fiction just five years ag
814.
by p1esk 3y ago
Oh wow, you’re right. Though it seems that they are using very small weight group sizes: either 16 or 32 (fp16 scaling factor per group). In this paper it seems there’s no weights grouping, so it’s a bit apples to oranges.
815.
by p1esk 3y ago
Wow, this is really cool!
816.
by p1esk 3y ago
If you find three symbols per weight shocking, this paper should completely blow your mind: https://arxiv.org/abs/1803.03764 I admit it did shock me when it came out.
817.
by p1esk 3y ago
Are you sure? I was under impression that 3b quantization still results in a significant degradation. Which quantization method are you talking about?
818.
by p1esk 3y ago
I’ve been programming directly on a server since 2014, through remote ssh configured in my IDE (Atom —> Pycharm —> VSCode). That’s pretty much the only way I can do it because I’m writing multi-GPU PyTorch code.
819.
by p1esk 3y ago
Ternary networks have been used since 2015. There are hundreds of papers. They all require full QAT (training from scratch). Not sure why you’re shocked.
820.
by p1esk 3y ago
In this scenario some people will still be rich and powerful while the rest will be poor and powerless. Only now the elites don’t really need the poor, right? So what’s going to happen when the next Hitler decides to shrink the world popula
821.
by p1esk 3y ago
How is this going to work exactly?
822.
by p1esk 3y ago
If robots carry out all the work there will be no 400T in salaries
823.
by p1esk 3y ago
I’m 46. The first time I got drunk was when I was 28.
824.
by p1esk 3y ago
Well, sure, why not. That can be an experiment.
825.
by p1esk 3y ago
So why don’t you try drinking and see if it helps? What do you have to lose?
826.
by p1esk 3y ago
Opposite for me - I drink more and more every year.
827.
by p1esk 3y ago
Not gonna lie, I’d probably sign up for a clinical trial of this procedure, just to experience it. But the risk of something going wrong is significant.
828.
by p1esk 3y ago
The problem is I don’t know how much time it will take to make it work, or if it’s even possible to make it work for my specific situation. I’ve wasted enough of my time trying to debug AMD and Graphcore chips to fall into this trap again.
829.
by p1esk 3y ago
Models can be sensitive to the location of a needle in the haystack of its input block. But only if we use some sort of attention optimization. For the quadratic attention algo it shouldn’t matter where the needle is, right?
830.
by p1esk 3y ago
Yes, the CoT paper does provide some evidence that a more verbose prompt works better. Thank you for pointing me to it. Though I still don’t quite understand what is going on in the dummy tokens paper - what is “computation width” and why w
831.
by p1esk 3y ago
How large is it?
832.
by p1esk 3y ago
Yes, more tokens means doing more compute, that much is true. The question is whether this extra compute helps or hurts. This question is yet to be answered, as far as I know. I tend to make my GPT-4 questions quite verbose, hoping it helps
833.
by p1esk 3y ago
Let me put it this way: if there’s even a slightest issue with my Pytorch code (training or inference) running on a non Nvidia chip it will be an automatic no from me. More than that - if I simply suspect there will be any issues I will not
834.
by p1esk 3y ago
Custom chips have to be much better than Nvidia to become attractive. Being 2x faster won’t be enough, 5x faster might be. Assuming perfectly functioning software.
835.
by p1esk 3y ago
This paper is a great illustration of how little is understood about this question. They discovered that appending dummy tokens (ignored during both training and inference) improves performance somehow. Don’t confuse their guess as to why t
836.
by p1esk 3y ago
It’s not clear that more tokens are better.
837.
by p1esk 3y ago
Because it’s a better prompt. Works better for people too.
838.
by p1esk 3y ago
I think they went MoE purely because straight up scaling from 175B to 1.8T is just too expensive. But it’s still 10x scaling, right?
839.
by p1esk 3y ago
Can’t be worse than releasing Full Self Driving (beta) software and allowing it to control your car on public roads.
840.
by p1esk 3y ago
I don’t think that “think” is a wrong word here. I believe people are machines - more complicated than GPT4, but machines nevertheless. Soon GPT-N will become more complicated than any human, and it will be more capable, so we might start s

More ›