zwaps

18 ms·

241.
by zwaps 3y ago
Sad that they also refuse to offer up any technical information. I am a bit salty given that all these companies basically use 90% OpenSource data, public research and most likely copy a good bit of their ideas from public repos. Alas, such
242.
by zwaps 3y ago
Super helpful! Do you have some more info on these issues and where this is discussed? Besides following y'all at explosion, any tipps of whom to follow so I don't get blindsided?
243.
by zwaps 3y ago
I bought it at some point thinking it was about applying Game Theory to life. This is not that.
244.
by zwaps 3y ago
I wish you luck then. Right now, you need CUDA to run things like bf16 or quantization, which are truly enablers in this space. So for now, I don't see AMD getting any traction. And apparently, the quality of the drivers has not been i
245.
by zwaps 3y ago
I mean the 35GB version of Falcon is maybe not something you'd want to use in production Also ironically, this version of Falcon will require CUDA.
246.
by zwaps 3y ago
Yeah but you see, we have been here before. "This time we are serious about ML/AI..." And if you went and bought an AMD card then, you'd have been wrong. My example about LLMs was just to show that AMD is simply not part
247.
by zwaps 3y ago
ROCm is so terrible the cloud providers rolled out their own chips rather than use AMD which has perfectly good GPUs and the worst software stack ever.
248.
by zwaps 3y ago
If you want to go an run inference/train on one of these fresh and sweet LLMs using bitsandbytes/PEFT, which is really where the excitement is at, you gotta use CUDA pretty much. This is the story now. And the story for the innova
249.
by zwaps 3y ago
Interesting low-ish means for Software Engineers for the FAANGs. Weren't they all supposed to earn 250k minimum or something?
250.
by zwaps 3y ago
But there are virtually no games on Mac. They recommed on their very website to play "Apple Arcade" games. Seriously?
251.
by zwaps 3y ago
Yes. I had to save string data in a column, which is frankly not something you want to do. HDF5 is super weird about encodings, quietly introduces errors, requires a fixed character length for that column and if you want to quickly access o
252.
by zwaps 3y ago
I really wonder how they made something like the windows2k interface in the 90's. Compared to that, today's interfaces are garbage. And this is especially true for the web or anything that is even remotely in contact with Javascri
253.
by zwaps 3y ago
The risk of marriage is what makes the commitment meaningful. Of course, you can always substitute for marriage with other forms of commitment in partnerships, marriage is just a particular choice. In your writing, however, it rather appear
254.
by zwaps 3y ago
I wish this was written with more care. None of the symbols are defined. Worst of all, they use "channel dimension" in a sequence model. What even is a channel in a sequence of tokens? This happens as soon as you have a single pe
255.
by zwaps 3y ago
You can still get a new Audi A/S/RS 4 without everything touchscreen and actual buttons everywhere. Up until end of this year, I think.
256.
by zwaps 3y ago
I literally can't and never could. And even though I type into my phone every day for years, it never improved. Most of the time some character entry will be wrong, except if I am super carefull. I guess I have sausage fingers? I can t
257.
by zwaps 3y ago
Worringly, I am it sure the people working on this really understand what a Transformer is Quote from them: “ There is still active research in non-transformer based language models though, such as Amazon’s AlexaTM 20B which outperforms GP
258.
by zwaps 3y ago
What makes you say it was a bad idea? Why do you think it didn't work out exactly as planned?
259.
by zwaps 3y ago
Also, if I am not reading this wrong, the paper shows that when Fine-tuning on a task, this additional Medical training (7 days on A100) essentially does nothing (benefit of maybe 1% over standard LLaMA). Zero shot comparisons are not shown
260.
by zwaps 3y ago
It said to be trained on 75 Billion tokens with 8 A100 in 7 days. However, in the paper they say that for each of the 5 epochs, they sample 512 tokens from each paper randomly. This would come down to an effective training size of 12 billio
261.
by zwaps 3y ago
I guess it connects your langchain bot to some API like e.g. slack?
262.
by zwaps 3y ago
Two reasons: They were only recently implemented in large production models and are not part of ye-standard-ML-coursera. And I mean, for half the papers claiming that their particular efficiency variant reduces O(n^2) to whatever without pe
263.
by zwaps 3y ago
The information might be formally lost for the given token, but remember that transformers train on huge amounts of data. The (absolute) positional encoding is an arbitrary but fixed bias (push into some direction). The word "cat"
264.
by zwaps 3y ago
You might be misunderstanding the use of sin-cos in early positional embeddings. The waves are shifted such that each position gets a unique positional encoding. These encodings are not relative in distance to other positions, they are at b
265.
by zwaps 3y ago
I agree that it's exciting to move toward a mathematical approach to characterize what can, and what can not be learned by a DNN! Maybe someone else has the knowledge to give you some more precise answers. As far as I understand we nee
266.
by zwaps 3y ago
The proposed use cases (folding, social networks) correspond to what was previously termed "Geometric Deep Learning". Here, things like Graph Neural Networks were understood in terms of (pairwise?) relations to be learned under th
267.
by zwaps 3y ago
I really wish that CS would switch to discussing related literature before (or with) contributions. The authors only cite themselves until the very end of the paper, and the whole time you are thinking: "Hasn't this been done befo
268.
by zwaps 3y ago
I am leading a similar initiative and I have also used Databricks for the preprocessing. Most interesting is what happens between the preprocessing and the model training - the hand-off to the cluster workers. I guess the efficient option i
269.
by zwaps 3y ago
Yeah this is how we used to do it :-)
270.
by zwaps 3y ago
To be fair, for 1999, this is an all around fantastic site.

More ›