espadrine

11 ms·

91.
by espadrine 3y ago
KataGo is an open-source algorithm derived from AlphaGo, but with a number of tricks so that it trained faster: https://katagotraining.org/ It likely surpasses AlphaGo, and just like Stockfish, it delivers a protocol that c
92.
by espadrine 3y ago
I have heard that some jobs were able to find a new home. The work that was needed for Simultaneous localization and mapping (SLAM) in the car, was adapted within the Vision Pro. But not all expertise translates easily.
93.
by espadrine 3y ago
Not recently. GPT-3 from 2020 requires even more RAM; the open-source BLOOM from 2022 did too. In my view, the main value of larger models is distillation (which we particularly witness, for instance, with how Claude Haiku matches release-d
94.
by espadrine 3y ago
Yes, they employed security terminology for something that was instead data pipeline contamination. As the saying goes, garbage in: garbage out. I don't mean to be harsh on them though: experimentation is useful, and it became a great
95.
by espadrine 3y ago
> 4chan basically found an “echo” type debug command or something like that That is certainly what Microsoft wanted people to think[0]: > a coordinated attack by a subset of people exploited a vulnerability in Tay. Realistically,
96.
by espadrine 3y ago
Here is an authoritative source[0]: > consent is not required [for] cookies that are strictly necessary to provide an online service that the person explicitly requested. e.g. […] when your customers use a shopping basket So shopping c
97.
by espadrine 3y ago
A number of AI companies have a naming/reproducibility issue. GPT4 Turbo, released last November, is a separate version that is much better than GPT-4 (winning 70% of human preferences in blind tests), released in March 2023. Claude 3
98.
Wordle Perfect Play with Generic Methods(espadrine.github.io)
2 points by espadrine 3y ago|0 comments
99.
by espadrine 3y ago
Another element is that Mamba required a very custom implementation down to custom fused kernels which I expect would need to be implemented in deepspeed or the equivalent library for a larger training run spanning thousands of GPUs.
100.
by espadrine 3y ago
It is disallowed by the executive order “on Addressing United States Investments in Certain National Security Technologies and Products in Countries of Concern”: https://www.whitehouse.gov/briefing-room/presidential-act
101.
by espadrine 3y ago
The part of MCTS that matters for solving it is the Bellman equation, which is a component of the Bellman-Ford shortest-path algorithm, so all of those methods are part of a continuum. Which one works best is pretty sensitive to the explora
102.
by espadrine 3y ago
As a data point, using the Huggingface Transformers 4-bit quantization yields reasonable results: https://twitter.com/espadrine/status/1760355758309298421
103.
by espadrine 3y ago
Do you have an example text in mind? You can use this playground to test it out: https://huggingface.co/spaces/Xenova/the-tokenizer-playgroun...
104.
by espadrine 3y ago
Hard to say definitively. Mistral’s token embeddings only account for <2% of the 7B parameters, while Gemma’s larger token vocabulary vampirized over 10%, leaving less space for the more important parts of the network. It is a somewhat s
105.
by espadrine 3y ago
It mostly means that there are tokens dedicated to rarer sequences of characters, even in foreign languages (note that Gemma is not intended to be good multilingually): “説明書” (instruction manual) has its own token, and so does “Nixon”, “آبا
106.
by espadrine 3y ago
Huh, indeed, that's what the config.json[0] says; the report[1] indicates “Feedforward hidden dims: 49152”. [0]: https://huggingface.co/google/gemma-7b-it/blob/main/config.j... [1]: https:/&#x2
107.
by espadrine 3y ago
It is already part of the release on Huggingface: https://huggingface.co/google/gemma-7b/blob/main/gemma-7b.gg... It is a pretty clean release! I had some 500 issues with Kaggle validating my license app
108.
by espadrine 3y ago
I notice a few divergences to common models: - The feedforward hidden size is 16x the d_model, unlike most models which are typically 4x; - The vocabulary size is 10x (256K vs. Mistral’s 32K); - The training token count is tripled (6T vs. L
109.
by espadrine 3y ago
There is no question in my mind that the transformer architecture will not stop to evolve. Already now, we are stretching the definition by calling current transformers that way; the 2017 transformer had an encoder block which is nearly alw
110.
by espadrine 3y ago
In part, yes! But also because what used to define it was the human-curated datasets: SL contained input/output pairs, while RL contained episodes with sporadic rewards. Nowadays, many datasets have different forms or are synthetic. DP
111.
by espadrine 3y ago
DPO is as close to RL as RLHF. The latter also uses the LLM as a reward model. I'm not a fan of the RL/SL dichotomy, because the line gets so foggy. If you squint, every loss is a negative reward, and every policy improvement a su
112.
by espadrine 3y ago
What will each party gain from the contract they signed? The first few sections left me with a lot of questions. From what I can tell, based on the later sections: > new experiences for Google Cloud customers to easily train and deploy
113.
by espadrine 3y ago
It is certainly interesting, and I would love to try it for my hobbyist use-cases. I don’t do much research at work, but a fair bit on the weekends. Are you filtering users however? I cannot sign up in a personal capacity with a GMail email
114.
by espadrine 3y ago
Demis Hassabis of Deepmind echoes a similar sentiment[0]: > I still think there are missing things with the current systems. […] I regard it a bit like the Industrial Revolution where there was all these amazing new ideas about energy a
115.
by espadrine 3y ago
> Mistral's latest just released model is well below GPT-3 out of the box The early information I see implies it is above. Mind you, that is mostly because GPT-3 was comparatively low: for instance its 5-shot MMLU score was 43.9%,
116.
by espadrine 3y ago
No. Without the inference code, the best we can have are guesses on its implementation, so the benchmark figures we can get could be quite wrong. It does seem better than Llama2-70B in my tests[0], which rely on the work done by Dmytro Dzhu
117.
by espadrine 3y ago
Once on llama.cpp, it will likely run on CPU with enough RAM, especially given that the GGUF mmap code only seems to use RAM for the parts of the weights that get used.
118.
by espadrine 3y ago
The latest update I see on that link is from 2023.11.21 on “Expanding Bard’s understanding of YouTube videos.” Is it possible that there are country-specific update pages? Edit: I found a Google page indicating that Gemini Pro is not access
119.
by espadrine 3y ago
Bard shows “PaLM2” in my answers, and it says “I can't create images yet so I'm not able to help you with that” when I ask it to do so, which Gemini ought to be able to since its transformer can output images. I don’t think Bard i
120.
by espadrine 3y ago
I understand the feeling. It may be caused by habit rather than objectivity, though. Those open-source AI hacks are undergoing early productization: while they were only research, their modularity mattered for experimentization, but as they

More ›