8 ms·
Aside from the sliding attention window, I don't see them mention any particular reasons for the incredible performance. I would like to see some tests for benc
by sp332 3y ago
Aside from the sliding attention window, I don't see them mention any particular reasons for the incredible performance. I would like to see some tests for benchmark contamination in the training set.
- rafaelero 3y agoYeah, they should have mentioned details about the dataset.
- brucethemoose2 3y agoThis ^ I am suspicious of contamination in every finetune I see, and very suspicious in a new foundational model like this. (For those reading and not following, "contamination" is training a model/finetune on the very test it will be tested on. Normally these known tests are specifically excluded from training datasets so the models can be properly evaluated, but throwing them in is an easy way to "cheat" and claim a model is better than it is. In a foundational model with a huge dataset, there's also a high probability that well-known evaluation questions snuck into the dataset by accident).
- Workaccount2 3y agoWe need an AI that can generate bespoke AI tests...
- brucethemoose2 3y agoThe existing models are technically bad candidates for this, as they will inevitably produce tests they can more effectively answer themselves.
- londons_explore 3y agoTheres also a good chance that simply people discussing the test questions and answers on forums like reddit sneaks into the dataset, even if the exact question is filtered out.
- brucethemoose2 3y agoYeah, this is exactly what I was thinking. Its supposed to be searched for and filtered out in the training datasets, but I can even see an active effort to weed them out missing some test Q/A pairs.
- loudmax 3y agoAgreed. It's great that they're model available under a permissive license. But without more information about the training data and methodology, it isn't exactly "open source".
- Tommstein 3y ago> But without more information about the training data and methodology, it isn't exactly "open source". Being or not being open source has exactly jack shit to do with that.
- falcor84 3y agoI actually would support that statement. An AI model is a software artifact generated as part of a complex "build" process. Without having access to the details of the process that generated the artifact, including the training data, it's no more open-source than a compiled binary.
- capableweb 3y agoI'd argue that it goes against the intent of open source very much. The idea behind OSS is that you're able to modify it yourself and then use it again from that point. With software, we enable this by making the source code public, and include instructions for how to build/run the project. Then I can achieve this. But with these "OSS" models, I cannot do this. I don't have the training data and I don't have the training workflow/setup they used for training the model. All they give me is the model itself. Similar to how "You can't see the source but here is a binary" wouldn't be called OSS, it feels slightly unfair to call LLM models being distributed this way OSS.
- computerex 3y agoWhilst not wrong, understand that having the weights be released explicitly under Apache is a hell of a lot better than the weights being released under a non open source license and commercially friendly license. At least people can legally use this for their solutions.
- pk-protect-ai 3y agoThey use some improvements on attention mechanisms. It is on the main page ... That is why 7B model requires 24GB VRAM, there might be increased amount of attention heads as well...
- sp332 3y agoThey clarified https://news.ycombinator.com/item?id=37677311 https://news.ycombinator.com/item?id=37677311 that it should only need 16GB, but the unoptimized runtime currently takes 24.