4 ms·
Note that this model can't really be used for most code tasks. The tokenizer removed repeating spaces, so it doesn't have a valid concept of indentation.
by ozr 3y ago
Note that this model can't really be used for most code tasks. The tokenizer removed repeating spaces, so it doesn't have a valid concept of indentation.
- baq 3y agoSo, useless for the one thing LLMs are actually properly grounded? Sounds like a self-inflicted wound.
- laurentlb 3y agoIn most languages though, you can reformat the code and get the indentation back.
- mk_stjames 3y agoI was going to try and be snarky and make a comment about the superior use of tabs, but I just ran samples into the tokenizer and it doesn't recognize tabs either. :-(
- TeMPOraL 3y agoThere's still space for snark - one might observe that the model is just, wisely, excluding Python.
- deleted 3y ago[deleted]
- searealist 3y agoSeems trivial to autoformat afterwards?
- teaearlgraycold 3y agoExcept with Python lol
- 2-718-281-828 3y agonot every language is indentation based ...
- ozr 3y agoNot strictly indentation, any sort of repeating spaces. That expands the blast radius of the mistake a bit.
- 2-718-281-828 3y agowhich other language does syntactically use repeating spaces? i can only think of print statements where sth like that would be found.
- hungrigekatze 3y agoFor some discussion on how to have the LLaMa tokenizer (properly) handle repeating spaces, please see this discussion: https://github.com/openlm-research/open_llama/issues/40 https://github.com/openlm-research/open_llama/issues/40
- seydor 3y agoUff. Did they know that when they were training it?