3 ms·
I presume this is due to tokenization or something, but there's a lot of extra whitespace in the code samples that make them look very unrealistic: def _ _ i
by jpatokal 8y ago
I presume this is due to tokenization or something, but there's a lot of extra whitespace in the code samples that make them look very unrealistic:
def _ _ init__(self, default):
" " "
See if the default value for the field on a view is
timespan.
" " "
< select >
< option > value < / option >
< option > value < / option >
< / select >
And indentation is also missing completely. Maybe you need to use another NN to guess which language the fake code is in and autoformat it accordingly!
- yeldarb 8y agoIt is, the tokenizer isn't reversible (and it adds spaces all over the place). But a lot of these I should be able to add to my regex that converts the output back into more human readable format (in the raw output, there's a space before every punctuation mark so I already remove those extraneous spaces from periods, commas, etc). I just haven't gotten around to adding in any heuristics specifically for code but adding a bit more post-processing is on my to-do list.
- yeldarb 8y agoI updated my regexes to clean up some of the tokenizer noise last night. So many of the formatting in the code snippets should look a bit more natural now.
- misterdoubt 8y agoFortunately, the problem minimally affects SQL. https://stackroboflow.com/#!/question/21712 https://stackroboflow.com/#!/question/21712 https://stackroboflow.com/#!/question/21690 https://stackroboflow.com/#!/question/21690 https://stackroboflow.com/#!/question/5959 https://stackroboflow.com/#!/question/5959 https://stackroboflow.com/#!/question/27941 https://stackroboflow.com/#!/question/27941 Looks about right.