2 ms·

> Right, and if there was a formal definition of CSV, that's the situation we'd be in. RFC 4180 isn't one. You're mixing things up. You're confusing "formal d

> Right, and if there was a formal definition of CSV, that's the situation we'd be in. RFC 4180 isn't one. You're mixing things up. You're confusing "formal definition", which rfc4180 is, with the concept of an official standard, which no RFC is. Yet, that doesn't stop the world from working based on RFCs such as this one. > It's a description of one dialect of CSV There you go. > FWIW, i wrote a JSON tokenizer in 235 lines, and it really should be shorter: I've browsed through your code and I have to say you did a good job. I see your coding style is very frugal with the linebreaks and in breaking up statements, and in a manner that doesn't affect readability that much. OTOH, your absence of comments should be taken care of. However, I should point out a couple of design flaws in your lexer. First, your lexer handles any invalid token by throwing exceptions. Not only is that bad form and a misuse of what exceptions actually are (a document with a parsing issue isn't an exceptional event), but any parser that uses your lexer is unable to catch any syntactical errors, provide informative error messages, and most importantly be able to recover from parsing issues. One standard technique is to support frequent lexial/syntactical errors in the parser and use them to handle parser errors. The other design flaw is that you failed to include length limits on your strings. That's a potential source of problems, including security issues. Another issue is that your lexer fails to parse JSON numbers, which is the most complex part of the lexer. Your lexer simply dumps any character matched by isNumber to a string, and then try to convert it to a number by calling Java's Double.parseDouble(). As you may know, JSON's definition of a number doesn't match JAVA's specification. If your lexer supported JSON's number format then it would easily double the number of states it already has. So, eventough your lexer is a couple of hunded loc shorted than my estimate, it is in fact missing basic and fundamental chunks required by a lexer for JSON. Once you add those in, you could recount your loc and see how it steers away from the 500 estimate I pointed out.

twic 10y ago
Good point about the numbers! Double::parseDouble accepts a superset of JSON numbers, so the risk here is that something which is not a valid JSON number will be accepted. That could be tackled by building up the string, checking it against a regular expression for JSON numbers, and then parsing it. It wouldn't add any states to the parser, and would take two lines of code. I would suggest that in a JSON parser, attempting to recover from parse errors, or supporting frequent errors, would be a mistake. JSON, unlike CSV, has a rigorous definition, and we wouldn't be doing the world any favours by tolerating deviations from it. I'm not sure what the security issues with long strings would be. Text gets accumulated in a StringBuilder, which will handle strings up to 2 GB and then throw an exception. I don't think there's any way to get into an incorrect state. Your other points relate to design decisions, and ones where i'm quite happy with the options i chose.

twic 10y ago

Good point about the numbers! Double::parseDouble accepts a superset of JSON numbers, so the risk here is that something which is not a valid JSON number will be accepted. That could be tackled by building up the string, checking it against a regular expression for JSON numbers, and then parsing it. It wouldn't add any states to the parser, and would take two lines of code. I would suggest that in a JSON parser, attempting to recover from parse errors, or supporting frequent errors, would be a mistake. JSON, unlike CSV, has a rigorous definition, and we wouldn't be doing the world any favours by tolerating deviations from it. I'm not sure what the security issues with long strings would be. Text gets accumulated in a StringBuilder, which will handle strings up to 2 GB and then throw an exception. I don't think there's any way to get into an incorrect state. Your other points relate to design decisions, and ones where i'm quite happy with the options i chose.