3 ms·

Extracting from a regex match with unknown number of captures is still a kludge, at least judging from my experience with python. Come on, what can you mess up

Extracting from a regex match with unknown number of captures is still a kludge, at least judging from my experience with python. Come on, what can you mess up splitting lines into fields? I'm not specific to C/C++. It's just often the easiest/most straightforward thing to do as opposed to popular opinion. C structs are a good tool for concise description of data inter-relations. (Ownership or mutability are totally orthogonal questions). Conversely, how do you check integrity of JSON inputs (meaning the tree as it was returned from a JSON parser library)? The odds are not. Much more work since you don't know at each step precisely what has to come. Yeah, sqlite3 is a popular option. But it's more an alternative answer to the question of in-memory storage. It's not particularly suited as a human-friendly data interchange format.

vinceguidry 11y ago
I think using flat file human-readable serialization for relational data is the real kludge, but I guess that's just me. But unknown numbers of captures indicates a non-regular grammar. You should simplify the grammar so you don't need them. I'd have each line adhere to /^(\w+) (.*)$/, and use the first capture to determine which regex you use to read the rest of the line. But this will break if there are newlines in the input. Hopefully it will error out if it gets a bad line, and even if it does, it will pass bad data in that case. > Come on, what can you mess up splitting lines into fields? Really depends on how clean your inputs are. A short list of things that can mess you up: newlines in your data, encoding inconsistencies, delimiters in your data. You have to handle all these manually if you write your own parser or risk unclean data entering the boundaries of your system. If you're in full control of your input, sure, go nuts, but I'd prefer a more robust solution because you never know what you're going to want to feed into it in the future. A standard format may have inconsistencies in implementation, but the format itself bakes in the solution to these problems, and also tends to have somewhat-useful error messages when they break. > Yeah, sqlite3 is a popular option. But it's more an alternative answer to the question of in-memory storage. It's not particularly suited as a human-friendly data interchange format. My suggestion would be to treat these as separate problems and not expect one solution to provide both of them. Why do you need human-friendliness? To me, the usual way to provide that is with an application and not a data format. I could knock one of those up pretty quickly with Rails and an admin backend, I'm sure you'd have your own way of doing that that you're comfortable with. I will use command-line applications if Rails is too heavyweight, I do this for a lot of my utility apps.
- jstimpfle 11y ago
  I appreciate the discussion, and I'm sorry I have to respectfully disagree :) > I think using flat file human-readable serialization for relational data is the real kludge, but I guess that's just me. Look into your /etc and tell me which of the files in there are a kludge, and which ones are not plain text relational (or key/value, for that matter). > newlines in your data Well, what I do is I split at newlines :). Very practical since it guarantees fields don't contain them. (In the rare cases where you want them, do percent-encoding, hex encoding, C string style, whatever). > encoding inconsistencies Parse at an abstract character level of your choice. In scripting languages you get basic Unicode sanity for free. In C I would not want to deal with Unicode. It's too big for my head. UTF-8 goes out of its way so I can ignore it (newlines etc. still work). C strings means no longer having to deal with painful questions like "is c.isspace() only for U+0020 or for also other characters in Unicode classes which I never even heard of (or which don't even exist yet)?". ASCII/C locale text fits in my head. I don't want to be concerned with more abstraction (~ additional semantic baggage) throughout most of a system. > delimiters in your data same as above. > you never know what you're going to want to feed into it in the future. You better have a rough idea. I don't see why you couldn't commit to non-whitespace user-ids now, for example. Conversely, if you don't want to enforce sensible constraints now, you end up with code which never quite knows what it can assume (and often it can not assume properties that are needed for a sensible implementation). > To me, the usual way to provide that is with an application and not a data format. Can I email that application to someone else? Can I grep it as easily as my text files? Can I version-control it? Can I quickly whip up a script, import the application into Excel or maybe sqlite and visualize or extract some relationships? Will that application server still serve my data in 10 years? Will that application still build next year? How do I access it when I don't have a stable connection?

vinceguidry 11y ago

I think using flat file human-readable serialization for relational data is the real kludge, but I guess that's just me. But unknown numbers of captures indicates a non-regular grammar. You should simplify the grammar so you don't need them. I'd have each line adhere to /^(\w+) (.*)$/, and use the first capture to determine which regex you use to read the rest of the line. But this will break if there are newlines in the input. Hopefully it will error out if it gets a bad line, and even if it does, it will pass bad data in that case. > Come on, what can you mess up splitting lines into fields? Really depends on how clean your inputs are. A short list of things that can mess you up: newlines in your data, encoding inconsistencies, delimiters in your data. You have to handle all these manually if you write your own parser or risk unclean data entering the boundaries of your system. If you're in full control of your input, sure, go nuts, but I'd prefer a more robust solution because you never know what you're going to want to feed into it in the future. A standard format may have inconsistencies in implementation, but the format itself bakes in the solution to these problems, and also tends to have somewhat-useful error messages when they break. > Yeah, sqlite3 is a popular option. But it's more an alternative answer to the question of in-memory storage. It's not particularly suited as a human-friendly data interchange format. My suggestion would be to treat these as separate problems and not expect one solution to provide both of them. Why do you need human-friendliness? To me, the usual way to provide that is with an application and not a data format. I could knock one of those up pretty quickly with Rails and an admin backend, I'm sure you'd have your own way of doing that that you're comfortable with. I will use command-line applications if Rails is too heavyweight, I do this for a lot of my utility apps.

jstimpfle 11y ago
I appreciate the discussion, and I'm sorry I have to respectfully disagree :) > I think using flat file human-readable serialization for relational data is the real kludge, but I guess that's just me. Look into your /etc and tell me which of the files in there are a kludge, and which ones are not plain text relational (or key/value, for that matter). > newlines in your data Well, what I do is I split at newlines :). Very practical since it guarantees fields don't contain them. (In the rare cases where you want them, do percent-encoding, hex encoding, C string style, whatever). > encoding inconsistencies Parse at an abstract character level of your choice. In scripting languages you get basic Unicode sanity for free. In C I would not want to deal with Unicode. It's too big for my head. UTF-8 goes out of its way so I can ignore it (newlines etc. still work). C strings means no longer having to deal with painful questions like "is c.isspace() only for U+0020 or for also other characters in Unicode classes which I never even heard of (or which don't even exist yet)?". ASCII/C locale text fits in my head. I don't want to be concerned with more abstraction (~ additional semantic baggage) throughout most of a system. > delimiters in your data same as above. > you never know what you're going to want to feed into it in the future. You better have a rough idea. I don't see why you couldn't commit to non-whitespace user-ids now, for example. Conversely, if you don't want to enforce sensible constraints now, you end up with code which never quite knows what it can assume (and often it can not assume properties that are needed for a sensible implementation). > To me, the usual way to provide that is with an application and not a data format. Can I email that application to someone else? Can I grep it as easily as my text files? Can I version-control it? Can I quickly whip up a script, import the application into Excel or maybe sqlite and visualize or extract some relationships? Will that application server still serve my data in 10 years? Will that application still build next year? How do I access it when I don't have a stable connection?