3 ms·

Good point. But, are there practical tools for this? I still have to decide how I store the data in memory. For example, struct Model { int numPers

Good point. But, are there practical tools for this? I still have to decide how I store the data in memory. For example, struct Model { int numPersons; int numAncestors; struct Person *person; struct Ancestor *ancestor; }; Or SOA instead of AOS? struct Model { int numPersons; int numAncestors; struct string *personname; struct string *persondesc; struct string *ancestorancestor; struct string *ancestordescendant; }; etc. Do you know tools which are in practice a better choice than the super boring ad-hoc code I described? Unless I'd have to define many of these types of formats the cost of going meta is just too high.

vinceguidry 11y ago
I use Ruby, so I can't help you re: tooling. But if you are parsing a regular grammar with regular expressions, how much tooling do you need? The regexp does all the work. All you have to do is ensure that your data format is sane, that your delimiters can never appear in the data. With stuff like your problem, I can never be sure the data is clean, so I always default to a format like CSV or JSON and use standard library tools. I would shudder to have to work with that kind of data in C / C++ for the very reason you're asking these questions in this thread. It's low-level code for a high-level task. You're bound to fuck it up eventually and not foresee an edge case. But it would depend on what you want to do with the data. If you're walking trees, then you're going to want to store them in memory as trees. Whichever memory structure would be the most efficient in the algorithm you've got planned. Personally, since it's relational, I wouldn't bother with file storage at all and just dump it into a relational database, even if it's just SQLite. That would give you the most flexibility.
- jstimpfle 11y ago
  Extracting from a regex match with unknown number of captures is still a kludge, at least judging from my experience with python. Come on, what can you mess up splitting lines into fields? I'm not specific to C/C++. It's just often the easiest/most straightforward thing to do as opposed to popular opinion. C structs are a good tool for concise description of data inter-relations. (Ownership or mutability are totally orthogonal questions). Conversely, how do you check integrity of JSON inputs (meaning the tree as it was returned from a JSON parser library)? The odds are not. Much more work since you don't know at each step precisely what has to come. Yeah, sqlite3 is a popular option. But it's more an alternative answer to the question of in-memory storage. It's not particularly suited as a human-friendly data interchange format.
  - vinceguidry 11y ago
    I think using flat file human-readable serialization for relational data is the real kludge, but I guess that's just me. But unknown numbers of captures indicates a non-regular grammar. You should simplify the grammar so you don't need them. I'd have each line adhere to /^(\w+) (.*)$/, and use the first capture to determine which regex you use to read the rest of the line. But this will break if there are newlines in the input. Hopefully it will error out if it gets a bad line, and even if it does, it will pass bad data in that case. > Come on, what can you mess up splitting lines into fields? Really depends on how clean your inputs are. A short list of things that can mess you up: newlines in your data, encoding inconsistencies, delimiters in your data. You have to handle all these manually if you write your own parser or risk unclean data entering the boundaries of your system. If you're in full control of your input, sure, go nuts, but I'd prefer a more robust solution because you never know what you're going to want to feed into it in the future. A standard format may have inconsistencies in implementation, but the format itself bakes in the solution to these problems, and also tends to have somewhat-useful error messages when they break. > Yeah, sqlite3 is a popular option. But it's more an alternative answer to the question of in-memory storage. It's not particularly suited as a human-friendly data interchange format. My suggestion would be to treat these as separate problems and not expect one solution to provide both of them. Why do you need human-friendliness? To me, the usual way to provide that is with an application and not a data format. I could knock one of those up pretty quickly with Rails and an admin backend, I'm sure you'd have your own way of doing that that you're comfortable with. I will use command-line applications if Rails is too heavyweight, I do this for a lot of my utility apps.
    - jstimpfle 11y ago
      I appreciate the discussion, and I'm sorry I have to respectfully disagree :) > I think using flat file human-readable serialization for relational data is the real kludge, but I guess that's just me. Look into your /etc and tell me which of the files in there are a kludge, and which ones are not plain text relational (or key/value, for that matter). > newlines in your data Well, what I do is I split at newlines :). Very practical since it guarantees fields don't contain them. (In the rare cases where you want them, do percent-encoding, hex encoding, C string style, whatever). > encoding inconsistencies Parse at an abstract character level of your choice. In scripting languages you get basic Unicode sanity for free. In C I would not want to deal with Unicode. It's too big for my head. UTF-8 goes out of its way so I can ignore it (newlines etc. still work). C strings means no longer having to deal with painful questions like "is c.isspace() only for U+0020 or for also other characters in Unicode classes which I never even heard of (or which don't even exist yet)?". ASCII/C locale text fits in my head. I don't want to be concerned with more abstraction (~ additional semantic baggage) throughout most of a system. > delimiters in your data same as above. > you never know what you're going to want to feed into it in the future. You better have a rough idea. I don't see why you couldn't commit to non-whitespace user-ids now, for example. Conversely, if you don't want to enforce sensible constraints now, you end up with code which never quite knows what it can assume (and often it can not assume properties that are needed for a sensible implementation). > To me, the usual way to provide that is with an application and not a data format. Can I email that application to someone else? Can I grep it as easily as my text files? Can I version-control it? Can I quickly whip up a script, import the application into Excel or maybe sqlite and visualize or extract some relationships? Will that application server still serve my data in 10 years? Will that application still build next year? How do I access it when I don't have a stable connection?

vinceguidry 11y ago

I use Ruby, so I can't help you re: tooling. But if you are parsing a regular grammar with regular expressions, how much tooling do you need? The regexp does all the work. All you have to do is ensure that your data format is sane, that your delimiters can never appear in the data. With stuff like your problem, I can never be sure the data is clean, so I always default to a format like CSV or JSON and use standard library tools. I would shudder to have to work with that kind of data in C / C++ for the very reason you're asking these questions in this thread. It's low-level code for a high-level task. You're bound to fuck it up eventually and not foresee an edge case. But it would depend on what you want to do with the data. If you're walking trees, then you're going to want to store them in memory as trees. Whichever memory structure would be the most efficient in the algorithm you've got planned. Personally, since it's relational, I wouldn't bother with file storage at all and just dump it into a relational database, even if it's just SQLite. That would give you the most flexibility.

jstimpfle 11y ago
Extracting from a regex match with unknown number of captures is still a kludge, at least judging from my experience with python. Come on, what can you mess up splitting lines into fields? I'm not specific to C/C++. It's just often the easiest/most straightforward thing to do as opposed to popular opinion. C structs are a good tool for concise description of data inter-relations. (Ownership or mutability are totally orthogonal questions). Conversely, how do you check integrity of JSON inputs (meaning the tree as it was returned from a JSON parser library)? The odds are not. Much more work since you don't know at each step precisely what has to come. Yeah, sqlite3 is a popular option. But it's more an alternative answer to the question of in-memory storage. It's not particularly suited as a human-friendly data interchange format.
- vinceguidry 11y ago
  I think using flat file human-readable serialization for relational data is the real kludge, but I guess that's just me. But unknown numbers of captures indicates a non-regular grammar. You should simplify the grammar so you don't need them. I'd have each line adhere to /^(\w+) (.*)$/, and use the first capture to determine which regex you use to read the rest of the line. But this will break if there are newlines in the input. Hopefully it will error out if it gets a bad line, and even if it does, it will pass bad data in that case. > Come on, what can you mess up splitting lines into fields? Really depends on how clean your inputs are. A short list of things that can mess you up: newlines in your data, encoding inconsistencies, delimiters in your data. You have to handle all these manually if you write your own parser or risk unclean data entering the boundaries of your system. If you're in full control of your input, sure, go nuts, but I'd prefer a more robust solution because you never know what you're going to want to feed into it in the future. A standard format may have inconsistencies in implementation, but the format itself bakes in the solution to these problems, and also tends to have somewhat-useful error messages when they break. > Yeah, sqlite3 is a popular option. But it's more an alternative answer to the question of in-memory storage. It's not particularly suited as a human-friendly data interchange format. My suggestion would be to treat these as separate problems and not expect one solution to provide both of them. Why do you need human-friendliness? To me, the usual way to provide that is with an application and not a data format. I could knock one of those up pretty quickly with Rails and an admin backend, I'm sure you'd have your own way of doing that that you're comfortable with. I will use command-line applications if Rails is too heavyweight, I do this for a lot of my utility apps.
  - jstimpfle 11y ago
    I appreciate the discussion, and I'm sorry I have to respectfully disagree :) > I think using flat file human-readable serialization for relational data is the real kludge, but I guess that's just me. Look into your /etc and tell me which of the files in there are a kludge, and which ones are not plain text relational (or key/value, for that matter). > newlines in your data Well, what I do is I split at newlines :). Very practical since it guarantees fields don't contain them. (In the rare cases where you want them, do percent-encoding, hex encoding, C string style, whatever). > encoding inconsistencies Parse at an abstract character level of your choice. In scripting languages you get basic Unicode sanity for free. In C I would not want to deal with Unicode. It's too big for my head. UTF-8 goes out of its way so I can ignore it (newlines etc. still work). C strings means no longer having to deal with painful questions like "is c.isspace() only for U+0020 or for also other characters in Unicode classes which I never even heard of (or which don't even exist yet)?". ASCII/C locale text fits in my head. I don't want to be concerned with more abstraction (~ additional semantic baggage) throughout most of a system. > delimiters in your data same as above. > you never know what you're going to want to feed into it in the future. You better have a rough idea. I don't see why you couldn't commit to non-whitespace user-ids now, for example. Conversely, if you don't want to enforce sensible constraints now, you end up with code which never quite knows what it can assume (and often it can not assume properties that are needed for a sensible implementation). > To me, the usual way to provide that is with an application and not a data format. Can I email that application to someone else? Can I grep it as easily as my text files? Can I version-control it? Can I quickly whip up a script, import the application into Excel or maybe sqlite and visualize or extract some relationships? Will that application server still serve my data in 10 years? Will that application still build next year? How do I access it when I don't have a stable connection?