3 ms·

Data, if you have to format it in text, is best done using a regular grammar, which you can read with regular expressions into objects. You would then do the va

Data, if you have to format it in text, is best done using a regular grammar, which you can read with regular expressions into objects. You would then do the validations on the objects rather than on the string-formatted data.

jstimpfle 11y ago
Good point. But, are there practical tools for this? I still have to decide how I store the data in memory. For example, struct Model { int numPersons; int numAncestors; struct Person *person; struct Ancestor *ancestor; }; Or SOA instead of AOS? struct Model { int numPersons; int numAncestors; struct string *personname; struct string *persondesc; struct string *ancestorancestor; struct string *ancestordescendant; }; etc. Do you know tools which are in practice a better choice than the super boring ad-hoc code I described? Unless I'd have to define many of these types of formats the cost of going meta is just too high.
- vinceguidry 11y ago
  I use Ruby, so I can't help you re: tooling. But if you are parsing a regular grammar with regular expressions, how much tooling do you need? The regexp does all the work. All you have to do is ensure that your data format is sane, that your delimiters can never appear in the data. With stuff like your problem, I can never be sure the data is clean, so I always default to a format like CSV or JSON and use standard library tools. I would shudder to have to work with that kind of data in C / C++ for the very reason you're asking these questions in this thread. It's low-level code for a high-level task. You're bound to fuck it up eventually and not foresee an edge case. But it would depend on what you want to do with the data. If you're walking trees, then you're going to want to store them in memory as trees. Whichever memory structure would be the most efficient in the algorithm you've got planned. Personally, since it's relational, I wouldn't bother with file storage at all and just dump it into a relational database, even if it's just SQLite. That would give you the most flexibility.
  - jstimpfle 11y ago
    Extracting from a regex match with unknown number of captures is still a kludge, at least judging from my experience with python. Come on, what can you mess up splitting lines into fields? I'm not specific to C/C++. It's just often the easiest/most straightforward thing to do as opposed to popular opinion. C structs are a good tool for concise description of data inter-relations. (Ownership or mutability are totally orthogonal questions). Conversely, how do you check integrity of JSON inputs (meaning the tree as it was returned from a JSON parser library)? The odds are not. Much more work since you don't know at each step precisely what has to come. Yeah, sqlite3 is a popular option. But it's more an alternative answer to the question of in-memory storage. It's not particularly suited as a human-friendly data interchange format.
    - vinceguidry 11y ago
      I think using flat file human-readable serialization for relational data is the real kludge, but I guess that's just me. But unknown numbers of captures indicates a non-regular grammar. You should simplify the grammar so you don't need them. I'd have each line adhere to /^(\w+) (.*)$/, and use the first capture to determine which regex you use to read the rest of the line. But this will break if there are newlines in the input. Hopefully it will error out if it gets a bad line, and even if it does, it will pass bad data in that case. > Come on, what can you mess up splitting lines into fields? Really depends on how clean your inputs are. A short list of things that can mess you up: newlines in your data, encoding inconsistencies, delimiters in your data. You have to handle all these manually if you write your own parser or risk unclean data entering the boundaries of your system. If you're in full control of your input, sure, go nuts, but I'd prefer a more robust solution because you never know what you're going to want to feed into it in the future. A standard format may have inconsistencies in implementation, but the format itself bakes in the solution to these problems, and also tends to have somewhat-useful error messages when they break. > Yeah, sqlite3 is a popular option. But it's more an alternative answer to the question of in-memory storage. It's not particularly suited as a human-friendly data interchange format. My suggestion would be to treat these as separate problems and not expect one solution to provide both of them. Why do you need human-friendliness? To me, the usual way to provide that is with an application and not a data format. I could knock one of those up pretty quickly with Rails and an admin backend, I'm sure you'd have your own way of doing that that you're comfortable with. I will use command-line applications if Rails is too heavyweight, I do this for a lot of my utility apps.

jstimpfle 11y ago

Good point. But, are there practical tools for this? I still have to decide how I store the data in memory. For example, struct Model { int numPersons; int numAncestors; struct Person *person; struct Ancestor *ancestor; }; Or SOA instead of AOS? struct Model { int numPersons; int numAncestors; struct string *personname; struct string *persondesc; struct string *ancestorancestor; struct string *ancestordescendant; }; etc. Do you know tools which are in practice a better choice than the super boring ad-hoc code I described? Unless I'd have to define many of these types of formats the cost of going meta is just too high.

vinceguidry 11y ago
I use Ruby, so I can't help you re: tooling. But if you are parsing a regular grammar with regular expressions, how much tooling do you need? The regexp does all the work. All you have to do is ensure that your data format is sane, that your delimiters can never appear in the data. With stuff like your problem, I can never be sure the data is clean, so I always default to a format like CSV or JSON and use standard library tools. I would shudder to have to work with that kind of data in C / C++ for the very reason you're asking these questions in this thread. It's low-level code for a high-level task. You're bound to fuck it up eventually and not foresee an edge case. But it would depend on what you want to do with the data. If you're walking trees, then you're going to want to store them in memory as trees. Whichever memory structure would be the most efficient in the algorithm you've got planned. Personally, since it's relational, I wouldn't bother with file storage at all and just dump it into a relational database, even if it's just SQLite. That would give you the most flexibility.
- jstimpfle 11y ago
  Extracting from a regex match with unknown number of captures is still a kludge, at least judging from my experience with python. Come on, what can you mess up splitting lines into fields? I'm not specific to C/C++. It's just often the easiest/most straightforward thing to do as opposed to popular opinion. C structs are a good tool for concise description of data inter-relations. (Ownership or mutability are totally orthogonal questions). Conversely, how do you check integrity of JSON inputs (meaning the tree as it was returned from a JSON parser library)? The odds are not. Much more work since you don't know at each step precisely what has to come. Yeah, sqlite3 is a popular option. But it's more an alternative answer to the question of in-memory storage. It's not particularly suited as a human-friendly data interchange format.
  - vinceguidry 11y ago
    I think using flat file human-readable serialization for relational data is the real kludge, but I guess that's just me. But unknown numbers of captures indicates a non-regular grammar. You should simplify the grammar so you don't need them. I'd have each line adhere to /^(\w+) (.*)$/, and use the first capture to determine which regex you use to read the rest of the line. But this will break if there are newlines in the input. Hopefully it will error out if it gets a bad line, and even if it does, it will pass bad data in that case. > Come on, what can you mess up splitting lines into fields? Really depends on how clean your inputs are. A short list of things that can mess you up: newlines in your data, encoding inconsistencies, delimiters in your data. You have to handle all these manually if you write your own parser or risk unclean data entering the boundaries of your system. If you're in full control of your input, sure, go nuts, but I'd prefer a more robust solution because you never know what you're going to want to feed into it in the future. A standard format may have inconsistencies in implementation, but the format itself bakes in the solution to these problems, and also tends to have somewhat-useful error messages when they break. > Yeah, sqlite3 is a popular option. But it's more an alternative answer to the question of in-memory storage. It's not particularly suited as a human-friendly data interchange format. My suggestion would be to treat these as separate problems and not expect one solution to provide both of them. Why do you need human-friendliness? To me, the usual way to provide that is with an application and not a data format. I could knock one of those up pretty quickly with Rails and an admin backend, I'm sure you'd have your own way of doing that that you're comfortable with. I will use command-line applications if Rails is too heavyweight, I do this for a lot of my utility apps.