6 ms·

GNU Recutils – Tools and libraries to access plain-text databases

89 points by networked 12y ago

Trombone12 12y ago
Doesn't mention utf-8, so I'll assume it dies as soon as I type something strange like an ö in my foolish attempt to categorize a thing outside of the US.
- onli 12y ago
  Why would you think that? If it is really old that may be possible, but everything halfway current and maintained should support UTF-8 by default by now, especially in the linux world. And in fact, it is working fine. I took the example from https://www.gnu.org/software/recutils/manual/recutils.html#A-Little-Example https://www.gnu.org/software/recutils/manual/recutils.html#A... and exchanged Emacs with Ämacs, and the selector works ~$ recsel -e "Title ~ '.*Ämacs.*'" -P Title books.rec GNU Ämacs Manual
- falcolas 12y ago
  This is a bit worrisome, but they're pretty explicit where they require ASCII, so I'm willing to give them the benefit of the doubt until proven otherwise. Interesting format, though, with some definite benefits over other text storage formats.
- networked 12y ago
  According to the manual [1] the format requires that field names match [a-zA-Z%][a-zA-Z0-9_]* but it allows any characters in the value other than an unescaped newline. For example, the command echo "\"country\",population\nÖsterreich,8.58" | csv2rec --strict | rec2csv works as expected. Somewhat worryingly, the format specification does not say anything about \0. I'd rather see use of \0 explicitly prohibited. I have found one place where the field name rule may not be enforced properly. It appears that csv2rec will use a noncompliant column name from a CSV file header as a key even in strict mode; however, rec2csv or relset will not accept the output it generates as input. I.e., echo "Österreich_population\n8.58" | csv2rec --strict | rec2csv blows up on rec2csv rather than on csv2rec (which is what I think should happen). [1] http://www.gnu.org/software/recutils/manual/recutils.html#The-Rec-Format http://www.gnu.org/software/recutils/manual/recutils.html#Th...
jordigh 12y ago
... is ... is the logo two turtles mating? I don't feel so bad about our GNU Octave logo now.
- icebraining 12y ago
  Yeap: http://www.gnu.org/software/recutils/faq.html#whyturtles http://www.gnu.org/software/recutils/faq.html#whyturtles
- pestaa 12y ago
  Two male turtles actually.
falcolas 12y ago
The only think I feel is missing is traditional SQL syntax. This makes it much harder to use as a drop-in replacement where someone would actually otherwise use the competitors mentioned on their homepage. I think its real competition is SQLLite and JSON / YAML, and it definitely has some benefits over the text formats, it can't compare with SQLLite.
- mhuffman 12y ago
  There are applications that let you query structured text with SQL. For example, https://github.com/dinedal/textql https://github.com/dinedal/textql there are others as well.
  - networked 12y ago
    There is a lot of tools of this kind [1] but nearly all of them work with CSV or other delimiter-separated data. What made recutils stand out for me is that it has tools to modify files in place, not just select data from them, and its file format is different. The rec format is interesting because it is a specification for what others have used before without one. It seems quite readable and does not suffer from the ",,,,,," problem of CSV. [1] Here's the list of ones I have run into so far: * https://github.com/harelba/q https://github.com/harelba/q * https://github.com/tobimensch/termsql https://github.com/tobimensch/termsql * https://github.com/samuel/squawk https://github.com/samuel/squawk * https://github.com/tjunier/sqawk https://github.com/tjunier/sqawk I've also written one: https://github.com/dbohdan/sqawk https://github.com/dbohdan/sqawk. It turned out I wasn't the only one to come up with the name.
- bootload 12y ago
  "The only think I feel is missing is traditional SQL syntax." Presumes everything is relational? I notice the mention of 'advanced database' technology joins, and then I notice this example: $ recsel -e "Dob >> '31 July 1994' && Dob << '01 August 1998'" -p Name,Email acquaintances.rec So yeah, I sort of agree, what is the standard unified search language? Is it as easy to learn, use as SQL? "can't compare with SQLLite." I'd agree with you on JSON/YAML but SQLLite does one thing each of these format/tools do not, converts "readable" text to binary. Text is good for smaller, less structured data. There maybe good reasons to not want it locked up. There is one area I didn't see in the docs that would make SQLLite to be really useful. A two-way bridge allowing an export/import of the recutils formats allowing both search via SQL, yet maintaining the text format data.
_delirium 12y ago
Oh, very nice. I've been using org-mode lists with ad-hoc "key: value" syntax inside of them for some kinds of stuff. This looks better structured, while still being friendlier to write than something like XML or YAML.
- Poiesis 12y ago
  a related tool for keeping track of what you've been up to is Brett Terpstra's Doing: http://brettterpstra.com/projects/doing/ http://brettterpstra.com/projects/doing/
binarylog 12y ago
Thank you for sharing! I just gave it a spin and wrote up a short tutorial here: http://binarylog.net/code/password-manager-recutils-tutorial.html http://binarylog.net/code/password-manager-recutils-tutorial... I foresee many more usecases for recutils in my day-to-day life.

89 points by networked 12y ago

Doesn't mention utf-8, so I'll assume it dies as soon as I type something strange like an ö in my foolish attempt to categorize a thing outside of the US.

onli 12y ago
Why would you think that? If it is really old that may be possible, but everything halfway current and maintained should support UTF-8 by default by now, especially in the linux world. And in fact, it is working fine. I took the example from https://www.gnu.org/software/recutils/manual/recutils.html#A-Little-Example https://www.gnu.org/software/recutils/manual/recutils.html#A... and exchanged Emacs with Ämacs, and the selector works ~$ recsel -e "Title ~ '.*Ämacs.*'" -P Title books.rec GNU Ämacs Manual
falcolas 12y ago
This is a bit worrisome, but they're pretty explicit where they require ASCII, so I'm willing to give them the benefit of the doubt until proven otherwise. Interesting format, though, with some definite benefits over other text storage formats.
networked 12y ago
According to the manual [1] the format requires that field names match [a-zA-Z%][a-zA-Z0-9_]* but it allows any characters in the value other than an unescaped newline. For example, the command echo "\"country\",population\nÖsterreich,8.58" | csv2rec --strict | rec2csv works as expected. Somewhat worryingly, the format specification does not say anything about \0. I'd rather see use of \0 explicitly prohibited. I have found one place where the field name rule may not be enforced properly. It appears that csv2rec will use a noncompliant column name from a CSV file header as a key even in strict mode; however, rec2csv or relset will not accept the output it generates as input. I.e., echo "Österreich_population\n8.58" | csv2rec --strict | rec2csv blows up on rec2csv rather than on csv2rec (which is what I think should happen). [1] http://www.gnu.org/software/recutils/manual/recutils.html#The-Rec-Format http://www.gnu.org/software/recutils/manual/recutils.html#Th...

jordigh 12y ago

... is ... is the logo two turtles mating? I don't feel so bad about our GNU Octave logo now.

icebraining 12y ago
Yeap: http://www.gnu.org/software/recutils/faq.html#whyturtles http://www.gnu.org/software/recutils/faq.html#whyturtles
pestaa 12y ago
Two male turtles actually.

falcolas 12y ago

The only think I feel is missing is traditional SQL syntax. This makes it much harder to use as a drop-in replacement where someone would actually otherwise use the competitors mentioned on their homepage. I think its real competition is SQLLite and JSON / YAML, and it definitely has some benefits over the text formats, it can't compare with SQLLite.

mhuffman 12y ago
There are applications that let you query structured text with SQL. For example, https://github.com/dinedal/textql https://github.com/dinedal/textql there are others as well.
- networked 12y ago
  There is a lot of tools of this kind [1] but nearly all of them work with CSV or other delimiter-separated data. What made recutils stand out for me is that it has tools to modify files in place, not just select data from them, and its file format is different. The rec format is interesting because it is a specification for what others have used before without one. It seems quite readable and does not suffer from the ",,,,,," problem of CSV. [1] Here's the list of ones I have run into so far: * https://github.com/harelba/q https://github.com/harelba/q * https://github.com/tobimensch/termsql https://github.com/tobimensch/termsql * https://github.com/samuel/squawk https://github.com/samuel/squawk * https://github.com/tjunier/sqawk https://github.com/tjunier/sqawk I've also written one: https://github.com/dbohdan/sqawk https://github.com/dbohdan/sqawk. It turned out I wasn't the only one to come up with the name.
bootload 12y ago
"The only think I feel is missing is traditional SQL syntax." Presumes everything is relational? I notice the mention of 'advanced database' technology joins, and then I notice this example: $ recsel -e "Dob >> '31 July 1994' && Dob << '01 August 1998'" -p Name,Email acquaintances.rec So yeah, I sort of agree, what is the standard unified search language? Is it as easy to learn, use as SQL? "can't compare with SQLLite." I'd agree with you on JSON/YAML but SQLLite does one thing each of these format/tools do not, converts "readable" text to binary. Text is good for smaller, less structured data. There maybe good reasons to not want it locked up. There is one area I didn't see in the docs that would make SQLLite to be really useful. A two-way bridge allowing an export/import of the recutils formats allowing both search via SQL, yet maintaining the text format data.

_delirium 12y ago

Oh, very nice. I've been using org-mode lists with ad-hoc "key: value" syntax inside of them for some kinds of stuff. This looks better structured, while still being friendlier to write than something like XML or YAML.

Poiesis 12y ago
a related tool for keeping track of what you've been up to is Brett Terpstra's Doing: http://brettterpstra.com/projects/doing/ http://brettterpstra.com/projects/doing/

binarylog 12y ago

Thank you for sharing! I just gave it a spin and wrote up a short tutorial here: http://binarylog.net/code/password-manager-recutils-tutorial.html http://binarylog.net/code/password-manager-recutils-tutorial... I foresee many more usecases for recutils in my day-to-day life.