5 ms·

kdb+/q is another really good choice for dsv[1] and json[2]. You can certainly create single-file databases (if you really want to e.g. for exchange), but splay

by geocar 9y ago

kdb+/q is another really good choice for dsv[1] and json[2]. You can certainly create single-file databases (if you really want to e.g. for exchange), but splayed table[3] is faster so you'd usually do that. [1]: http://code.kx.com/q/ref/filenumbers/#load-csv http://code.kx.com/q/ref/filenumbers/#load-csv [2]: http://code.kx.com/q/ref/dotj/ http://code.kx.com/q/ref/dotj/ [3]: http://code.kx.com/q/cookbook/splayed-tables/ http://code.kx.com/q/cookbook/splayed-tables/

dancek 9y ago
The problem with that might be the licensing costs once you use it commercially (eg. at work). IIRC the license prices aren't public, but you're looking at over $10k in any case. I personally prefer J to K in the APL family of languages. They also have a relatively cheap database, Jd [1]. Individual licenses are $600. Still a bit too much for my data mangling needs. :) [1] http://code.jsoftware.com/wiki/Jd/Index http://code.jsoftware.com/wiki/Jd/Index
- geocar 9y ago
  $10k isn't a lot (assuming that's right; it could be). I mean, it's a lot if you're used to something like MySQL or Postgres-levels of quality, but I've seen quotes for Oracle being almost $50k per core. MS-SQL is something like $7k per core, and kdb+ is definitely a lot more useful to me than MS-SQL. There's also a per-core/minute pricing which might be useful.
  - dancek 9y ago
    Sure, kdb+ would probably be worth every penny even at $100k/year when it's the right tool for the job. I gather it's genuinely the best in-memory database for computing arrays of varying rank. But a lot of the use cases these other tools are good for are small tasks every now and then. I feel kdb+ is in a different category.
    - aplorbust 9y ago
      Anecdote: I frequently use kdb+ for small tasks. For me, its in the "all-purpose" category. The limitations are only in the ability I have to use it. For example, removing nonconsecutive, duplicate lines from a file, such as a CSV file: exec echo "k).Q.fs[l:0::\`:$1];l:?:l;\`:$1 0:l"|exec q >&2; where Q.fs is a function in a script thats bundled with the interpreter; the chunk size for reading the file into memory is adjustable by editing the function.
      - geocar 9y ago
        You can make it simpler: l:0;.Q.fs[{if[x~l;:];-1 l::x}each]`:input or if you have memory: -1 distinct read0`:input ...or if you want to use k: -1@?0:`:input
        aplorbust 9y ago
        Stupid question: With -1, how would I suppress the return value? Use a function? k)a:{-1@?0:`:input;};a[]
        geocar 9y ago
        It's not a stupid question. ; What this does is return generic-null :: which .Q.s doesn't print.

by geocar 9y ago

dancek 9y ago

The problem with that might be the licensing costs once you use it commercially (eg. at work). IIRC the license prices aren't public, but you're looking at over $10k in any case. I personally prefer J to K in the APL family of languages. They also have a relatively cheap database, Jd [1]. Individual licenses are $600. Still a bit too much for my data mangling needs. :) [1] http://code.jsoftware.com/wiki/Jd/Index http://code.jsoftware.com/wiki/Jd/Index

geocar 9y ago
$10k isn't a lot (assuming that's right; it could be). I mean, it's a lot if you're used to something like MySQL or Postgres-levels of quality, but I've seen quotes for Oracle being almost $50k per core. MS-SQL is something like $7k per core, and kdb+ is definitely a lot more useful to me than MS-SQL. There's also a per-core/minute pricing which might be useful.
- dancek 9y ago
  Sure, kdb+ would probably be worth every penny even at $100k/year when it's the right tool for the job. I gather it's genuinely the best in-memory database for computing arrays of varying rank. But a lot of the use cases these other tools are good for are small tasks every now and then. I feel kdb+ is in a different category.
  - aplorbust 9y ago
    Anecdote: I frequently use kdb+ for small tasks. For me, its in the "all-purpose" category. The limitations are only in the ability I have to use it. For example, removing nonconsecutive, duplicate lines from a file, such as a CSV file: exec echo "k).Q.fs[l:0::\`:$1];l:?:l;\`:$1 0:l"|exec q >&2; where Q.fs is a function in a script thats bundled with the interpreter; the chunk size for reading the file into memory is adjustable by editing the function.
    - geocar 9y ago
      You can make it simpler: l:0;.Q.fs[{if[x~l;:];-1 l::x}each]`:input or if you have memory: -1 distinct read0`:input ...or if you want to use k: -1@?0:`:input
      - aplorbust 9y ago
        Stupid question: With -1, how would I suppress the return value? Use a function? k)a:{-1@?0:`:input;};a[]
        geocar 9y ago
        It's not a stupid question. ; What this does is return generic-null :: which .Q.s doesn't print.