4 ms·

I'm happy to see the stdin -> spyql -> stdout model. Plain text is great :) Columnar storage and processing could give you some performance benefits. SPyQL see

• dmoura 5y ago
  Yes, column storage is great, specially when you only need a small subset of columns as there much less reading operations. I have experience working with Parquet files, so I can see the benefits of CTFs, even if we don't take into account summary metadata and so forth. In its current version, SPyQL only reads from the stdin so it could only read a single column (a single file), unless you had an util merging the required columns beforehand (which could very well eliminate any performance gains of CTFs). I am planning to support reading from files/directories (as an argument) and in that case it's a matter of extending the processor class to deal with CTFs (column names = file names, etc). For taking full advantage of CTFs we would need to only open/read the required columns/files, but that seems very doable. How often with CTFs would you have your column files (individually) compressed? What would be the most popular compression formats? Thanks! BTW, nice R lib, I also love R :-)