3 ms·
So these ingested files are JSON or something semi-structured? I saw how the table was created, so maybe is was doing the schema-less key-value pair thing on ta
by jnordwick 10y ago
So these ingested files are JSON or something semi-structured? I saw how the table was created, so maybe is was doing the schema-less key-value pair thing on table creation and running queries against that? I'm sure this is all in the docs somewhere, but while I have a lot of big db experience, I have about 2 hours of AWS.
- sunsu 10y agoThe files are anything separated by line. Log files, CSV exports, anything. No specific format required.
- jnordwick 10y agoSo this is like a distributed awk, reparsing on every query?
- makmanalp 10y agoSo in this example the image says "lineitem_1t_parq" which makes me think that's probably parquet files (optimized and perhaps compressed columnar file format). But it does the same stuff with CSVs too, perhaps a bit less efficiently. JSON is also definitely possible, but a bit more tricky because of the inherently nested structure and more complex format. Take a look at this for the details on a similar idea: http://stratos.seas.harvard.edu/files/stratos/files/nodb-cacm.pdf http://stratos.seas.harvard.edu/files/stratos/files/nodb-cac...
- kakoni 10y agoThat paper discusses PostgresRAW, which can be found here; https://github.com/HBPMedical/PostgresRAW https://github.com/HBPMedical/PostgresRAW I wonder if there are any similar projects using psql as an engine?
- makmanalp 10y agoI think it's pretty common - it's open source, nice and modular codebase, reasonably advanced execution engine with stuff like MVCC. But also redshift itself, citus, hadapt and greenplum are examples that come to my mind.