4 ms·

I'm surprised they didn't give a justification as to why they couldn't just adopt Parquet[0]. [0] https://parquet.apache.org/ https://parquet.apache.org/

fhoffa 11y ago
BigQuery is faster than anything else I've seen. Why switch to what everyone else is using then? (See http://tech.marksblogg.com/billion-nyc-taxi-rides-bigquery.html http://tech.marksblogg.com/billion-nyc-taxi-rides-bigquery.h... vs all other benchmarks for the same dataset Mark got) Disclaimer: I'm Felipe Hoffa and I work for Google (https://twitter.com/felipehoffa https://twitter.com/felipehoffa). But you can try BigQuery in the next 5 minutes and check the speed claims :).
- michaelmior 11y ago
  Thanks for the reply :) I'll have to play around with BigQuery and see for myself! > BigQuery is faster than anything else I've seen. > Why switch to what everyone else is using then? This really depends on how much faster. For a marginal drop in performance, many would think it's worthwhile to stick with an established format. That said, I'm willing to believe the performance delta for BigQuery is worth it :)
obulpathi 11y ago
Parquet is the Open Source implementation of Big Query's first generation columnar storage format. It is based on record shredding and assembly techniques described in Dremel paper (Google's internal name for Big Query). What they are talking here is the next generation (second generation) storage format.
int_handler 11y ago
It is clearly stated in the article: When Google published the Dremel paper in 2010, it explained how this structure is preserved within column store. ... The definition and repetition levels encoding is so efficient for semistructured data that other open source columnar formats, such as Parquet, also adopted this technique. Parquet is an open-source reimplementation of the columnar storage format described in Google's 2010 Dremel paper. Capacitor is Google's next-generation columnar storage format.
- michaelmior 11y ago
  That answer isn't very clear to me, but thanks for pulling the relevant bits.
estefan 11y ago
Not sure why there are all the down votes (if OP is asking why they don't support parquet as an input format). If Google support avro, why not parquet? JSON isn't exactly efficient if it's about efficiency. We dump the output of spark jobs into BQ for exploration and having to produce JSON in addition to parquet is an irritating (and expensive) overhead.
- pacala 11y ago
  The post is about the [adaptive] format BQ uses internally for fast queries. Your problem is having BQ ingesting data in Parquet format. While BQ should support Parquet ingestion, it should also rewrite it on-the-fly into whatever format brings faster query performance.
  - estefan 11y ago
    I agree.

fhoffa 11y ago

BigQuery is faster than anything else I've seen. Why switch to what everyone else is using then? (See http://tech.marksblogg.com/billion-nyc-taxi-rides-bigquery.html http://tech.marksblogg.com/billion-nyc-taxi-rides-bigquery.h... vs all other benchmarks for the same dataset Mark got) Disclaimer: I'm Felipe Hoffa and I work for Google (https://twitter.com/felipehoffa https://twitter.com/felipehoffa). But you can try BigQuery in the next 5 minutes and check the speed claims :).

michaelmior 11y ago
Thanks for the reply :) I'll have to play around with BigQuery and see for myself! > BigQuery is faster than anything else I've seen. > Why switch to what everyone else is using then? This really depends on how much faster. For a marginal drop in performance, many would think it's worthwhile to stick with an established format. That said, I'm willing to believe the performance delta for BigQuery is worth it :)

obulpathi 11y ago

Parquet is the Open Source implementation of Big Query's first generation columnar storage format. It is based on record shredding and assembly techniques described in Dremel paper (Google's internal name for Big Query). What they are talking here is the next generation (second generation) storage format.

int_handler 11y ago

It is clearly stated in the article: When Google published the Dremel paper in 2010, it explained how this structure is preserved within column store. ... The definition and repetition levels encoding is so efficient for semistructured data that other open source columnar formats, such as Parquet, also adopted this technique. Parquet is an open-source reimplementation of the columnar storage format described in Google's 2010 Dremel paper. Capacitor is Google's next-generation columnar storage format.

michaelmior 11y ago
That answer isn't very clear to me, but thanks for pulling the relevant bits.

estefan 11y ago

Not sure why there are all the down votes (if OP is asking why they don't support parquet as an input format). If Google support avro, why not parquet? JSON isn't exactly efficient if it's about efficiency. We dump the output of spark jobs into BQ for exploration and having to produce JSON in addition to parquet is an irritating (and expensive) overhead.

pacala 11y ago
The post is about the [adaptive] format BQ uses internally for fast queries. Your problem is having BQ ingesting data in Parquet format. While BQ should support Parquet ingestion, it should also rewrite it on-the-fly into whatever format brings faster query performance.
- estefan 11y ago
  I agree.