4 ms·

Disclosure: I work on Google Cloud. This is super awesome for customers. I am also beyond excited for all the open-source connectors to finally be simplified s

by boulos 6y ago

Disclosure: I work on Google Cloud. This is super awesome for customers. I am also beyond excited for all the open-source connectors to finally be simplified so that they don't have to deal with the "oh right, gotta be careful because of list consistency". It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop, and I hope to see the same across the S3-focused connectors. I'm now much more inclined to make my GCS FUSE rewrite in Rust also support S3 directly :).

basicneo 6y ago
> It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop Is this public?
- ceocoder 6y ago
  Yes, https://github.com/GoogleCloudDataproc/hadoop-connectors https://github.com/GoogleCloudDataproc/hadoop-connectors disclosure: I work at Google as well.
  - boulos 6y ago
    Right, https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v1.9.0 https://github.com/GoogleCloudDataproc/hadoop-connectors/rel... was apparently the release: > Delete metadata cache functionality because Cloud Storage has strong native list operation consistency already. If folks are actually interested in these connectors, I'd also recommend this blogpost from last year: https://cloud.google.com/blog/products/data-analytics/new-release-of-cloud-storage-connector-for-hadoop-improving-performance-throughput-and-more https://cloud.google.com/blog/products/data-analytics/new-re... because even with consistency, GCS and S3 still aren't filesystems :).
    - basicneo 6y ago
      Do you have a link to the commits that removed the code. It'd be good to see what sort of complexity this sort of strong consistency can make redundant.
      - deleted 6y ago
        [deleted]
      - boulos 6y ago
        (Phone reply, sorry for the brevity) Just comparing to the previous release: https://github.com/GoogleCloudDataproc/hadoop-connectors/compare/v1.8.1...v1.9.0 https://github.com/GoogleCloudDataproc/hadoop-connectors/com... there are some big deletions like in: https://github.com/GoogleCloudDataproc/hadoop-connectors/commit/6854ea8d395f738d0ceac59405d6f9db4db38d18 https://github.com/GoogleCloudDataproc/hadoop-connectors/com... and https://github.com/GoogleCloudDataproc/hadoop-connectors/commit/ec01f4915f414ac7072b595262fa2733aea23150 https://github.com/GoogleCloudDataproc/hadoop-connectors/com... Igor would know more :)
      - macksd 6y ago
        In the case of Hadoop's S3 connector, this could eliminate this entire directory, plus its tests, plus a bunch of hooks in the main code: https://github.com/apache/hadoop/tree/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard https://github.com/apache/hadoop/tree/trunk/hadoop-tools/had.... There's an argument in favor of keeping it in case other S3-compatible stores need it (though you'd still need DynamoDB or some equivalent) and because it makes metadata lookups so much faster than S3 scans, which helps with query planning performance. But I imagine even fewer people will take the trade-off now that Amazon S3 itself is consistent.
  - waynesonfire 6y ago
    thanks for the disclosure!!! nobody cares.
gopalv 6y ago
> It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop It's been a few years, but I lost customer data on gs:// due to listing consistency - create a bunch of files, list dir, rename them one by one to commit the dir into Hive - listing missed a file & the rename skipped the missing one. Put a breakpoint and the bug disappears, hit the rename and listing from the same JVM, the bug disappears. Consistency issues are hell. > It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop, and I hope to see the same across the S3-focused connectors. Yes! S3guard was a complete pain to secure, since it was built on top of dynamodb and since any client could be a writer for some file (any file), you had to give all clients write access to the entire dynamodb table, which was quite a security headache.
- boulos 6y ago
  If it's been a few years, it was likely before the migration to Spanner: https://cloud.google.com/blog/products/gcp/how-google-cloud-storage-offers-strongly-consistent-object-listing-thanks-to-spanner https://cloud.google.com/blog/products/gcp/how-google-cloud-... We posted that blog in early 2018, but stated: > Last year we migrated all of Cloud Storage metadata to Spanner, Google’s globally distributed and strongly consistent relational database. So maybe pre-2017?
  - gigatexal 6y ago
    Had no idea this happened. Very cool to see. We use both GCS and S3.
deleted 6y ago
[deleted]
ofek 6y ago
Is your rewrite available?
- boulos 6y ago
  Not yet, it’s too crappy :). I only get to it every once in a while and it shows. As an example, until last weekend, I would just hardcode an access token rather than doing an oauth dance. Luckily, tame-oauth [1] from the folks at Embark was reasonably easy to integrate with. I also got a little depressed that zargony/rust-fuse was stuck on an ancient version until I learned that Chris Berner from OpenAI had forked it earlier this year to modernize / update it [2]. [1] https://github.com/EmbarkStudios/tame-oauth https://github.com/EmbarkStudios/tame-oauth [2] https://github.com/cberner/fuser https://github.com/cberner/fuser
  - ofek 6y ago
    Well when it becomes production-ready, feel free to open a PR or issue to https://github.com/ofek/csi-gcs https://github.com/ofek/csi-gcs I'd prefer to use a memory-safe version :)

by boulos 6y ago

basicneo 6y ago

> It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop Is this public?

ceocoder 6y ago
Yes, https://github.com/GoogleCloudDataproc/hadoop-connectors https://github.com/GoogleCloudDataproc/hadoop-connectors disclosure: I work at Google as well.
- boulos 6y ago
  Right, https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v1.9.0 https://github.com/GoogleCloudDataproc/hadoop-connectors/rel... was apparently the release: > Delete metadata cache functionality because Cloud Storage has strong native list operation consistency already. If folks are actually interested in these connectors, I'd also recommend this blogpost from last year: https://cloud.google.com/blog/products/data-analytics/new-release-of-cloud-storage-connector-for-hadoop-improving-performance-throughput-and-more https://cloud.google.com/blog/products/data-analytics/new-re... because even with consistency, GCS and S3 still aren't filesystems :).
  - basicneo 6y ago
    Do you have a link to the commits that removed the code. It'd be good to see what sort of complexity this sort of strong consistency can make redundant.
    - deleted 6y ago
      [deleted]
    - boulos 6y ago
      (Phone reply, sorry for the brevity) Just comparing to the previous release: https://github.com/GoogleCloudDataproc/hadoop-connectors/compare/v1.8.1...v1.9.0 https://github.com/GoogleCloudDataproc/hadoop-connectors/com... there are some big deletions like in: https://github.com/GoogleCloudDataproc/hadoop-connectors/commit/6854ea8d395f738d0ceac59405d6f9db4db38d18 https://github.com/GoogleCloudDataproc/hadoop-connectors/com... and https://github.com/GoogleCloudDataproc/hadoop-connectors/commit/ec01f4915f414ac7072b595262fa2733aea23150 https://github.com/GoogleCloudDataproc/hadoop-connectors/com... Igor would know more :)
    - macksd 6y ago
      In the case of Hadoop's S3 connector, this could eliminate this entire directory, plus its tests, plus a bunch of hooks in the main code: https://github.com/apache/hadoop/tree/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard https://github.com/apache/hadoop/tree/trunk/hadoop-tools/had.... There's an argument in favor of keeping it in case other S3-compatible stores need it (though you'd still need DynamoDB or some equivalent) and because it makes metadata lookups so much faster than S3 scans, which helps with query planning performance. But I imagine even fewer people will take the trade-off now that Amazon S3 itself is consistent.
- waynesonfire 6y ago
  thanks for the disclosure!!! nobody cares.

gopalv 6y ago

> It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop It's been a few years, but I lost customer data on gs:// due to listing consistency - create a bunch of files, list dir, rename them one by one to commit the dir into Hive - listing missed a file & the rename skipped the missing one. Put a breakpoint and the bug disappears, hit the rename and listing from the same JVM, the bug disappears. Consistency issues are hell. > It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop, and I hope to see the same across the S3-focused connectors. Yes! S3guard was a complete pain to secure, since it was built on top of dynamodb and since any client could be a writer for some file (any file), you had to give all clients write access to the entire dynamodb table, which was quite a security headache.

boulos 6y ago
If it's been a few years, it was likely before the migration to Spanner: https://cloud.google.com/blog/products/gcp/how-google-cloud-storage-offers-strongly-consistent-object-listing-thanks-to-spanner https://cloud.google.com/blog/products/gcp/how-google-cloud-... We posted that blog in early 2018, but stated: > Last year we migrated all of Cloud Storage metadata to Spanner, Google’s globally distributed and strongly consistent relational database. So maybe pre-2017?
- gigatexal 6y ago
  Had no idea this happened. Very cool to see. We use both GCS and S3.

deleted 6y ago

[deleted]

ofek 6y ago

Is your rewrite available?

boulos 6y ago
Not yet, it’s too crappy :). I only get to it every once in a while and it shows. As an example, until last weekend, I would just hardcode an access token rather than doing an oauth dance. Luckily, tame-oauth [1] from the folks at Embark was reasonably easy to integrate with. I also got a little depressed that zargony/rust-fuse was stuck on an ancient version until I learned that Chris Berner from OpenAI had forked it earlier this year to modernize / update it [2]. [1] https://github.com/EmbarkStudios/tame-oauth https://github.com/EmbarkStudios/tame-oauth [2] https://github.com/cberner/fuser https://github.com/cberner/fuser
- ofek 6y ago
  Well when it becomes production-ready, feel free to open a PR or issue to https://github.com/ofek/csi-gcs https://github.com/ofek/csi-gcs I'd prefer to use a memory-safe version :)