4 ms·
Do you have a link to the commits that removed the code. It'd be good to see what sort of complexity this sort of strong consistency can make redundant.
by basicneo 6y ago
Do you have a link to the commits that removed the code. It'd be good to see what sort of complexity this sort of strong consistency can make redundant.
- deleted 6y ago[deleted]
- boulos 6y ago(Phone reply, sorry for the brevity) Just comparing to the previous release: https://github.com/GoogleCloudDataproc/hadoop-connectors/compare/v1.8.1...v1.9.0 https://github.com/GoogleCloudDataproc/hadoop-connectors/com... there are some big deletions like in: https://github.com/GoogleCloudDataproc/hadoop-connectors/commit/6854ea8d395f738d0ceac59405d6f9db4db38d18 https://github.com/GoogleCloudDataproc/hadoop-connectors/com... and https://github.com/GoogleCloudDataproc/hadoop-connectors/commit/ec01f4915f414ac7072b595262fa2733aea23150 https://github.com/GoogleCloudDataproc/hadoop-connectors/com... Igor would know more :)
- macksd 6y agoIn the case of Hadoop's S3 connector, this could eliminate this entire directory, plus its tests, plus a bunch of hooks in the main code: https://github.com/apache/hadoop/tree/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard https://github.com/apache/hadoop/tree/trunk/hadoop-tools/had.... There's an argument in favor of keeping it in case other S3-compatible stores need it (though you'd still need DynamoDB or some equivalent) and because it makes metadata lookups so much faster than S3 scans, which helps with query planning performance. But I imagine even fewer people will take the trade-off now that Amazon S3 itself is consistent.