3 ms·
Did you use Camus for ETL, and if so, did you have to modify it to work with S3?
by optimusclimb 11y ago
Did you use Camus for ETL, and if so, did you have to modify it to work with S3?
- paladin314159 11y agoWe don't use Camus; IIRC, it didn't exist at the time that we built most of the infrastructure. We just read data directly out of Kafka using client libraries.
- optimusclimb 11y agoApologies for delayed response, I'm guessing you won't see it, but...without Camus in place...did you do anything to ensure exactly-once semantics in moving the data to the Batch Layer? For the real time layer I see it as not being mission critical for most data sets to be 100% correct, but for the ETL part of the process, the guarantees provided by Camus (ensured by the OutputCommitters part of MR I believe) are invaluable.