6 ms·
Show HN: C++ AWS MSK IAM Auth Implementation – Goodbye Kafka Passwords
In 2023, AWS announced[1] IAM authentication for MSK Kafka clusters with support for "all programming languages"… except C++. While Java[2], Python[3], Go[4], and others got official SDKs, C++ developers/vendors were stuck hardcoding SCRAM-SHA credentials in code/configs or relying on heavier Java-based tools like Kafka Connect or Apache Flink.
Later, community projects added Rust[5] and Ruby[6] support. Why no C++? Rust might be the new favorite, but C++ is still king for high-performance data systems: minimal dependencies, lean resource use, and raw speed.
At Timeplus, we needed IAM auth for our C++ streaming engine, Proton, so we built it ourselves. Today, we’re open-sourcing our code for AWS MSK IAM authentication. It’s live in Timeplus Proton 1.6.12
Just attach an IAM role to your EC2 instance or EKS pod, then put the Timeplus Proton single binary inside, start the server, then run the following SQL to read or write MSK:
CREATE EXTERNAL STREAM msk_stream(column_defs)
SETTINGS
type='kafka',topic='topic2',
brokers='prefix.kafka.us-west-2.amazonaws.com:9098',
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM';
The core logic is just two files under 200 lines and you can reuse the code anywhere.
https://github.com/timeplus-io/proton/blob/develop/src/IO/Kafka/AwsMskIamSigner.h https://github.com/timeplus-io/proton/blob/develop/src/IO/Ka...
https://github.com/timeplus-io/proton/blob/develop/src/IO/Kafka/AwsMskIamSigner.cpp https://github.com/timeplus-io/proton/blob/develop/src/IO/Ka...
We’d love to get your feedback and work together to make this a standalone library—or even get it into ClickHouse or AWS SDK for C++.
For those curious about Timeplus Proton: it’s an open-source streaming engine we built in C++ (think “FlinkSQL in C++” meets ClickHouse’s columnar storage). Later this month, we will also open-source our C++ code for Apache Iceberg read&write. Stay tuned.
Links:
[1] https://aws.amazon.com/blogs/big-data/amazon-msk-iam-authentication-now-supports-all-programming-languages/ https://aws.amazon.com/blogs/big-data/amazon-msk-iam-authent...
[2] https://github.com/aws/aws-msk-iam-auth https://github.com/aws/aws-msk-iam-auth
[3] https://github.com/aws/aws-msk-iam-sasl-signer-python https://github.com/aws/aws-msk-iam-sasl-signer-python
[4] https://github.com/aws/aws-msk-iam-sasl-signer-go https://github.com/aws/aws-msk-iam-sasl-signer-go
[5] https://docs.rs/aws-msk-iam-sasl-signer https://docs.rs/aws-msk-iam-sasl-signer
[6] https://rubygems.org/gems/aws-msk-iam-sasl-signer/ https://rubygems.org/gems/aws-msk-iam-sasl-signer/
- fsafdsaewr 2y ago[flagged]
- fdafdsfe 2y ago[dead]
- iwriawei 2y ago[dead]
- soijaijte 2y ago[dead]
- mdaniel 2y agoCongratulations, and how shameful for AWS not to meet developers where they are
- jit_hacker 2y agoMSK IAM support has long mystified me. I think they only supported Java for the first 9 months or so. Even then they still don't have GO or PHP support. It's not a ton of work, they're reusing request signer code anyways.
- jovezhong 2y agoAccording to my teammate who actually wrote the C++ code for this, there are lack of documentations of how the AWS_MSK_IAM is supposed to work. He has to check the Java/Python implementation line by line to avoid those guesswork
- mdaniel 2y agoWell, there's precedent for that since the $(aws eks get-token) is just a base64 pre-signed GetCallerIdentity URL but I don't think that's documented anywhere, either, but can be spotted by squinting at aws-iam-authenticator source My suspicion is that if they didn't want to bother to write a C++ client, they for sure wouldn't have the empathy(?) to document how anyone else could, too. I said empathy but I kind of wonder if by publishing how something works they're committing to it, versus they're currently only one commit away from changing it in their clients, without having to notify anyone