3 ms·
>> Message size limit is 1MB. It defaults to this, but it can be set much higher if needed. What is it? I was surprised to hear it was this low.
by geoduck14 5y ago
>> Message size limit is 1MB.
It defaults to this, but it can be set much higher if needed.
What is it? I was surprised to hear it was this low.
- EdwardDiego 5y agoIt's set to 1MB (and some additional bytes for log overhead) because it was found (by LinkedIn? I think?) to be a reasonable default for throughput. Technically you can set it up to anywhere near MAX_INT bytes, would be interesting to see what happens when you send a 1GiB message. :D You'd most likely need to tune other properties depending on how big you're going. And message size, just to be clear, relates to a message from a producer which can contain 1 to N records, often compressed to reduce network contention. The 1MB limit applies to the bytes which are received by the broker, so you can send batches of records which are >1MB if you're compressing them. Compression imposes some CPU cost on producers and consumers, but unless you've specifically configured a topic to use a mismatching compression algorithm (this defaults to "just accept whatever algorithm the producer" used), there's no cost on the broker, it's written as is, and then consumers pull it still compressed.