Talked to a few LinkedIn engineers a while ago and was astonished at the size of some payloads they’re pushing over Kafka.. in almost all production deployments

Talked to a few LinkedIn engineers a while ago and was astonished at the size of some payloads they’re pushing over Kafka.. in almost all production deployments I’ve done with different teams in the past once we got rid of the big payloads and stored them on block storage and instead used pointers in the messages most pressure issues disappeared — switching to protobuf sounds like an attempt to decrease the payload sizes without offloading the huge payloads… which almost never works because hey if we’re faster now why not send more messages? That’s how it always goes…