Clustering suggestion for CoAP server

Hello Hackers! Has anyone tried connecting more than 65K clients to a CoAP server such as Californium? During our scale tests at iotify we found that a single instance of CoAP server can not handle more than 65535 unique client IP/port combination. We use multiple source IPs so 16 bit UDP port shouldn't be a limitation. I haven't seen any clustering solution for CoAP so far except Eclipse Leshan, which unfortunately requires LWM2M protocol on top. Any help would be appreciated.