jemalloc works much like what you're describing. The one missing piece in terms of userland/kernel cooperation is that jemalloc aggressively discards dirty unu

jemalloc works much like what you're describing. The one missing piece in terms of userland/kernel cooperation is that jemalloc aggressively discards dirty unused pages because it doesn't know whether other processes need those pages. If the kernel were to communicate the severity of memory pressure to userland, it would be possible to more intelligently rate-limit the purging.