That's how a regular KVM works as well (mmap a physical memory block) and the mismatch comes from the fact that all the regular OS memory management mechanisms

That's how a regular KVM works as well (mmap a physical memory block) and the mismatch comes from the fact that all the regular OS memory management mechanisms -- reclaim, swap, page cache, etc. -- are all broken by the abstraction provided. FV guests can similarly have a balloon driver, that doesn't mean the interface isn't awkward.