I would advise against buying into a large framework too much, because in my experience they aren't very performant, and you usually do need to care about speed

I would advise against buying into a large framework too much, because in my experience they aren't very performant, and you usually do need to care about speed here. I think you're better off taking the modules you need from open source libraries, favouring fast C and C++ implementations, and then writing the glue yourself. That way, you won't pay for abstractions you don't need. The glue shouldn't take very long to write.