My understanding of OpenGL (which could be wrong) is that buffers are only transferred when loading textures onto the graphics card, which generally only happen

My understanding of OpenGL (which could be wrong) is that buffers are only transferred when loading textures onto the graphics card, which generally only happens at startup, and is time-consuming anyway (since they must find their way from disk to graphics memory). So memory buffer handles can still be passed around, since clients rarely touch the raw data. (Indeed, this is exactly how X11 works.)