A large part of the complexity comes from the GPU (often) having a separate memory from the CPU connected via a relatively slow bus. Another large part is that

A large part of the complexity comes from the GPU (often) having a separate memory from the CPU connected via a relatively slow bus. Another large part is that the GPU runs asynchronously from the CPU. Another large part is that you must write your code in a way that allows thousands of concurrent threads of execution or you won't run faster than the CPU. Another large part is the fact that the instruction set and features and general performance level of the GPU are unknown at development time and wildly different on common hardware. Despite all that, reaching near peak hardware utilization and real-time interactive performance is expected by users on all hardware regardless of the differences. All these things together mean that the obvious and simple designs for APIs unfortunately don't work very well.