You did the work by moving the magnets away from each other. This converts some of the kinetic energy you created by moving them into potential energy. As you m

You did the work by moving the magnets away from each other. This converts some of the kinetic energy you created by moving them into potential energy. As you moved the magnets back closer to each other, some of this potential energy converted back to kinetic energy, until they were close enough that the force exerted by the magnetic field exceeded that of gravity, at which point there was a large conversion of magnetic potential energy into kinetic, and then gravitational potential energy. Further, note that since a force was exerted over a distance , work was indisputably done. Exactly what did that work is a matter of opinion, but work was certainly done.