Okay i'm officially a dumbass - looking at the pinout i can clearly see Arduino pins and normal GPIO pins are mixed up on the board I used (seeed xiao rp2040).

Okay i'm officially a dumbass - looking at the pinout i can clearly see Arduino pins and normal GPIO pins are mixed up on the board I used (seeed xiao rp2040). Arduino strikes again! Just checked - of course if works fine on proper pins