Yes, NVIDIA GPU SIMD units are 32-wide (also called a warp), AMD GPU units are 64-wide, and Intel is anything from 4-16 depending on what processor you're using

Yes, NVIDIA GPU SIMD units are 32-wide (also called a warp), AMD GPU units are 64-wide, and Intel is anything from 4-16 depending on what processor you're using. CUDA/OpenCL allow one to query this width and optimize one's kernel's to avoid pointless synchronization within a vector unit. CUDA 8 now has a NOP synchronization that only fires between warps to canonize this sort of programming going forward.