A high end gaming card has thousands of those. If you read the CUDA documentation you can find out more about the hardware. Physically a high end card has sever

A high end gaming card has thousands of those. If you read the CUDA documentation you can find out more about the hardware. Physically a high end card has several streaming multiprocessors each with their own caches and access to a global level 2 cache, instructions are sorted into categories (integer math, floating point math, load/store/special functions) each of those correspond to functional units on the chip that execute in lock-step. The current generation has 32/64 of those (called the warp size, notice 192/32 = 6). High end cards have several of those streaming multiprocessors, for example the Titan 30 * 192 = 5760. A warp scheduler on each chip decides what to run, so for all 192 "cores" to be in use you would have to have independent floating point / integer and special function instructions to do. It is the same marketing trick that AMD uses to sell two integer functional units in a four core design as 8 'cores'. Overall cuda models the underlying hardware fairly faithfully, so if you are interested in the hardware design you should read the CUDA documentation.