It really depends on your instruction mix. If only one in twenty instruction uses AVX, the rest of your instructions are running slower due to the lower clock

It really depends on your instruction mix. If only one in twenty instruction uses AVX, the rest of your instructions are running slower due to the lower clock and they aren't getting double the throughput. On top of that it could be some other thread using AVX, clocking down the entire core and harming the given thread that isn't using AVX. Intel has done a lot of things to try to balance this. One of those things is they don't even bother turning half the vector unit on unless you use it a lot. If you seldom issue an op with 512-bit operands, the CPU will actually dispatch them as multiple 256-bit operations, in which case you won't incur the drop in clock, but you also don't get the supposed benefit of double throughput. Furthermore the performance may be much worse if the CPU decides to turn up the remaining vector bits, because the clock drops dramatically while those units are charging up. So you can see that for someone trying to wring out every last bit of performance on a recent Intel CPU using all the advertised vector capabilities, optimization can become quite complicated.