Having more speculators actually makes it more likely that the asset is fairly valued. Speculators are the driving force which "correct" the price of overvalue

Having more speculators actually makes it more likely that the asset is fairly valued. Speculators are the driving force which "correct" the price of overvalued or undervalued assets. More speculators mean that assets will reach their "fair price" more quickly. In this way speculators actually reduce volatility and make it harder to manipulate the market. Keep in mind that speculators are not necessarily in long positions, they could also be shorting Bitcoin (which would be the logical thing to do if you believe there's a bubble).