I believe order is important. The way to verify this is by running a simulation with players whose skills are all equal but vary with some normal distribution,

I believe order is important. The way to verify this is by running a simulation with players whose skills are all equal but vary with some normal distribution, and you will find that skill levels to do move about quite a bit even though the underlying skill level is fixed (I am thinking about order as just random shuffling here, so if we take a dataset where there is just randomness and in converges quickly then order doesn't matter...this doesn't really happen with ELO). This isn't to say that rankings don't converge. They do. But things like order definitely can get in the way. Glicko controls for this by modelling uncertainty in a skill estimate (iirc, proportional to the number of matches seen). In practice, you would tend to burn the first few matches using ELO to try and get rankings to converge to a limit. But, again, even then I have found that issues. ELO will quickly identify the worst and best but there is a lot of volatility around the middle grouping of teams. And this does vary with the underlying activity you are modelling too: activities with a lot of randomness take longer to converge. You can also control convergence with the K-factor. I think this is one of the trade-offs that comes with the time dependent nature of ELO. You get more variance but less bias. Imo, it usually isn't a huge issue in practice because the upside of this is your method isn't static, and your ranking can respond to things like lineup changes (which tend to occur in many applications)...it works well in most applications.