Initial encoding happens on different machines (obviously) but a stream multiplexer that maximizes the end user experience is a lot more complicated than just e

Initial encoding happens on different machines (obviously) but a stream multiplexer that maximizes the end user experience is a lot more complicated than just encoding to a preset selection of rates, after all every keyframe and subsequent updates have their encoding scheme determined by the throughput measured during the transmission of the keyframe(s)+updates just preceding it. It's really not that difficult; if you have one encode going per-user, you can just reinitialize the encoder's ratecontrol, and if you don't have one per user, you can simply use a short GOP and swap the bitrate at each keyframe, in the same way that H.264 extended profile proposed using Switching Pictures (but in a way that real software actually supports). Of course this latter method requires multiple simultaneous encodes per channel, or an SVC-like system (which almost nobody supports, and for which there aren't any good software implementations anyways). Is it really that important for Justin.TV to have less than one second latency? The vast majority of the site is completely non-latency-critical (from my cursory browsing of it). Heck, the broadcast company I worked for last summer had over 10 second latency and nobody ever complained about that--and that was live television, including sports games and so forth. Plus, if Justin.TV is already doing a scheme like this, surely upgrading to a better, faster encoder is going to decrease, not increase, their CPU load. Video is relatively forgiving by the way, the audio is the real kicker, a single packet delivered a little bit too late will instantly cause annoyance whereas with the video you can be a little early or late and it doesn't seem to matter too much. It sounds as if your buffer size for audio is too small. There is no reason why video or audio should act differently in that regard; in fact, audio should be easier, as losing a video packet will result in a loss of video until the next keyframe, while in audio, every single frame is a keyframe.