jhgg

26 ms·

241.
by jhgg 8y ago
The root of this is definitely GCS. We noticed some keys in a GCS bucket go inaccessible since like 19:15-19:20ish PDT, and then we noticed increased timeouts/503's ramping up at 19:30 and plateauing at 19:45. But error rate for u
242.
by jhgg 8y ago
Load balancing a bunch of connections between a bunch of processes across a cluster of nodes is pretty well understood. I don't think that's what I'm trying say here. My post is more on the power of pre-emptive scheduling bu
243.
by jhgg 8y ago
Because Kafka is definitely not a tool suited for real time chat and event distribution to millions of clients. Once a message hits our distribution it’s fanned our to clients on average 5-10ms later.
244.
by jhgg 8y ago
Yes. We run bare VMs for this workload (n1-standard-X). They do in the event of hardware failure or other unexpected fault. Generally it does not get to that though, as google is fairly good at migrating your VMs off of a host if it detects
245.
by jhgg 8y ago
Can we all take a step back here and for a second realize that Node.JS is based upon a language/event model that was purpose-built and designed to handle client side browser operations, and has since been expanded into the server space
246.
by jhgg 8y ago
Who knows? We would have had a whole other set of issues - of which golang and JVM struggle at. There is more discussion in the rest of the comments in this post. Beam/OTP provide a fantastic foundation for building distributed soft re
247.
by jhgg 8y ago
All the Diacord API libraries are community built and maintained.
248.
by jhgg 8y ago
Since this blog post, we are well over 5m ccu (I don't think we have shared our current peaks publicly yet). We do egress 4m-6m websocket messages/sec through our system though at peak. So it's a lot more than just holding on
249.
by jhgg 8y ago
We would still use our own homegrown system. And we do use kafka internally already for a plethora of other things. Just not on the real-time chat side of things.
250.
by jhgg 8y ago
We do indeed. Rustler is very cool!
251.
by jhgg 8y ago
We actually added this 2nd half of last year. Message send operations are now deduped and idempotent given a client-generated nonce. Are you still experiencing this issue?
252.
by jhgg 8y ago
>I'm curious what they think of Workers. We are a very heavy user of cloudflare workers. Our marketing/developer/web app are entirely served from the edge using workers - and we use workers for request authentication for d
253.
by jhgg 8y ago
The high reduction cost is deliberate. Reductions in BEAM are an arbitrary value that certain operations in ERTS assigned. Sending to a remote node has a high reduction cost, which means that the more sends you do, the more your process has
254.
by jhgg 8y ago
We've since ditched fastglobal actually, as we found that it'd take too long to recompile code as the number of processes on the BEAM VM grew. As it turns out, the cost to recompile is linear scaling up with the # of processes on
255.
by jhgg 8y ago
The tldr is we wrote our own highly efficient SFU - and we used our own transport layer that is not dtls, but xsalsa poly over udp.
256.
by jhgg 8y ago
How does a company who owns a trademark get a domain early like Slack and Github did?
257.
by jhgg 8y ago
The cost savings here is in download/package size, and not runtime performance. Chances are latest chromium is faster than the system web-browser.
258.
by jhgg 8y ago
Tried to google “is google down”... oops.
259.
by jhgg 8y ago
We run between 200k and 2m processes per beam VM. I don't know how it'd be possible to get as precise metrics as we need just from relying on linux utilities. And in the same cluster, some processes although identical in code have
260.
by jhgg 8y ago
demonitor flush can be harmful in some instances as it invokes a selective receive to yank the down message from your process's mailbox. In the event that you are demonitor+flushing on a process that has a growing message queue, every
261.
by jhgg 8y ago
Rust however does not offer nearly the same level of power when it comes to runtime introspection that BEAM does. Coming from a company that uses elixir heavily and has made a significant investment in rust, I don’t think we would ever use
262.
by jhgg 8y ago
Access to these values are programmed in your cloudflare worker script. It's up to you to decide how they can be accessed.
263.
by jhgg 8y ago
It's a clustered service that is behind an SSL Load Balancer[1] that is behind Cloudflare. So not really a viable DDoS target. [1]: https://cloud.google.com/load-balancing/docs/ssl/
264.
by jhgg 8y ago
It depends on the activity of the server. But generally, you're only receiving message create events and updates to the server. But you aren't receiving things like member list updates, presence or typing events from the server un
265.
by jhgg 8y ago
Would rather waste bandwidth than CPU cycles in this case. Would take way too much CPU time to mux audio streams together server-side, and then recompress. (Means we have to buffer data for each sender, deal with silence, deal with retransm
266.
by jhgg 8y ago
We don't transcode audio/video on the server. Each stream is processed and muxed by the client. The server is merely relaying rtp packets and tagging them with a given ssrc per peer. The client does the rest of the work. The bulk
267.
by jhgg 8y ago
We host our voice server on dedicated hardware, not VPS. Visualization overhead for networking is too high for the cost. Additionally, you can buy bandwidth for much cheaper from dedicated hosting providers as opposed to cloud providers. Fo
268.
by jhgg 8y ago
We don't host our voice servers on google cloud as that's actually really expensive. We rent commodity dedicated servers. Our voice fleet is basically JBOS (just-a-bunch-of-servers). We use really low-spec servers too (4 core - 8
269.
by jhgg 8y ago
Perhaps I can answer some of these questions. We've dealt with scenarios where we've had to fail-over entire data-centers to other regions, or other data-centers within that region. We have a few mitigations in-place for stuff lik
270.
How Discord Handles Two and Half Million Concurrent Voice Users Using WebRTC(blog.discordapp.com)
649 points by jhgg 8y ago|244 comments

More ›