3 ms·
Since it's Day 2, def appears to be an architectural scalability issue, not some bugs. Quick scan of their engineering blog shows the following stack: -Zookeep
by ceohockey60 7y ago
Since it's Day 2, def appears to be an architectural scalability issue, not some bugs. Quick scan of their engineering blog shows the following stack:
-Zookeeper, Kafka, Spark, Airflow, Faust (Kafka Streams in Python)
-Kubernetes (was/is SALT + Terraform)
-AWS Aurora, PG, ES, Influx
-Presto via AWS Athena, Redshift
-homegrown solutions for a data lake & managing k8s microservices
Fwiw, their team does appear to have a rigorous internal post-mortem process (SEV reviews) [0] drawn from industry best practices.
Let's hope a public one will be made when issues are resolved. (FINRA will be asking for one anyway.)
Not piling on. I genuinely feel for all the infra engineers staying up all day/night fixing this.
[0] https://robinhood.engineering/creating-a-sev-process-that-scales-with-robinhood-4b433f9c439b https://robinhood.engineering/creating-a-sev-process-that-sc...