4 ms·
Is there a good sub or discussion board for the design and implementation of complex backend app architectures that need to operate at scale? By complex I mean
by atombender 5y ago
Is there a good sub or discussion board for the design and implementation of complex backend app architectures that need to operate at scale? By complex I mean something beyond just CRUD. I'm thinking of consensus systems, transactional systems, queuing systems, distributed locking, sharding techniques — but also just practical engineering stuff, like which off-the-shelf tools to pick for certain problems.
For example, recently I ran into something where an app needs to send a certain volume of telemetry data to BigQuery, but buffering the outgoing data in memory isn't possible, because BQ ingest is slow enough that the processes would OOM; but also, because it must be possible to kill the app without losing the data. Turns out even Google Pub/Sub can't handle the volume. Writing to Postgres would work, but now we're talking about writing a whole little app with a schema just for some short-lived linear data. Must be a solved problem? But the best solution I found was to send the data to RedPanda and then have a simple little consumer ingest it into BQ asynchronously. When I had this question, I realized I had nobody to bounce ideas back and forth with. Stack Overflow doesn't have the right people (in my experience), and "Ask HN" doesn't get any replies because people don't seem to watch that space for new threads.
Another problem I ran into the other day, which I still don't have a solution to, is spinning up end-to-end integration tests on Kubernetes. There must be some orchestration tool that can let us reuse all our existing Kustomize manifests for our apps so that I can say "this test requires apps A, B, C, please boot them and their required databases in a new empty namespace, then wait for them to get ready, then run the tests and then tear down everything, and then report all the results somewhere l, and also plug this into the GitHub PR status checks and log outcomes to Slack". Again, I don't know where to go for this. I tried the CNCF Slack community, but it was all tumbleweeds. I asked on "Ask HN" and got a reply from a CircleCI person that misunderstood my question. Surely this is a common problem? We use ArgoCD for deploys, but it doesn't have a solution to this problem, and Argo Workflows (which we currently use for e2e, but we are unhappy with it) curiously has no synergy with Argo app definitions.
- atombender 5y ago(Wow, something removed all the line breaks from my comment.)