4 ms·

I've built a complex CRM that handles 2.1 million USD in transactions every year. It is running sqlite with a simple in-memory lru cache (just a dict) that gets

I've built a complex CRM that handles 2.1 million USD in transactions every year. It is running sqlite with a simple in-memory lru cache (just a dict) that gets purged when a mutating query (INSERT, UPDATE or DELETE) is executed. It is very simple and more than fast enough. Friendly reminder that you shouldn't spend time fine tuning your horizontal autoscaler in k8s before making money.

midrus 6y ago
Do you mean I don't need Go microservices talking gRPC deployed in multiple kubernetes clusters with bash script based migrations via GitOps with my hand made multi cloud automation (in case we move clouds) following all the SCRUM practices to ship working software? Mindblowing.
- kwertyoowiyop 6y ago
  You will for your blog series that you mention prominently on your resume that gets you your next Senior Architect gig.
  - midrus 6y ago
    Agh!... that's the catch... what I'm gonna give talks about and what do I write on medium then.... now I get it. Thanks!.
- anticristi 6y ago
  > my hand made multi cloud automation (in case we move clouds) Aren't these symptoms of a deeper problem? Many Product Manager I talk to wanted me to build something that is as flexible as possible and solves all problems for everyone, everywhere. Go microservices with gRPC and Kubernetes feels like the only high-level technical decisions I can take in light of such information. :)
juancampa 6y ago
How do you ensure data is not lost to oblivion if a catastrophic system failure occurs?
- michaelmcmillan 6y ago
  Backups.
  - antoinealb 6y ago
    Does that mean it's okay for your application to loose transactions (which occured between the backup point and the failure point) or do you have other mitigations ?
    - benbjohnson 6y ago
      I'm the author of Litestream, which is an open-source tool for streaming replication for SQLite. That could be a good option if you need to limit your window for data loss. We have a pretty active Slack if you need help getting up and running. https://litestream.io/ https://litestream.io/
    - habibur 6y ago
      Guess this daily 2 seconds of downtime is worth it, when that reduces cost say from $2000/month to $20/month.
      - mrweasel 6y ago
        Many banks still “shutdown” for hours every night to do backups.
        finiteseries 6y ago
        I’m not anywhere near the banking industry but from HN alone I’ve been led to believe dailyish huge file transfers are also the norm in a variety of situations (aka SQLite’s backup strategy).
outworlder 6y ago
> Friendly reminder that you shouldn't spend time fine tuning your horizontal autoscaler in k8s Oh, but most companies need this! The biggest feat of microservices (which require ways to manage them, like k8s) was to provide the ability for companies to ship their organizational chart to production. If you don't need to ship your org chart and you can focus on designing a product, then you can go a long way without overly complicating your architecture.
- antonvs 6y ago
  I think this org chart issue has been overstated. If you have two services that are completely orthogonal, then combining them into a single application for deployment and operation purposes can be limiting. Developers of all people should understand the benefits of decoupling. There are a lot of downsides that come with jamming a whole lot of unrelated functionality into the same deployment unit. It's similar to the problems created by global variables. Once you start to depend on your services operating in a monolith, you can easily create rigidity that can be difficult to roll back. It's not that you can't design a monolith well, but microservices force you to consider important boundaries as opposed to simply violating them for the sake of convenience. It's another "human" issue, but it's unrelated to org charts. This doesn't mean that everything should be a microservice, though. Good architecures address the requirements of the systems being developed.
  - __jem 6y ago
    I think the deployment benefits have been even more overstated. This is an absolutely enormous trade-off between logical modularity and run time operational complexity. If your devs aren't good enough to enforce modular design in a single application, what makes you think they're good enough to handle complex distributed systems?
    - antonvs 6y ago
      > If your devs aren't good enough to enforce modular design in a single application, what makes you think they're good enough to handle complex distributed systems? That's a simplistic take on the reasons that monoliths tend towards breaking modularity. What makes me think managing microservices is perfectly viable is that the last three companies I've been at have been able to do it successfully, with a typical mix of good and less good developers, including cheap offshore devs. As for handling "complex distributed systems," in many cases all that's really needed is something like a managed container platform. Services like Fargate or Cloud Run, or managed Kubernetes, can do a good job of this. Developers can deploy and publish new services with minimal effort, and most of the operational complexity is managed by the platform. You do ideally want someone paying attention to overall architecture to avoid obvious pitfalls, such as effectively doing distributed joins via REST calls, and so on. This isn't that hard to understand, though, and teams that don't do the necessary architecture upfront tend to figure it out once they run into those problems themselves.

midrus 6y ago

Do you mean I don't need Go microservices talking gRPC deployed in multiple kubernetes clusters with bash script based migrations via GitOps with my hand made multi cloud automation (in case we move clouds) following all the SCRUM practices to ship working software? Mindblowing.

kwertyoowiyop 6y ago
You will for your blog series that you mention prominently on your resume that gets you your next Senior Architect gig.
- midrus 6y ago
  Agh!... that's the catch... what I'm gonna give talks about and what do I write on medium then.... now I get it. Thanks!.
anticristi 6y ago
> my hand made multi cloud automation (in case we move clouds) Aren't these symptoms of a deeper problem? Many Product Manager I talk to wanted me to build something that is as flexible as possible and solves all problems for everyone, everywhere. Go microservices with gRPC and Kubernetes feels like the only high-level technical decisions I can take in light of such information. :)

juancampa 6y ago

How do you ensure data is not lost to oblivion if a catastrophic system failure occurs?

michaelmcmillan 6y ago
Backups.
- antoinealb 6y ago
  Does that mean it's okay for your application to loose transactions (which occured between the backup point and the failure point) or do you have other mitigations ?
  - benbjohnson 6y ago
    I'm the author of Litestream, which is an open-source tool for streaming replication for SQLite. That could be a good option if you need to limit your window for data loss. We have a pretty active Slack if you need help getting up and running. https://litestream.io/ https://litestream.io/
  - habibur 6y ago
    Guess this daily 2 seconds of downtime is worth it, when that reduces cost say from $2000/month to $20/month.
    - mrweasel 6y ago
      Many banks still “shutdown” for hours every night to do backups.
      - finiteseries 6y ago
        I’m not anywhere near the banking industry but from HN alone I’ve been led to believe dailyish huge file transfers are also the norm in a variety of situations (aka SQLite’s backup strategy).

outworlder 6y ago

> Friendly reminder that you shouldn't spend time fine tuning your horizontal autoscaler in k8s Oh, but most companies need this! The biggest feat of microservices (which require ways to manage them, like k8s) was to provide the ability for companies to ship their organizational chart to production. If you don't need to ship your org chart and you can focus on designing a product, then you can go a long way without overly complicating your architecture.

antonvs 6y ago
I think this org chart issue has been overstated. If you have two services that are completely orthogonal, then combining them into a single application for deployment and operation purposes can be limiting. Developers of all people should understand the benefits of decoupling. There are a lot of downsides that come with jamming a whole lot of unrelated functionality into the same deployment unit. It's similar to the problems created by global variables. Once you start to depend on your services operating in a monolith, you can easily create rigidity that can be difficult to roll back. It's not that you can't design a monolith well, but microservices force you to consider important boundaries as opposed to simply violating them for the sake of convenience. It's another "human" issue, but it's unrelated to org charts. This doesn't mean that everything should be a microservice, though. Good architecures address the requirements of the systems being developed.
- __jem 6y ago
  I think the deployment benefits have been even more overstated. This is an absolutely enormous trade-off between logical modularity and run time operational complexity. If your devs aren't good enough to enforce modular design in a single application, what makes you think they're good enough to handle complex distributed systems?
  - antonvs 6y ago
    > If your devs aren't good enough to enforce modular design in a single application, what makes you think they're good enough to handle complex distributed systems? That's a simplistic take on the reasons that monoliths tend towards breaking modularity. What makes me think managing microservices is perfectly viable is that the last three companies I've been at have been able to do it successfully, with a typical mix of good and less good developers, including cheap offshore devs. As for handling "complex distributed systems," in many cases all that's really needed is something like a managed container platform. Services like Fargate or Cloud Run, or managed Kubernetes, can do a good job of this. Developers can deploy and publish new services with minimal effort, and most of the operational complexity is managed by the platform. You do ideally want someone paying attention to overall architecture to avoid obvious pitfalls, such as effectively doing distributed joins via REST calls, and so on. This isn't that hard to understand, though, and teams that don't do the necessary architecture upfront tend to figure it out once they run into those problems themselves.