7 ms·

How is this setup fault tolerant? What happens if there is a hardware failure? How do you partition access in a way that means an extremely active user doesn't

How is this setup fault tolerant? What happens if there is a hardware failure? How do you partition access in a way that means an extremely active user doesn't impact availability?

bob1029 5y ago
> How is this setup fault tolerant? It is not. > What happens if there is a hardware failure? The product would suffer a total outage until manual intervention takes place. A restore of the VM from snapshot would be carried out by the customer. Some loss of the most recent business data would occur (i.e. between latest snapshot and time of failure). All of this is understood and agreed to by our customers. > How do you partition access in a way that means an extremely active user doesn't impact availability? Partitioning is not applicable. Our entire product fits on 1 machine and dividing the I/O along any dimension does not add much value to the performance equation.
- vidanay 5y ago
  Wait a minute....are you implying that I don't need Azure and FB level infrastructure for my local recipe application? /s
  - lkxijlewlf 5y ago
    I'm pretty sure you do need FB and Azure level infra for you recipe app. I've read a couple blog posts, watched a video on YouTube and copied the code from Stack Overflow, so I'm pretty much an expert on this, so trust me.
    - vidanay 5y ago
      My recipe application is going to be written in a purely functional language with CQRS and a message bus that provides "guaranteed once" determinism. I should be able to spin up no more than a half dozen containers on two VM's to handle this architecture. The database will probably be redundant MSSQL and Oracle (just in case one technology is fundamentally broken). Database writes will be proxied by a Redis instance. I will use 3x 3080 GPU to train a ML model used for scaling the recipe depending on the number of people you wish to serve. Hmmm...I might need a third VM.
      - kevin_thibedeau 5y ago
        It's never gonna scale. Get on the NoSQL train now to future proof and woo the VCs.
        xcambar 5y ago
        Yes. Make it web scale. Also, think early on about your compensation packages. You don't want to lose a 10x engineer to a FAANG, do you?
        fzil 5y ago
        > Make it web scale. Oh god whenever I read/hear that it reminds of the ridiculously funny video about nodejs and Apache servers. [1] [1]. https://youtu.be/bzkRVzciAZg https://youtu.be/bzkRVzciAZg
        xcambar 5y ago
        Glad you liked it! The reference to this series of videos was intended of course. The specific video I had in mind was actually https://youtu.be/b2F-DItXtZs https://youtu.be/b2F-DItXtZs Enjoy!
        fzil 5y ago
        Oh yes! I’ve seen this before as well. Should’ve guessed it’s this considering we were talking about sql.
        usrbinbash 5y ago
        This video is just gold. asyncio is probably one of the worst things that happened to programming in the last 2 decades. It's essentially a step backwards, to the technology of the 90s (cooperative multitasking), when a badly written text editor could cause an entire OS to freeze.
        joconde 5y ago
- qvrjuec 5y ago
  Interesting. For an extremely specific use case and with users who understand and accept the caveats of this approach I'm sure it would work well enough. The most confusing thing to me is that there is apparently an intersection of users who are ok with an outage and data loss with users who want a product which can > execute queries and reliably receive results within microseconds What is your product? Who are these users?
  - bob1029 5y ago
    I agree that 'reliably' would be a poor choice of diction when considered in your context. The user group of our product is effectively only professional bankers and their managers. No one in the general public has any direct access.
  - lkxijlewlf 5y ago
    - qvrjuec 5y ago
      No need to be condescending when I'm trying to learn. The premise of using SQLite because it was easier and to save fractions of a second didn't make sense to me as a tradeoff for potentially losing data
  - gbear605 5y ago
    I can think of plenty of services that an occasional (once a year? less?) outage is okay. Heck, anything relying on AWS us-east-1 is going to have outages that frequently based on the last few months. Meanwhile, almost any service is better off when its response times are cut drastically. I’ve seen many instances where a service’s response times are more than half waiting for a db to respond.
    - qvrjuec 5y ago
      It's not the threat of an outage with data loss that is concerning to me- I just want to understand use case that needs fractions of a second shaved off of query times by using SQLite in this way that is also ok with the possibility of data loss.

How is this setup fault tolerant? What happens if there is a hardware failure? How do you partition access in a way that means an extremely active user doesn't impact availability?

bob1029 5y ago

> How is this setup fault tolerant? It is not. > What happens if there is a hardware failure? The product would suffer a total outage until manual intervention takes place. A restore of the VM from snapshot would be carried out by the customer. Some loss of the most recent business data would occur (i.e. between latest snapshot and time of failure). All of this is understood and agreed to by our customers. > How do you partition access in a way that means an extremely active user doesn't impact availability? Partitioning is not applicable. Our entire product fits on 1 machine and dividing the I/O along any dimension does not add much value to the performance equation.

vidanay 5y ago
Wait a minute....are you implying that I don't need Azure and FB level infrastructure for my local recipe application? /s
- lkxijlewlf 5y ago
  I'm pretty sure you do need FB and Azure level infra for you recipe app. I've read a couple blog posts, watched a video on YouTube and copied the code from Stack Overflow, so I'm pretty much an expert on this, so trust me.
  - vidanay 5y ago
    My recipe application is going to be written in a purely functional language with CQRS and a message bus that provides "guaranteed once" determinism. I should be able to spin up no more than a half dozen containers on two VM's to handle this architecture. The database will probably be redundant MSSQL and Oracle (just in case one technology is fundamentally broken). Database writes will be proxied by a Redis instance. I will use 3x 3080 GPU to train a ML model used for scaling the recipe depending on the number of people you wish to serve. Hmmm...I might need a third VM.
    - kevin_thibedeau 5y ago
      It's never gonna scale. Get on the NoSQL train now to future proof and woo the VCs.
      - xcambar 5y ago
        Yes. Make it web scale. Also, think early on about your compensation packages. You don't want to lose a 10x engineer to a FAANG, do you?
        fzil 5y ago
        > Make it web scale. Oh god whenever I read/hear that it reminds of the ridiculously funny video about nodejs and Apache servers. [1] [1]. https://youtu.be/bzkRVzciAZg https://youtu.be/bzkRVzciAZg
        xcambar 5y ago
        Glad you liked it! The reference to this series of videos was intended of course. The specific video I had in mind was actually https://youtu.be/b2F-DItXtZs https://youtu.be/b2F-DItXtZs Enjoy!
        fzil 5y ago
        Oh yes! I’ve seen this before as well. Should’ve guessed it’s this considering we were talking about sql.
        usrbinbash 5y ago
        This video is just gold. asyncio is probably one of the worst things that happened to programming in the last 2 decades. It's essentially a step backwards, to the technology of the 90s (cooperative multitasking), when a badly written text editor could cause an entire OS to freeze.
        joconde 5y ago
qvrjuec 5y ago
Interesting. For an extremely specific use case and with users who understand and accept the caveats of this approach I'm sure it would work well enough. The most confusing thing to me is that there is apparently an intersection of users who are ok with an outage and data loss with users who want a product which can > execute queries and reliably receive results within microseconds What is your product? Who are these users?
- bob1029 5y ago
  I agree that 'reliably' would be a poor choice of diction when considered in your context. The user group of our product is effectively only professional bankers and their managers. No one in the general public has any direct access.
- lkxijlewlf 5y ago
  - qvrjuec 5y ago
    No need to be condescending when I'm trying to learn. The premise of using SQLite because it was easier and to save fractions of a second didn't make sense to me as a tradeoff for potentially losing data
- gbear605 5y ago
  I can think of plenty of services that an occasional (once a year? less?) outage is okay. Heck, anything relying on AWS us-east-1 is going to have outages that frequently based on the last few months. Meanwhile, almost any service is better off when its response times are cut drastically. I’ve seen many instances where a service’s response times are more than half waiting for a db to respond.
  - qvrjuec 5y ago
    It's not the threat of an outage with data loss that is concerning to me- I just want to understand use case that needs fractions of a second shaved off of query times by using SQLite in this way that is also ok with the possibility of data loss.