6 ms·

No SQL database is the one thing I disagree with. What if you want to write a SQL query/report? Do you have transactions? As the app gets bigger you might want

by jasfi 6y ago

No SQL database is the one thing I disagree with. What if you want to write a SQL query/report? Do you have transactions? As the app gets bigger you might want that.

onion2k 6y ago
Adding in a service like a SQL database because "you might want that" in the future is not a good idea. Apps should do what they need to do and no more. "Future proofing" is almost always a waste of time - even if you definitely want the feature in the future it's usually better to add it in when it's needed than adding it early and have to support something that adds no value yet. As for reports and transactions specifically, noSQL databases support both.
- d1sxeyes 6y ago
  OP isn't talking about noSQL databases, but no database at all (if I understand correctly - OP says '- No SQL database; I just keep everything in memory in native data structures'. In this case, I don't think OP is even thinking about noSQL databases.
  - kouteiheika 6y ago
    Correct. Sorry, I should have worded that better; unfortunately I cannot edit the post anymore.
- arkitaip 6y ago
  Unless, of course, switching to SQL turns out to be a major headache because it requires major rewrites, organizational changes, etc.
- zepolen 6y ago
  > "Future proofing" is almost always a waste of time Not really, unless you lack the experience to know what to future proof and what not to future proof. I've saved days/weeks of development by future proofing certain things, I've also saved days/weeks of development by taking pragmatic YAGNI in the right places. Obviously this will only surface on projects that last longer than a couple months ;)
kouteiheika 6y ago
> No SQL database is the one thing I disagree with. I'm not saying this approach is a good idea in every case. There are many situations where I would use a proper database. It really depends on what kind of app you're building, what kind of queries you want to run, how big is your dataset, your failover and availability requirements, your programming language, etc. In my case it was a great decision. The code is simple, and the data is convenient to access since it is already packed in native data structures. And it's really, really fast since all of the "queries" are basically LLVM-optimized machine code. > What if you want to write a SQL query/report? I just write a few lines of normal Rust code to do that. Sure, it's not as convenient. But I don't need to do that as often, so I don't care. > Do you have transactions? The concept of 'transaction' doesn't really make sense in this approach, since I'm directly modifying the data in memory, every change I make is atomic and can't fail, and I flush the data to the disk atomically as well.
- jolux 6y ago
  >The concept of 'transaction' doesn't really make sense in this approach, since I'm directly modifying the data in memory, every change I make is atomic and can't fail, and I flush the data to the disk atomically as well. So you don't have mutable data shared between threads then?
  - pastage 6y ago
    The basic stuff Rust will solve for you, but it is easy to forget logical dependencies and get an inconsistent state anyways... Just the ordinary close a session, send error message if not alive has bitten us in a 20 person project.
  - kouteiheika 6y ago
    > So you don't have mutable data shared between threads then? I might have worded that one sentence a little better, sorry. I do technically need to share data between threads, since my event loop runs on multiple threads, so if I want to mutably modify something I do need to wrap it either in a `Mutex` or an `RwLock` and lock it. What I meant was that it is not the same as a "transaction" in the SQL sense. It can't fail, there is no rollback, I decide on my own what level of granularity I need, and I bake the necessary atomicity into the data structures itself. So there are no SQL-style transactions anywhere - I just use a lock where it's appropriate to directly modify the memory, just as you'd do in any other situation. (Either by wrapping the whole structure in a lock, or wrapping only a single field, or by simply making the field atomic by itself.)
weego 6y ago
Isn't this the problem? "What if I need to do something I don't need to do right now? I know I best add it in just incase!" You have now recreated Drupal.
eloff 6y ago
Yeah, this seems problematic. What if you hard power off your server, how do you know your data didn't get corrupted? As someone who has written databases, and knows how difficult this is to get right, it makes my skin crawl. Backups and restore will also have to be DIY. Point in time restore probably won't be possible - and that's a life saver when you need it.
- kouteiheika 6y ago
  > What if you hard power off your server, how do you know your data didn't get corrupted? Atomically replacing the whole file instead of updating the data in-place, and using a checksumming filesystem like ZFS should give you such guarantee, no? > Backups and restore will also have to be DIY. Well, yeah. A daily cron job that `scp`s the data to an offsite location. Personally I don't really see much problem with that? It's easy to backup, and easy to restore.
  - eloff 6y ago
    > Atomically replacing the whole file instead of updating the data in-place, and using a checksumming filesystem like ZFS should give you such guarantee, no? No. You need to use fsync to ensure the file is flushed to disk. Also careful not to rename the file across mount points (/tmp is often a different mount and the most common location for people trying this.) > Well, yeah. A daily cron job that `scp`s the data to an offsite location. Personally I don't really see much problem with that? It's easy to backup, and easy to restore. That's fine, but your backup and restore granularity is daily. If you introduce a bug that writes the wrong / partial data, you can't recover anything that isn't in the previous day's backup. With a typical database you have a transaction log and you could restore back to any point during the day. This has been necessary about a dozen times during my career (once every two years or so on average.)
    - kouteiheika 6y ago
      > No. You need to use fsync to ensure the file is flushed to disk. Are you sure? At least in case of ZFS AFAIK the file wouldn't get corrupted, since it uses copy-on-write internally. Sure, you might get the old data back, but you wouldn't get corruption. And it automatically checksums your files anyway so it will detect any data that did get corrupted, and if you're using RAID-1 (as you should for any important data) it will automatically repair it. > Also careful not to rename the file across mount points (/tmp is often a different mount and the most common location for people trying this.) You can't rename across mount points. (Linux's `rename` syscall and the corresponding C API will return an EXDEV error if you try.) > If you introduce a bug that writes the wrong / partial data, you can't recover anything that isn't in the previous day's backup. Sure. If I was working in a bank not being able to do that would be totally unacceptable. But in my case that's the tradeoff I'm willing to live with.

by jasfi 6y ago

No SQL database is the one thing I disagree with. What if you want to write a SQL query/report? Do you have transactions? As the app gets bigger you might want that.

onion2k 6y ago

Adding in a service like a SQL database because "you might want that" in the future is not a good idea. Apps should do what they need to do and no more. "Future proofing" is almost always a waste of time - even if you definitely want the feature in the future it's usually better to add it in when it's needed than adding it early and have to support something that adds no value yet. As for reports and transactions specifically, noSQL databases support both.

d1sxeyes 6y ago
OP isn't talking about noSQL databases, but no database at all (if I understand correctly - OP says '- No SQL database; I just keep everything in memory in native data structures'. In this case, I don't think OP is even thinking about noSQL databases.
- kouteiheika 6y ago
  Correct. Sorry, I should have worded that better; unfortunately I cannot edit the post anymore.
arkitaip 6y ago
Unless, of course, switching to SQL turns out to be a major headache because it requires major rewrites, organizational changes, etc.
zepolen 6y ago
> "Future proofing" is almost always a waste of time Not really, unless you lack the experience to know what to future proof and what not to future proof. I've saved days/weeks of development by future proofing certain things, I've also saved days/weeks of development by taking pragmatic YAGNI in the right places. Obviously this will only surface on projects that last longer than a couple months ;)

kouteiheika 6y ago

> No SQL database is the one thing I disagree with. I'm not saying this approach is a good idea in every case. There are many situations where I would use a proper database. It really depends on what kind of app you're building, what kind of queries you want to run, how big is your dataset, your failover and availability requirements, your programming language, etc. In my case it was a great decision. The code is simple, and the data is convenient to access since it is already packed in native data structures. And it's really, really fast since all of the "queries" are basically LLVM-optimized machine code. > What if you want to write a SQL query/report? I just write a few lines of normal Rust code to do that. Sure, it's not as convenient. But I don't need to do that as often, so I don't care. > Do you have transactions? The concept of 'transaction' doesn't really make sense in this approach, since I'm directly modifying the data in memory, every change I make is atomic and can't fail, and I flush the data to the disk atomically as well.

jolux 6y ago
>The concept of 'transaction' doesn't really make sense in this approach, since I'm directly modifying the data in memory, every change I make is atomic and can't fail, and I flush the data to the disk atomically as well. So you don't have mutable data shared between threads then?
- pastage 6y ago
  The basic stuff Rust will solve for you, but it is easy to forget logical dependencies and get an inconsistent state anyways... Just the ordinary close a session, send error message if not alive has bitten us in a 20 person project.
- kouteiheika 6y ago
  > So you don't have mutable data shared between threads then? I might have worded that one sentence a little better, sorry. I do technically need to share data between threads, since my event loop runs on multiple threads, so if I want to mutably modify something I do need to wrap it either in a `Mutex` or an `RwLock` and lock it. What I meant was that it is not the same as a "transaction" in the SQL sense. It can't fail, there is no rollback, I decide on my own what level of granularity I need, and I bake the necessary atomicity into the data structures itself. So there are no SQL-style transactions anywhere - I just use a lock where it's appropriate to directly modify the memory, just as you'd do in any other situation. (Either by wrapping the whole structure in a lock, or wrapping only a single field, or by simply making the field atomic by itself.)

weego 6y ago

Isn't this the problem? "What if I need to do something I don't need to do right now? I know I best add it in just incase!" You have now recreated Drupal.

eloff 6y ago

Yeah, this seems problematic. What if you hard power off your server, how do you know your data didn't get corrupted? As someone who has written databases, and knows how difficult this is to get right, it makes my skin crawl. Backups and restore will also have to be DIY. Point in time restore probably won't be possible - and that's a life saver when you need it.

kouteiheika 6y ago
> What if you hard power off your server, how do you know your data didn't get corrupted? Atomically replacing the whole file instead of updating the data in-place, and using a checksumming filesystem like ZFS should give you such guarantee, no? > Backups and restore will also have to be DIY. Well, yeah. A daily cron job that `scp`s the data to an offsite location. Personally I don't really see much problem with that? It's easy to backup, and easy to restore.
- eloff 6y ago
  > Atomically replacing the whole file instead of updating the data in-place, and using a checksumming filesystem like ZFS should give you such guarantee, no? No. You need to use fsync to ensure the file is flushed to disk. Also careful not to rename the file across mount points (/tmp is often a different mount and the most common location for people trying this.) > Well, yeah. A daily cron job that `scp`s the data to an offsite location. Personally I don't really see much problem with that? It's easy to backup, and easy to restore. That's fine, but your backup and restore granularity is daily. If you introduce a bug that writes the wrong / partial data, you can't recover anything that isn't in the previous day's backup. With a typical database you have a transaction log and you could restore back to any point during the day. This has been necessary about a dozen times during my career (once every two years or so on average.)
  - kouteiheika 6y ago
    > No. You need to use fsync to ensure the file is flushed to disk. Are you sure? At least in case of ZFS AFAIK the file wouldn't get corrupted, since it uses copy-on-write internally. Sure, you might get the old data back, but you wouldn't get corruption. And it automatically checksums your files anyway so it will detect any data that did get corrupted, and if you're using RAID-1 (as you should for any important data) it will automatically repair it. > Also careful not to rename the file across mount points (/tmp is often a different mount and the most common location for people trying this.) You can't rename across mount points. (Linux's `rename` syscall and the corresponding C API will return an EXDEV error if you try.) > If you introduce a bug that writes the wrong / partial data, you can't recover anything that isn't in the previous day's backup. Sure. If I was working in a bank not being able to do that would be totally unacceptable. But in my case that's the tradeoff I'm willing to live with.