8 ms·

There are some web apps still in production that I wrote almost a decade ago in Node+Express in the simplest, dumbest style imaginable. The only dependencies ar

by pavlov 5y ago

There are some web apps still in production that I wrote almost a decade ago in Node+Express in the simplest, dumbest style imaginable. The only dependencies are Express and some third-party API connectors. The database is an append-only file of JSON objects separated by newlines. When the app restarts, it reads the file and rebuilds its memory image. All data is in RAM. I figured these toys would be replaced pretty quickly, but turns out they do the job for these small businesses and need very little maintenance. Moving the app to a new server instance is dead simple because there's basically just the script and the data file to copy over, so you can do OS updates and RAM increases that way. Nobody cares about a few minutes of downtime once a year when that happens. There are good reasons why we have containers and orchestration and stuff, but it's interesting to see how well this dumb single-process style works for apps that are genuinely simple.

derefr 5y ago
How do you feel about SQLite? Because when I read this architecture description, it mostly vibes with me, until I think about what happens to the data in the event of a power cut. Is there logic in your app to potentially throw the last line away (incl. truncating it from the file) if it's invalid due to being the result of a non-atomic write? If so, seems a bit Not-Invented-Here compared to just using a (runtime-embedded) library that does that for you :)
- justinpombrio 5y ago
  If you're not writing data very often, this isn't a concern. For example, if you assume 5 microseconds to write a line, and 1 line written per hour on average, then the chance that the power goes out while you're writing a line is 10^-9, i.e. will never happen.
- pavlov 5y ago
  The AOF reader will discard and emit a warning about lines that can’t be parsed, but that’s the extent of it. These apps are on Digital Ocean, and I don’t remember ever having unplanned downtime with them. They do sometimes migrate instances with advance notice, but that’s a clean shutdown. I’m sure SQLite is a better choice for almost any app. My reason for not using it was to try avoiding dependencies out of curiosity, and also that I honestly really don’t like writing SQL — it just feels boring and error-prone. (Like eating celery, I know objectively it’s good for me.)
  - derefr 5y ago
    > My reason for not using it was to try avoiding dependencies out of curiosity, and also that I honestly really don’t like writing SQL — it just feels boring and error-prone. Well, sure, but you can just read and write JSON blobs to a single-column table in SQLite. See also, "SQLite makes a better BLOB store than the filesystem does" (https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html)
- weaksauce 5y ago
  > SQLite responds gracefully to memory allocation failures and disk I/O errors. Transactions are ACID even if interrupted by system crashes or power failures. All of this is verified by the automated tests using special test harnesses which simulate system failures. from the sqlite about page. it's one of the most bullet proof and hardened pieces of software out there I think. it was made basically for exactly the ops use case of a file on disk. but who knows what the use case is for them. maybe writes to the db are few and far between so it's a fairly moot point.
- eternityforest 5y ago
  SQLite doesn't do "lines" like a text file, or truncate anything(Or, it probably does internally, but that's not how users think of it). It's a real SQL DB with real records and transactions, and it is one of the most trusted and reliable pieces of software ever made. Like, check out the change log to get a sense of how they do stuff.
  - derefr 5y ago
    I think you misread what I said, or maybe didn't read the post I was replying to. I was pointing out that flat files have a problem with truncation; and that SQLite, despite being a very different thing than an append-only text file, can be effectively used as "an append only text file, but ACID." My question on how the GP poster feels about SQLite, is down to SQLite being the obvious solution to a problem I wasn't sure they realized they had — but also potentially still being "too many dependencies" for them.
spurgu 5y ago
I have literal dozens of these kinds (Node+Express under PM2) of small apps running around everywhere in production (almost 8 years = almost a decade ;). Using SQLite (when you need an actual DB) makes things a lot easier in this regard as well. I've tried doing some thing in Python (my initial programming love) over the years but I keep going insane because every time I'm forced to read up on the state of ecosystem (choosing versions and package managers and whatnot) and it drives me insane. I just install Node+Express and can get to work immediately (and finish quickly).
mbrodersen 5y ago
I suspect that 95% of business applications could be implemented just fine with that architecture. However I would use SQLite instead of a plain file. Just for added commit safety.
- kragen 5y ago
  For simple write patterns like this, I've had better robustness experiences with dumb filesystem access than with SQLite.
  - deterministic 5y ago
    That makes sense.
sam0x17 5y ago
I did a very similar setup to this as a boy using a PHP file that re-wrote itself as a sort of key-based data store. This was fantastic on free shared web hosting sites since no free db options were available. This was circa 2005.
anandh07 5y ago
Totally agree with simple architecture for simple and small use cases. Just curious to know, are you running any small businesses this way?
dmw_ng 5y ago
> database is an append-only file of JSON objects separated by newlines. When the app restarts, it reads the file and rebuilds its memory image. All data is in RAM Apps like this tend to perform like an absolute whippet too (or if they dont, getting them to perform well is often a 5 line change). It's really freeing to be able to write scans and filters with simple loops that still return results faster than a network roundtrip to a database. The problem is always growth, either GC jank from a massive heap, running out of RAM, or those loops eventually catching up with you. Fixing any one of these eventually involves either serialization or IO, at which point the balance is destroyed and a real database wins again.
- bob1029 5y ago
  > The problem is always growth, either GC jank from a massive heap, running out of RAM, or those loops eventually catching up with you Absolutely. The challenge is having enough faith that it will take long enough to catch up to you. Statistically speaking, it won't catch up to you and if it does, it will take so long you should have seen it coming from miles away and had time to prepare. In my systems that use an in-memory/append-only technique, I try to keep only the pointers and basic indexes in memory. With modern PCIe flash storage, there is no good justification for keeping big fat blobs around in memory anymore.
  - aprdm 5y ago
    Could you expand what you mean by keeping pointers and basic index in memory?
    - bob1029 5y ago
      Pointers are tuples of (Id, LogOffset) and are used to map logical identities to positions of those objects in the append-only log. Indexes are usually a tuple of (Some64BitKey, Id) and are used to map physical business keys to logical object identities. These entries are only candidates in the case where the key material needs to be hashed and inspected for actual equivalence. One big advantage with this approach is that you can stream big blobs directly out of the log to a caller-supplied buffer or stream. No intermediate allocations required aside from some small buffers.
ammanley 5y ago
Reminds me a lot of this (first paragraph): https://litestream.io/blog/why-i-built-litestream/ https://litestream.io/blog/why-i-built-litestream/ Well done on building an easy-to-maintain single node app with few dependencies. You would be the SWE I would send prayers of thanks too after onboarding (and for not making me crawl through a massive Helm chart/CloudFormation template hell).
smm11 5y ago
This. Not that I'm all about janky, but my road is littered with stuff I didn't think would make it through summer, and everything I check is still ticking 5, 7, 10 years later. LONG ago I was amused by a Sun box in a closet that nobody knew anything about. I heard about the serial label printer that stopped working eight months ago, which was eight months after I shut off the Sun. I brought it back up again late one Friday, and the old/broken label printer magically worked again. Now my stuff is that.
- uxamanda 5y ago
  Ha, now you can use the label printer to label the machine!
mftb 5y ago
I do almost this exact thing for all my personal stuff. I have 5 or 6 going in a vm for simple things like my bookmarks, etc... works great. I could definitely see it solving many small business use-cases.
danenania 5y ago
Built-in first-class concurrency (ala node, golang, rust, etc.) is a huge win for simple architectures, since it lets you avoid adding a background queue, or at least delay it for a very long time. I think people are also too quick to add secondary data stores and caches. If you can do everything with a transactional SQL database + app process memory instead, that is generally going to save you tons of trouble on ops, consistency, and versioning issues, and it can perform about as well with the right table design and indexes. For example: instead of memcache/redis, set aside ~100 MB of memory in your app process for an LRU cache. When an object is requested, hit the DB with an indexed query for just the 'updatedAt' timestamp (should be a sub-10ms query). If it hasn't been modified, return the cached object from memory, otherwise fetch the full object from the DB and update the local cache. For bonus points, send an internal invalidation request to any other app instances you have running when an object gets updated. Now you have a fast, scalable, consistent, distributed cache with minimal ops complexity. It's also quite economical, since the RAM it uses is likely already over-provisioned. This is exactly the approach that EnvKey v2[1] is using, and it's a huge breath of fresh air compared to our previous architecture. Just MySQL, Node/TypeScript, and eventually consistent replication to S3 for failover. We also moved to Fargate from EKS (AWS kubernetes product), and that's been a lot simpler to manage as well. 1 - https://v2.envkey.com https://v2.envkey.com
- conradfr 5y ago
  > Built-in first-class concurrency (ala node, golang, rust, etc.) is a huge win for simple architectures, since it lets you avoid adding a background queue, or at least delay it for a very long time. >For example: instead of memcache/redis, set aside ~100 MB of memory in your app process for an LRU cache. Erlang/Elixir for the win with (almost transparent multi-core) concurrency and ETS ;)
  - danenania 5y ago
    Oh yeah, Erlang/Elixir certainly belong in that list (probably at the front of it).
- gregmac 5y ago
  > For example: instead of memcache/redis, set aside a ~100 MB of memory in your app process for an LRU cache. When an object is requested, hit the DB with an indexed query for just the 'updatedAt' timestamp (should be a sub-10ms query). If it hasn't been modified, return the cached object from memory, otherwise fetch the full object from the DB and update the local cache. I've never built something with this type of mechanism for a DB query, but it's interesting. I don't think I've ever timed a query like this, but I feel like it's going to be an "it depends" situation based on what fields you're pulling back, if you're using a covering index, just how expensive the index seek operation is, and how frequently data changes. I've mainly always treated it as "avoid round trips to the database" -- zero queries is better than one, and one is better than five. I also guess it depends on how frequently it's updated: if 100% of the time the timestamp is changed, you might as well just fetch (no caching). Based on all the other variables above, the inflection point where it makes sense to do this is going to change. Interesting idea though, thanks. > For bonus points, send an internal invalidation request to any other app instances you have running when an object gets updated. Now you have a fast, scalable, consistent, distributed cache with minimal ops complexity. Now you have to track what other app servers exist, handle failures/timeouts/etc in the invalidation call, as well as have your app's logic able to work properly if this invalidation doesn't happen for any reason (classic cache invalidation problem). My inclination is at this point you're on the path of replicating a proper cache service anyways, and using Redis/Memcache/whatever would ultimately be simpler.
uuyi 5y ago
I would love to see more stuff like that. An application I have written recently for personal use is a double entry accounting system after GNUcash hosed itself and gave me a headache. This is based on Go and SQLite. The entire thing is one file (go embed rocks) and serves a simple http interface with a few JS functions like it is 2002 again. The back end is a proper relational model that is stored in one .db file. It is fully transactional with integrity checks. To run it you just start program and open a browser. To backup you just copy the .db file. You can run reports straight out of SQLite in a terminal if you want. This whole concept could scale to tens of users fine for LOB applications and consume little memory or resources.
- weaksauce 5y ago
  sqlite really does very well with reads but not as much for locking writes. not saying it couldn't scale to many users but I think the other person is a bit optimistic on a double entry accounting app being only reads. I would imagine it could certainly easily serve a few hundred though if not a few thousand.
  - kirab 5y ago
    SQLite does really well for locking writes as well as soon as you activate the WAL.
- powersurge360 5y ago
  Check out alpinejs or stimulusjs and combine it with htmx to get to a SPA like experience with very little additional complexity! Htmx let’s you serve partials over the wire instead of a page load so you can update the page incrementally and alpine and stimulus are both tools to add JS sprinkles like you’ve described in a way that is unobtrusive.
  - mbrodersen 5y ago
    Adding dependencies is the opposite of simple.
  - uuyi 5y ago
epolanski 5y ago
I often rewrite in my free time what I do at work without dependencies and I'm often amazed at how far and faster you can move.

by pavlov 5y ago

derefr 5y ago

How do you feel about SQLite? Because when I read this architecture description, it mostly vibes with me, until I think about what happens to the data in the event of a power cut. Is there logic in your app to potentially throw the last line away (incl. truncating it from the file) if it's invalid due to being the result of a non-atomic write? If so, seems a bit Not-Invented-Here compared to just using a (runtime-embedded) library that does that for you :)

justinpombrio 5y ago
If you're not writing data very often, this isn't a concern. For example, if you assume 5 microseconds to write a line, and 1 line written per hour on average, then the chance that the power goes out while you're writing a line is 10^-9, i.e. will never happen.
pavlov 5y ago
The AOF reader will discard and emit a warning about lines that can’t be parsed, but that’s the extent of it. These apps are on Digital Ocean, and I don’t remember ever having unplanned downtime with them. They do sometimes migrate instances with advance notice, but that’s a clean shutdown. I’m sure SQLite is a better choice for almost any app. My reason for not using it was to try avoiding dependencies out of curiosity, and also that I honestly really don’t like writing SQL — it just feels boring and error-prone. (Like eating celery, I know objectively it’s good for me.)
- derefr 5y ago
  > My reason for not using it was to try avoiding dependencies out of curiosity, and also that I honestly really don’t like writing SQL — it just feels boring and error-prone. Well, sure, but you can just read and write JSON blobs to a single-column table in SQLite. See also, "SQLite makes a better BLOB store than the filesystem does" (https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html)
weaksauce 5y ago
> SQLite responds gracefully to memory allocation failures and disk I/O errors. Transactions are ACID even if interrupted by system crashes or power failures. All of this is verified by the automated tests using special test harnesses which simulate system failures. from the sqlite about page. it's one of the most bullet proof and hardened pieces of software out there I think. it was made basically for exactly the ops use case of a file on disk. but who knows what the use case is for them. maybe writes to the db are few and far between so it's a fairly moot point.
eternityforest 5y ago
SQLite doesn't do "lines" like a text file, or truncate anything(Or, it probably does internally, but that's not how users think of it). It's a real SQL DB with real records and transactions, and it is one of the most trusted and reliable pieces of software ever made. Like, check out the change log to get a sense of how they do stuff.
- derefr 5y ago
  I think you misread what I said, or maybe didn't read the post I was replying to. I was pointing out that flat files have a problem with truncation; and that SQLite, despite being a very different thing than an append-only text file, can be effectively used as "an append only text file, but ACID." My question on how the GP poster feels about SQLite, is down to SQLite being the obvious solution to a problem I wasn't sure they realized they had — but also potentially still being "too many dependencies" for them.

spurgu 5y ago

I have literal dozens of these kinds (Node+Express under PM2) of small apps running around everywhere in production (almost 8 years = almost a decade ;). Using SQLite (when you need an actual DB) makes things a lot easier in this regard as well. I've tried doing some thing in Python (my initial programming love) over the years but I keep going insane because every time I'm forced to read up on the state of ecosystem (choosing versions and package managers and whatnot) and it drives me insane. I just install Node+Express and can get to work immediately (and finish quickly).

mbrodersen 5y ago

I suspect that 95% of business applications could be implemented just fine with that architecture. However I would use SQLite instead of a plain file. Just for added commit safety.

kragen 5y ago
For simple write patterns like this, I've had better robustness experiences with dumb filesystem access than with SQLite.
- deterministic 5y ago
  That makes sense.

sam0x17 5y ago

I did a very similar setup to this as a boy using a PHP file that re-wrote itself as a sort of key-based data store. This was fantastic on free shared web hosting sites since no free db options were available. This was circa 2005.

anandh07 5y ago

Totally agree with simple architecture for simple and small use cases. Just curious to know, are you running any small businesses this way?

dmw_ng 5y ago

> database is an append-only file of JSON objects separated by newlines. When the app restarts, it reads the file and rebuilds its memory image. All data is in RAM Apps like this tend to perform like an absolute whippet too (or if they dont, getting them to perform well is often a 5 line change). It's really freeing to be able to write scans and filters with simple loops that still return results faster than a network roundtrip to a database. The problem is always growth, either GC jank from a massive heap, running out of RAM, or those loops eventually catching up with you. Fixing any one of these eventually involves either serialization or IO, at which point the balance is destroyed and a real database wins again.

bob1029 5y ago
> The problem is always growth, either GC jank from a massive heap, running out of RAM, or those loops eventually catching up with you Absolutely. The challenge is having enough faith that it will take long enough to catch up to you. Statistically speaking, it won't catch up to you and if it does, it will take so long you should have seen it coming from miles away and had time to prepare. In my systems that use an in-memory/append-only technique, I try to keep only the pointers and basic indexes in memory. With modern PCIe flash storage, there is no good justification for keeping big fat blobs around in memory anymore.
- aprdm 5y ago
  Could you expand what you mean by keeping pointers and basic index in memory?
  - bob1029 5y ago
    Pointers are tuples of (Id, LogOffset) and are used to map logical identities to positions of those objects in the append-only log. Indexes are usually a tuple of (Some64BitKey, Id) and are used to map physical business keys to logical object identities. These entries are only candidates in the case where the key material needs to be hashed and inspected for actual equivalence. One big advantage with this approach is that you can stream big blobs directly out of the log to a caller-supplied buffer or stream. No intermediate allocations required aside from some small buffers.

ammanley 5y ago

Reminds me a lot of this (first paragraph): https://litestream.io/blog/why-i-built-litestream/ https://litestream.io/blog/why-i-built-litestream/ Well done on building an easy-to-maintain single node app with few dependencies. You would be the SWE I would send prayers of thanks too after onboarding (and for not making me crawl through a massive Helm chart/CloudFormation template hell).

smm11 5y ago

This. Not that I'm all about janky, but my road is littered with stuff I didn't think would make it through summer, and everything I check is still ticking 5, 7, 10 years later. LONG ago I was amused by a Sun box in a closet that nobody knew anything about. I heard about the serial label printer that stopped working eight months ago, which was eight months after I shut off the Sun. I brought it back up again late one Friday, and the old/broken label printer magically worked again. Now my stuff is that.

uxamanda 5y ago
Ha, now you can use the label printer to label the machine!

mftb 5y ago

I do almost this exact thing for all my personal stuff. I have 5 or 6 going in a vm for simple things like my bookmarks, etc... works great. I could definitely see it solving many small business use-cases.

danenania 5y ago

Built-in first-class concurrency (ala node, golang, rust, etc.) is a huge win for simple architectures, since it lets you avoid adding a background queue, or at least delay it for a very long time. I think people are also too quick to add secondary data stores and caches. If you can do everything with a transactional SQL database + app process memory instead, that is generally going to save you tons of trouble on ops, consistency, and versioning issues, and it can perform about as well with the right table design and indexes. For example: instead of memcache/redis, set aside ~100 MB of memory in your app process for an LRU cache. When an object is requested, hit the DB with an indexed query for just the 'updatedAt' timestamp (should be a sub-10ms query). If it hasn't been modified, return the cached object from memory, otherwise fetch the full object from the DB and update the local cache. For bonus points, send an internal invalidation request to any other app instances you have running when an object gets updated. Now you have a fast, scalable, consistent, distributed cache with minimal ops complexity. It's also quite economical, since the RAM it uses is likely already over-provisioned. This is exactly the approach that EnvKey v2[1] is using, and it's a huge breath of fresh air compared to our previous architecture. Just MySQL, Node/TypeScript, and eventually consistent replication to S3 for failover. We also moved to Fargate from EKS (AWS kubernetes product), and that's been a lot simpler to manage as well. 1 - https://v2.envkey.com https://v2.envkey.com

conradfr 5y ago
> Built-in first-class concurrency (ala node, golang, rust, etc.) is a huge win for simple architectures, since it lets you avoid adding a background queue, or at least delay it for a very long time. >For example: instead of memcache/redis, set aside ~100 MB of memory in your app process for an LRU cache. Erlang/Elixir for the win with (almost transparent multi-core) concurrency and ETS ;)
- danenania 5y ago
  Oh yeah, Erlang/Elixir certainly belong in that list (probably at the front of it).
gregmac 5y ago
> For example: instead of memcache/redis, set aside a ~100 MB of memory in your app process for an LRU cache. When an object is requested, hit the DB with an indexed query for just the 'updatedAt' timestamp (should be a sub-10ms query). If it hasn't been modified, return the cached object from memory, otherwise fetch the full object from the DB and update the local cache. I've never built something with this type of mechanism for a DB query, but it's interesting. I don't think I've ever timed a query like this, but I feel like it's going to be an "it depends" situation based on what fields you're pulling back, if you're using a covering index, just how expensive the index seek operation is, and how frequently data changes. I've mainly always treated it as "avoid round trips to the database" -- zero queries is better than one, and one is better than five. I also guess it depends on how frequently it's updated: if 100% of the time the timestamp is changed, you might as well just fetch (no caching). Based on all the other variables above, the inflection point where it makes sense to do this is going to change. Interesting idea though, thanks. > For bonus points, send an internal invalidation request to any other app instances you have running when an object gets updated. Now you have a fast, scalable, consistent, distributed cache with minimal ops complexity. Now you have to track what other app servers exist, handle failures/timeouts/etc in the invalidation call, as well as have your app's logic able to work properly if this invalidation doesn't happen for any reason (classic cache invalidation problem). My inclination is at this point you're on the path of replicating a proper cache service anyways, and using Redis/Memcache/whatever would ultimately be simpler.

uuyi 5y ago

I would love to see more stuff like that. An application I have written recently for personal use is a double entry accounting system after GNUcash hosed itself and gave me a headache. This is based on Go and SQLite. The entire thing is one file (go embed rocks) and serves a simple http interface with a few JS functions like it is 2002 again. The back end is a proper relational model that is stored in one .db file. It is fully transactional with integrity checks. To run it you just start program and open a browser. To backup you just copy the .db file. You can run reports straight out of SQLite in a terminal if you want. This whole concept could scale to tens of users fine for LOB applications and consume little memory or resources.

weaksauce 5y ago
sqlite really does very well with reads but not as much for locking writes. not saying it couldn't scale to many users but I think the other person is a bit optimistic on a double entry accounting app being only reads. I would imagine it could certainly easily serve a few hundred though if not a few thousand.
- kirab 5y ago
  SQLite does really well for locking writes as well as soon as you activate the WAL.
powersurge360 5y ago
Check out alpinejs or stimulusjs and combine it with htmx to get to a SPA like experience with very little additional complexity! Htmx let’s you serve partials over the wire instead of a page load so you can update the page incrementally and alpine and stimulus are both tools to add JS sprinkles like you’ve described in a way that is unobtrusive.
- mbrodersen 5y ago
  Adding dependencies is the opposite of simple.
- uuyi 5y ago

epolanski 5y ago

I often rewrite in my free time what I do at work without dependencies and I'm often amazed at how far and faster you can move.