32 ms·

SQLite Is Serverless

501 points by alexellisuk 7y ago

verdverm 7y ago
Choose your own definitions... but SQLite is not serverless by common parlance. Author creates two definitions for serverless which don't match the common usage. Serverless is more about DevOps / deploy experience than how the program leverages OS processes internally. Apparently MS and AWS are ISPs? Maybe SQLite could be serverless if you defined it as incapable or running as a server on its own?
- gpvos 7y ago
  Serverless used to have a very clear definition: not having or using a server, so SQLite is a perfect example of serverlessness. I was extremely confused when the newfangled definition (called neo-serverless on TFA) showed up, or your definition for that matter. Who ever thought up theses confusing meanings for a word that used to be perfectly clear? You don't need an ISP to have a server. Any computer or program that listens to a network port is a server.
  - aksss 7y ago
    Or is it any program that _responds_ to comms on a network port. The "serve" part of "server" ;)
- forsakenharmony 7y ago
  MS and AWS are indeed ISPs, but not because they have servers. They have their own IPs and global networking infrastructure
manigandham 7y ago
Why even write this? It's yet another convoluted use of the term and meaningless for SQLite of all things. Re: downvotes - what are people disagreeing with? That the term is not convoluted? That it's actually useful? That it helps to have more sub-definitions in an industry known for overloaded terms? I guarantee not a single person here has used "classic serverless" over "in-process" or "embedded" in their entire career.
- NhanH 7y ago
  I have seen that page for around 10 years or so (except the section that was added in 2018). The term has a very clear meaning and consistent uses within the docs (and consistent with my common sense understanding of the word too). It is not SQLite's fault that AWS and co muddles the water with their usage of the term
  - IAmEveryone 7y ago
    It's not their fault. It's not anybody's fault. Language evolves. But if you wouldn't write it this way today, you should just change it, instead of drawing your readers into a pointless fight over semantics.
- kick 7y ago
  The article is from 2007, and explicitly calls out that the "neo" usage of the term is different than the original. If you scroll to the fifth line or so, you'll find the update. I didn't have to scroll on my device, though i can imagine there are some smaller devices that may not have that line on the screen when opening the page.
  - manigandham 7y ago
    Where does it say 2007? Regardless it's vague and irrelevant material. Classic serverless has always been known as "in-process". The attempt at differences seems like adding marketing fluff rather than just removing the page entirely.
    - XelNika 7y ago
      It doesn't say 2007 anywhere, but the page is clearly at least that old. https://web.archive.org/web/20071115173112/https://www.sqlite.org/serverless.html https://web.archive.org/web/20071115173112/https://www.sqlit... I also found material from 2004 with the term. https://www.tcl.tk/community/tcl2004/Presentations/D.RichardHipp/slides/slides-all.html https://www.tcl.tk/community/tcl2004/Presentations/D.Richard... > Regardless it's vague and irrelevant material It's not vague at all, the term makes sense to distinguish "in-process" from client/server models. The page also includes an explanation on the very next line. > Adding an original timestamp to the page would've been 100x more helpful Not everyone reading about a database product is a developer. A timestamp would not help anyone unfamiliar with the history of the serverless term. The alternative is to rewrite all instances of "serverless" in the documentation which is a waste of time. Modern "serverless" is a stupid buzzword and this page clearly serves as a protest.
Eikon 7y ago
> SQLite is an example of a classic serverless database engine. With SQLite, there are no other processes, threads, machines, or other mechanisms (apart from host computer OS and filesystem) to help provide database services or implementation. There really is no server. Well, it's bending the overall consensus defining serverless as a managed / and or stateless service. I never saw anyone using the term "serverless" to mean "embeded". Using this definition, anything and everything that is not requiring a specific server to be served / distributed can be described as serverless.
- IAmEveryone 7y ago
  There's a section added in 2018 to deal with the apparent confusion. It'd be a better idea to just delete the page. It may have been written long before the meaning changed, but it's pointless to fight a losing and completely insignificant battle over language.
  - chungy 7y ago
    Hard disagree. The "neo-serverless" version has always been extremely confusing to me. I expect serverless to mean the absence of a server. In SQLite's particular case, it's subverting the expectation that has persisted since the beginning of time (of databases) that a database must be managed by a server.
- rumanator 7y ago
  > Well, it's bending the overall consensus defining serverless as a managed / and or stateless service. That assertion is quite the stretch because there is no consensus on what serverless actually means. The only thing that exists is that the concept of function-as-a-service is being forced as a placeholder for serverless, but some vendors try to manipulate the definition to include their managed services offerings.
- masklinn 7y ago
  The SQLite page dates back to 2007 at least: https://web.archive.org/web/20071115173112/https://www.sqlite.org/serverless.html https://web.archive.org/web/20071115173112/https://www.sqlit... > Using this definition, anything and everything that is not requiring a specific server to be served / distributed can be described as serverless. The distinction is useful to make when similar systems traditionally rely on a client/server model. Which many DBMS do, to say nothing of RDBMS. That it is server-less is a distinctive feature of SQLite as an RDBMS.
deleted 7y ago
[deleted]
jankotek 7y ago
I think better name (and coined since 1990ies) is embedded db engine. MySQL, Firebird (InterBase) etc can also be linked into binary and run in the same process, without any type of sockets.
tiku 7y ago
CSV is also serverless..
- hestefisk 7y ago
  Very serverless. The new thing.
- threatripper 7y ago
  But usually not multi-user-write with transaction support.
- deleted 7y ago
  [deleted]
- bob1029 7y ago
  CSV doesn't have a write-ahead log.
  - inopinatus 7y ago
    The file system it is stored on does.
  - masklinn 7y ago
    I don't think sqlite had WAL when that page was created.
    - threatripper 7y ago
      I think bob means transactions.
      - bob1029 7y ago
        Transactions, durability and acceptable performance with concurrent access.
- masklinn 7y ago
  CSV does not traditionally involve a server. RDBMS do.
oldgradstudent 7y ago
> Of those that are serverless, SQLite is the only one known to this author that allows multiple applications to access the same database at the same time. IIRC, MS Access allowed that, which explained a lot of its popularity.
- EGreg 7y ago
  Waaaaait wouldn’t that mean the file system is the server, with some binary API and responsible for handling concurrent access and locks for the entire file? LOL.
  - simias 7y ago
    Well you effectively report some of the workload normally handled by the server to the OS's filesystem layer, that's true. In particular you rely heavily on the FS locking working correctly. Calling the FS a "server" is a bit of a stretch though.
  - m-p-3 7y ago
    Serverless in this situation means that you don't really have to provision or setup an actual server to handle the database, the client itself just need the ability to read and write an SQLite file.
    - EGreg 7y ago
      Because many of the locking things are outsourced to the file system which acts as server for concurrent threads. A server is something that listens for commands from various clients and executes them.
  - idclip 7y ago
    That just circle jerk. There is an agreed upon def of serverless. If we reduce things to the absurd we stop being able to reason about things. https://en.m.wikipedia.org/wiki/Serverless_computing https://en.m.wikipedia.org/wiki/Serverless_computing
geokon 7y ago
a bit tangential, but when do you move form using in-application data structures (maps, trees, vector/arrays) to using a database? Is it basically when the data doesn't fit in memory? I've been programming for almost a decade and I've never come across needing a database... (for context, it's ten years without anything web related) I'm interested in them and I'd love to learn SQL but I can't even think of a use case outside of managing user in some web based application
- manigandham 7y ago
  Persistence, portability, and durability. SQLite is an actual file on disk, can be easily read by multiple programs/languages, and has write-ahead logging to keep your data safe from crashes.
- dragontamer 7y ago
  > when do you move form using in-application data structures (maps, trees, vector/arrays) to using a database? The killer feature of databases, including no-sql or older ones like BerkeleyDB, is multiprocess synchronization. If two programs (or two copies of the same program) need to coordinate their data, a database is often far easier to use than writing your own transaction layer through files, mutexes, and other primitives. Web applications, such as forums or discussion boards, have many users and processes adding data simultaneously. That's why databases are so popular in web backends.
- typon 7y ago
  Any medium size or greater application either uses SQL or recreates a shitty version of it. You just can't write complex code without relations.
- d0mine 7y ago
  Anytime you want persistency: sqlite is a replacement for fopen() https://www.sqlite.org/whentouse.html https://www.sqlite.org/whentouse.html
- shakna 7y ago
  Structured transactions between applications is one place. You can get by with files, but they're slower. A DB is the right choice. Structured data anytime you're hitting multiuser - web, networked games, collab programs, live maps, etc. Anything that's massively stateful. A DB hands you a lot of guarantees. The filesystem has some of them, but is much slower.
eb0la 7y ago
Serverless, like all embedded databases ;-) In the sense you don't need to provision an additional piece of infrastructure to power your application :-)
ashildr 7y ago
I find this article valuable. I made a mental note to check if SQLite is enough every time I consider one of the “serverless” options.
k__ 7y ago
Two definitions of serverless? I heard many, but what they're writing isn't one of them. As far as I can tell SQLite was always called an embedded database, and never a serverless one.
- masklinn 7y ago
  The page is more that 10 years old. And it uses "serverless" in the sence of server-less aka not having a server. Which is pretty damn sensible terminology, as opposed to the newer "serverless" meaning" runs on some utility server outside of your care or control".
  - k__ 7y ago
    I know, but have you anyone calling "embedded databases" "serverless databases" in the last 10 years? It's like someone found this page and felt very smart about it, because they stick it to the serverless crowd.
    - SQLite 7y ago
      "embedded database" does not imply "serverless database". Other embedded RDBMSes run servers; they just run the server as a separate thread rather than a separate process. SQLite is different in that there is no separate thread of control. SQLite runs in the same thread as the application that calls it. There is no separate thread hanging around to clean up or handle background tasks after an SQLite function call returns.
      - deleted 7y ago
        [deleted]
imafish 7y ago
My understanding of serverless = easily scalable, managed service. But somehow the word annoys people. Maybe we should find a better word?
- afroboy 7y ago
  serverless it mean no server but everything use servers, yep we should find a better expression.
  - threatripper 7y ago
    Peer-to-peer networking in its most pure form does not need a dedicated/central server. Each machine hosts a client/server instance locally.
- vbezhenar 7y ago
  My understanding was that cloud = easily scalable, managed service.
  - iampims 7y ago
    That was mine too. It’s possible that this previously applied to primitives (cpu, memory, network, storage) and now refers to applications (keyvalue store, SQL database, message queues, etc.)
- rumanator 7y ago
  > My understanding of serverless = easily scalable, managed service. There is already a term for that concept: managed services. There is no such thing as a managed service that's designed not to be scalable. Some implementations may be better at scaling than others, but that's it. The serverless buzzword is pure marketing.
  - nck4222 7y ago
    On the one hand, the term serverless is really frustrating to me. On the other hand, there's a certain amount of amusement I get from seeing "the cloud" become a marketing buzzword in the mid-late '00's, and now seeing the same thing happen with "serverless." When you look at the implementation of the two "technologies", they're about 95% the same. Yet somehow they're pitched as these big revolutions. In another decade when terminals or p2p become popular again we'll be hearing about some new buzzword like "Terran" computing, or "social" architecture or something.
JDiculous 7y ago
This is not new or unknown information, why is this on the front page?
- klez 7y ago
  Because other people deemed it interesting.
  - JDiculous 7y ago
    The fact that SQLite is serverless is common knowledge and not interesting unless you're brand new to databases
threatripper 7y ago
In case SQLite is not enough and you need redundant servers or clustering, there's also database servers that use SQLite as storage engine: http://rqlite.com/ http://rqlite.com/ https://dqlite.io/ https://dqlite.io/
- Gys 7y ago
  So you run an app with sqlite on one server and sync to sqlite on another server? What would be the benefit to using a separate db server like in the 'neo-serveless' setup?
  - threatripper 7y ago
    No, in this case you always have to use rqlite/dqlite because they manage the network synchronization. They use SQLite as storage engine (one SQLite database per server instance).
    - Gys 7y ago
      I understand that in those cases rqlite/dqlite is used. But that it just a technical detail. My point is that I am running two servers: one with the app and Xqlite and another one with Xqlite. In case of a neo-serveless setup, I also have two servers: one with the app, the other one with the db server. So what are the benefits of the Xqlite setup? I looked into that before and for one thing, Xqlite is slower (obviously) then just sqlite. So speed is not a key benefit. I also will have to manage both servers myself. At least for a seperate db server I have the benefit that I can buy that as a service, incl management, backups and such.
      - tracker1 7y ago
        Not the author, or knowing of all the technical details... simplistic replication structure and redundancy/failover without an expensive or more complex RDBMS solution while still self-hosting the service. There are still a lot of instances where you cannot use a cloud provider for your app or database. To be honest, I'd probably lean more towards a nosql database that has in the box, relatively easy replication strategy, though that might mean 3+ db servers for good performance. (RethinkDB, MongoDB, Cassandra/ScyllaDB, Cockroach). Just depends on the budget and resources.
Gys 7y ago
The main problem that I see for using sqlite is exactly for it being 'Classic Serverless'. Because how does one keep an up te date backup? Deploying to Heroku, Dokku, AWS Lambda and such means the sqlite file will be lost on a crash or new deploy. Even a VM can crash. Export to S3 on every write? Maybe if changes do not happen often, so only for specific use cases (and actually I think you should just generate static html in that case). I use sqlite for local tests for cases where the live app has a 'neo-serverless' database. It is very, very fast so the tests run almost instantly.
- coleifer 7y ago
  >how do you back up a file
  - wongarsu 7y ago
    It's a valid concern in so far as that any standalone database setup you would use at a serverless provider has backups already taken care of (for example AWS Lambda with AWS RDS has backups built into RDS). If I deploy SQLite instead I have to take care of backups myself, and that might not be trivial to get right.
- hashhar 7y ago
  It's just a single file. Easier to backup than any other database. Also I don't think any cloud provider/database provides an always up to date backup other than a standby replica (which isn't also a backup exactly).
  - wongarsu 7y ago
    If I remember the "ways to corrupt SQLite databases" page discussed a few days ago, backing up SQLite isn't entirely trivial. By default there are up to three files that have to be copied simultaneously, else you risk corruption. The optimum is to run the backup command to create a copy, but that requires realizing this exists.
    - 7y ago
greatjack613 7y ago
This document is golden, serverless means an application reading and writing to the database in the process directly
dana321 7y ago
It does allow multiple applications to access the same database at the same time, but when you do so it really hurts performance. I noticed this when i wrote a web crawler in go and used sqlite as the backend. As soon as i connect using the command line interface, it slows down significantly. Just something to bear in mind if you want to use it with multiple processes!
- iampims 7y ago
  Even with WAL enabled?
  - FpUser 7y ago
    Let's say there are concurrent writes to a single table with few indexes. Process of updating said table and indices eventually boils down to update of a single B+ tree from a multiple threads. In general until serialized it is physically impossible without corrupting the tree. Sure there might be other specially crafted implementation of table more friendly to concurrent writes but there will be trade-offs. There are no miracles in the world. As for particular case with WAL, all it does in this particular scenario is act as a queue. If your database load is spiky then it can even out load and make an impression of faster response. Under constant load it will not speed up the things and will internally serialize all actual updates
    - catalogia 7y ago
      If you need concurrent/parallel writes, SQLite is not the right tool for you. You may as well lament that hammers are no good for driving screws.
      - FpUser 7y ago
        Can you please point me to where did I say anything about SQLite being right tool for concurrent write? My point was that true concurrent writes to a generic database table with indexes are are physically impossible in a general case disregarding what database it is.
bullen 7y ago
I integrated my database with the app server, but I still open a socket internally to access it. So my system is also "serverless" in that meaning. Acronyms used to be confusing, this is just ridiculous. http://root.rupy.se http://root.rupy.se
WolfOliver 7y ago
I like the definition of classical- and neo- serverless. Does anyone knows more articles about serverless in terms of classical serverless?
peterwwillis 7y ago
"Microsoft Azure Cosmo DB and Amazon S3 are examples of a neo-serverless databases." Can we back away a bit from the bandwagoning of misused terminology? Serverless literally means "running your apps on somebody else's server". S3 is not a server you run your apps on, it is SaaS that you manipulate through an API - you don't put your apps on it. If S3 is serverless, then literally every network service of any kind is serverless.
- tracker1 7y ago
  I always preferred XaaS nomenclature myself. "XaaS with autoscaling"... is probably better than "Serverless"
mettamage 7y ago
Does this mean that you can hack some database storage (w/ sqlite) together on frontend only hosting platforms like Github or Netlify? I think not, but I wonder if some hack is available by virtue of it simply being a file that you can read (and somehow) write to. The best I came up with: let's say you have a toy project, and you call the Github API and replace the file upon every write. Implementing a read is easier as you know where the file is located. This hack shows that you somehow need write access to get any form of performance out of it, because this hack is super slow.
- aikah 7y ago
  > Does this mean that you can hack some database storage (w/ sqlite) together on frontend only hosting platforms like Github or Netlify? You can run SQLlite in the client via webassembly therefore open a SQLite file in the browser to query it yes, you just can't write anything in it and expect it to persist somehow on the static hosting service itself. > The best I came up with: let's say you have a toy project, and you call the Github API and replace the file upon every write. Implementing a read is easier as you know where the file is located. This hack shows that you somehow need write access to get any form of performance out of it, because this hack is super slow. No you'll need a server for that, you can't make random HTTP requests to any server in the browser, because of CORS/SAME ORIGIN policies. "SQLite is serverless" is meaningless buzzword. It just means that SQLite is equivalent a flat file where you'd shove some data, just that you can use SQL to query that file instead of having to index data in it yourself.
  - giancarlostoro 7y ago
    You could make it persist if you have it uploaded somewhere. But then I guess thats not serverless.
    - Frost1x 7y ago
      "Serverless" is a garbage marketing term in general. It sounds sexy to nontechnical management folks who are used to hearing a bunch of expensive costs and the word "server" associated with them in some fashion. From that view, anything that gets rid of those pesky "servers" sounds like a win.
m3kw9 7y ago
I don’t get it, then any db can be classic serverless if they run in the same computer as the app?
- flatline 7y ago
  No, there’s still some form of IPC going on then. SQLite is an embedded database - everything happens in the memory space of the same application. Of course you can still throw a web front-end in front of your application but there’s still no extra hop to the database, per se.
- jasode 7y ago
  >any db can be classic serverless if they run in the _same computer_ as the app? Not just the "same computer" ... it's the same process id (PID). Extract of relevant text from that webpage: >Classic Serverless: The database engine runs within the same process, thread, and address space as the application. There is no message passing or network activity. In other words, when you compile and link "sqlite.c" into your own executable, the same PID (process) that handles text input and paints pixels on the screen -- is the same PID that writes to the sqlite database file. It's all the same process. That's what they mean by "classic serverless". In contrast, if you make a Go executable that writes to MySQL/PostgreSQL db and make them both run on the same physical computer, that's not "classic serverless". It's because when you enter "ps -aux" to list all running processes, you see separate PIDs for the Go executable and the MySQL db engine. Other jargon used might be "in-process" vs "out-of-process" or "embedded" vs "external". SQLite is sometimes characterized as "in-process embedded database" but MySQL is an "out-of-process" db.
- l34ch 7y ago
  Nope, only SQLite doesn't use another process as an intermediary to read/write to the DB.
- manigandham 7y ago
  Yes, technically, although they make the distinction is that it's in the same process. It's a bad definition. I would stick with the terms "embedded" or "in-process" which have been around for decades and are well-understood.
loukrazy 7y ago
Isn’t it easier to say “embedded” and “severless”
- manigandham 7y ago
  Not in this thread apparently. I guarantee nobody here as ever used the term "serverless" over "embedded" or "in-process" in their entire career but apparently the purity and nostalgia of SQLite overrides it.
  - filleduchaos 7y ago
    This article was written in 2007, so yes, people have been using the word "serverless" before managed service providers decided to take it up as a buzzword.
    - manigandham 7y ago
      Nobody used it before. Even the about page says: "SQLite is an embedded SQL database engine." "serverless" is a marketing term, then and now. Not sure why there's such a big defense of it. If you have to argue this much over the provenance of a term that clearly isn't used the same way today then it's a good sign that it's not very useful.
      - catalogia 7y ago
        > Nobody used it before. The archive.org links already provided to you prove that you're flat wrong about that. SQLite used the term many years before it was a webshit buzzword. They used it in the intuitive straight forward english sense; somebody who has no hat is hatless, somebody who has no home is homeless, and a database system that has no server process is serverless. In 2007 when it was written, nobody would have batted at eye at this term, the meaning would have been immediately clear to anybody who had any familiarity with databases.
        manigandham 7y ago
bane 7y ago
I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/write methods. SQLite is not really any different, except the read/write work is already done, and you use SQL to format the data values and encode the read/write logic instead of the language you are working in. The file format has some cool extensions like text indexing, geospatial, etc. But it's no more a RDBMS than reading and writing a JSON file is. "But there's indexes!" Yes, just liked you might build an index on your JSON file and read that before reading the JSON file to know where things are faster -- and then you have to write all the code to do that. SQLite is just a file format, where you can also build indexes and all the code for that is already done for you. It's just a file format with an API that's similar to the ones you'd use for any regular old RDBMS and uses SQL as the domain language to read/write data. It's just a file format. Anything you can do/can't do with a file format you do with SQLite. It's just a file format. A nice convenient one that you are probably better off using than most other things for most purposes. It's just a file format. edit I'm glad this comment is getting such a response. I'm not trying to be mean, just help clarify thinking. Here's two thought experiments: 1) If SQLite didn't require you to use SQL as the read/write logic and was called "datalite", and instead just forced you to use C function calls, exactly like you would if you were working with literally any other file format on the planet, would you still be confused as to what it is? 2) Do you consider reading and writing to any other file format anywhere in the hierarchy of RDBMSs? Consider Python's csv module. Is that an RDBMS? Let's move away from tabular data, how about python-docx?
- lucideer 7y ago
  I'm not sure I understand your comment. What's the signifigance of it being "just a file format"? What are the aspects of "approaching it from a classic RDBMS angle" that are incompatible with it being "just a file format"? I've always seen it as "just a file format" myself, and I think I've always approached it "from a classic RDBMS angle", but I've never felt confused, and I don't see what I'm overcomplicating.
pm90 7y ago
What I really want: SQLite storage backend driver for s3/gcs. No need for disks then. I haven’t been able to find such a solution though; and am not technically proficient enough in C (the Lang SQLite is written in) to do so myself.
- bshipp 7y ago
  that would be cool! as an interim step, if your data is small enough, perhaps you could run an in-memory sqlite db and periodically backup to a permanent s3 file? APSW exposes the sqlite backup api so you could do them online without shutting down the database.
- tn1 7y ago
  The challenge with this is that S3 and friends are object stores, meaning you upload or download the whole file each time. As you can imagine, this will cost you tremendous bandwidth even to insert one row. Furthermore, it doesn't solve the multiple writers problem, because (afaik) there's no way to lock a file on S3.
  - pm90 7y ago
    > As you can imagine, this will cost you tremendous bandwidth even to insert one row. Is this the cost of network transfers changed by AWS or GCP? What if I'm hosting my app on e.g. EKS or GKE respectively?
    - kondro 7y ago
      It costs roughly $5/million PUTs and $0.40/million GETs on S3 in addition to the bandwidth and storage you use. S3 objects are also immutable. Once they’re written, they can’t be updated. A read-only version of this might be useful, but probably wouldn’t work in-place. Something that might be of interest is S3 SELECT support that lets you query a single (optionally compressed) CSV, JSON or Parquet file server-side at the same cost of a regular S3 GET. https://docs.aws.amazon.com/AmazonS3/latest/dev/selecting-content-from-objects.html https://docs.aws.amazon.com/AmazonS3/latest/dev/selecting-co... And if you really want relational (i.e. JOINs, aggregations and sub-queries) semantics on a bucket full of CSV, JSON, Parquet, ORC or regular-expression-describable files in a cost effective way that has great performance on buckets containing 100’s of TBs of data, definitely look at Athena which is only $5/TB of data scanned during a query.
deleted 7y ago
[deleted]
nicholaides 7y ago
I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are just files that are easy to copy, store, version, etc and don't require a centralized service). I'm not saying ALWAYS use SQLite for these cases, but in the right scenario it can simplify things significantly. Another similar use case would be AI/ML models that require a bunch of data to operate (e.g. large random forests). If you store that data in Postgres, Mongo or Redis, it becomes hard to ship your model alongside with updated data sets. If you store the data in memory (e.g. if you just serialize your model after training it), it can be too large to fit in memory. SQLite (or other embedded database, like BerkleyDB) can give the best of both worlds-- fast random access, low memory usage, and easy to ship.
- at_a_remove 7y ago
  Using SQLite in my ETL processes is something I have done for over a decade. It's just so convenient and, at the end, I have this file that can be examined and queried to see where something might have gone wrong. All of my "temporary" tables are right there for me to look at. It is wonderful!
  - chrisweekly 7y ago
    Yes! Along these lines I heartily recommend `lnav` ^1, a fantastic, lightweight, scriptable CLI mini-ETL tool w embedded sqlite engine, ideally suited for working with moderately-sized data sets (ie, millions of rows not billions) ... so useful! 1. https://lnav.org https://lnav.org
    - tenken 7y ago
      I have used it to inspect say the history of a users' requests on a load-balanced server. I like to permanently store the results of the logfile excerpt to a DB table for posterity and future reporting. Figuring out how to enter "sql" mode in lnav, generate a logfile table, and then persist it from an in-memory sqlite db to a saved-to-disk sqlite db .... was frustratingly annoying. It boils down to: :create-logline-table custom_log ;ATTACH DATABASE `test02.db` AS bkup; ;create table bkup.custom_log as select * from custom_log; ;detach database bkup; if i recall you cannot call sqlite commands ".backup" or similar in lnavs sql mode. So lnavs interjection into the sqlite command processing is annoying (I'm actually very familiar with sqlite).
cle 7y ago
> It is important to understand these two different definitions for "serverless". When a database claims to be "serverless", be sure to discern whether they mean "classic serverless" or "neo-serverless". It's really not important to understand that distinction, because this author seems to be the only one making it. Everyone knows what "serverless" means at this point, and it's not an embedded DB.
- sgt 7y ago
  After this article has been uncovered by HN, that may just be about to change.
- 0xff00ffee 7y ago
  "neo-serverless" LOL GTFOUTTAHERE! Such an arrogant programmer. I see this on HN every now and then: a programmer discovers something and thinks "I bet I'm first to notice this!", then proceeds to invent new lexicon to feel special.
- hinkley 7y ago
  Meh. If we allow serverless to make REST calls, is accessing a file system any different? I had more trouble with the assertion that the embedded DB would be maintenance-free. I started a top level comment asking for someone to explain how that would work. The part of my brain that protects me from scams is screaming “something for nothing”.
  - catalogia 7y ago
    > "I had more trouble with the assertion that the embedded DB would be maintenance-free. I started a top level comment asking for someone to explain how that would work." When is the last time a certed-up DBA had to do any maintenance on your Firefox install's places.sqlite database? That's what's meant by maintenance free; you can reasonably employ sqlite databases on users' computers, without users having the foggiest idea of what a database even is.
- skrebbel 7y ago
  I agree! Also the cryptography people should stop calling their hobby "crypto", everybody knows crypto means bitcoin and stuff. Snobs. (here's SQLite's "Serverless" page the way it was in 2007: https://web.archive.org/web/20071115173112/https://www.sqlite.org/serverless.html https://web.archive.org/web/20071115173112/https://www.sqlit...)
hinkley 7y ago
> Neo-Serverless: The database engine runs in a separate namespace from the application, probably on a separate machine, but the database is provided as a turn-key service by the hosting provider, requires no management or administration by the application owners, and is so easy to use that the developers can think of the database as being serverless even if it really does use a server under the covers. I’m having a failure of imagination here. What would I use a database for that I could reasonably assert to my peers and superiors that no maintenance whatsoever is required? Backups and restores count as maintenance. Multi region is now common, if not pervasive. Are there public datasets that are so common that it would be worth it to provide it as a service? What other service would behave like S3 but look like SQLite? I get that one might be safe to assume that “serverless” isn’t just pure functions. It could reach out to other services that are not serverless and still not consume (further) resources on a set of machines while not in use. But a severless database... I’d have to have something aggressively read-mostly, written to S3 at intervals and read from serverless processes. But is that a new thing or reading data from S3?
macdice 7y ago
"There's no server process" vs "there's no server on your itemised bill".
dacort 7y ago
SQLite is amazingly prevalent as well. Your own phone probably has hundreds of SQLite databases on it. One challenge with SQLite, though, is you have to download the whole thing to make use of it. Shameless plug, but I made a fun side project that allows Amazon Athena to read SQLite databases from S3. https://github.com/dacort/athena-sqlite https://github.com/dacort/athena-sqlite
chmaynard 7y ago
I don't think we fully appreciate just how pervasive SQLite is in our computing devices. Apple recognized its power and utility and embedded it in macOS and iOS, and Google apparently followed suit in Android. The universality of SQLite is due in no small measure to its stellar quality and the trustworthiness of its author, D. Richard Hipp.
pachico 7y ago
I currently use it to build a read-only rest service for GeoNames gazetteer. Each instance contains a copy of the database which means its really linearly scalable.
cluoma 7y ago
I use SQLite as a backend for my website and absolutely love it. Not having to setup a full dbms just for a small blog is a dream, and backups are as easy as copying a file.
zubairq 7y ago
Yep SQLite is in fact serverless. We based our open source web based IDE on SQLite and it works fantastically. The best part is that you can even take a SQLite database and use sql.js and run it offline in the browser!

501 points by alexellisuk 7y ago

Choose your own definitions... but SQLite is not serverless by common parlance. Author creates two definitions for serverless which don't match the common usage. Serverless is more about DevOps / deploy experience than how the program leverages OS processes internally. Apparently MS and AWS are ISPs? Maybe SQLite could be serverless if you defined it as incapable or running as a server on its own?

gpvos 7y ago
Serverless used to have a very clear definition: not having or using a server, so SQLite is a perfect example of serverlessness. I was extremely confused when the newfangled definition (called neo-serverless on TFA) showed up, or your definition for that matter. Who ever thought up theses confusing meanings for a word that used to be perfectly clear? You don't need an ISP to have a server. Any computer or program that listens to a network port is a server.
- aksss 7y ago
  Or is it any program that _responds_ to comms on a network port. The "serve" part of "server" ;)
forsakenharmony 7y ago
MS and AWS are indeed ISPs, but not because they have servers. They have their own IPs and global networking infrastructure

manigandham 7y ago

Why even write this? It's yet another convoluted use of the term and meaningless for SQLite of all things. Re: downvotes - what are people disagreeing with? That the term is not convoluted? That it's actually useful? That it helps to have more sub-definitions in an industry known for overloaded terms? I guarantee not a single person here has used "classic serverless" over "in-process" or "embedded" in their entire career.

NhanH 7y ago
I have seen that page for around 10 years or so (except the section that was added in 2018). The term has a very clear meaning and consistent uses within the docs (and consistent with my common sense understanding of the word too). It is not SQLite's fault that AWS and co muddles the water with their usage of the term
- IAmEveryone 7y ago
  It's not their fault. It's not anybody's fault. Language evolves. But if you wouldn't write it this way today, you should just change it, instead of drawing your readers into a pointless fight over semantics.
kick 7y ago
The article is from 2007, and explicitly calls out that the "neo" usage of the term is different than the original. If you scroll to the fifth line or so, you'll find the update. I didn't have to scroll on my device, though i can imagine there are some smaller devices that may not have that line on the screen when opening the page.
- manigandham 7y ago
  Where does it say 2007? Regardless it's vague and irrelevant material. Classic serverless has always been known as "in-process". The attempt at differences seems like adding marketing fluff rather than just removing the page entirely.
  - XelNika 7y ago
    It doesn't say 2007 anywhere, but the page is clearly at least that old. https://web.archive.org/web/20071115173112/https://www.sqlite.org/serverless.html https://web.archive.org/web/20071115173112/https://www.sqlit... I also found material from 2004 with the term. https://www.tcl.tk/community/tcl2004/Presentations/D.RichardHipp/slides/slides-all.html https://www.tcl.tk/community/tcl2004/Presentations/D.Richard... > Regardless it's vague and irrelevant material It's not vague at all, the term makes sense to distinguish "in-process" from client/server models. The page also includes an explanation on the very next line. > Adding an original timestamp to the page would've been 100x more helpful Not everyone reading about a database product is a developer. A timestamp would not help anyone unfamiliar with the history of the serverless term. The alternative is to rewrite all instances of "serverless" in the documentation which is a waste of time. Modern "serverless" is a stupid buzzword and this page clearly serves as a protest.

Eikon 7y ago

> SQLite is an example of a classic serverless database engine. With SQLite, there are no other processes, threads, machines, or other mechanisms (apart from host computer OS and filesystem) to help provide database services or implementation. There really is no server. Well, it's bending the overall consensus defining serverless as a managed / and or stateless service. I never saw anyone using the term "serverless" to mean "embeded". Using this definition, anything and everything that is not requiring a specific server to be served / distributed can be described as serverless.

IAmEveryone 7y ago
There's a section added in 2018 to deal with the apparent confusion. It'd be a better idea to just delete the page. It may have been written long before the meaning changed, but it's pointless to fight a losing and completely insignificant battle over language.
- chungy 7y ago
  Hard disagree. The "neo-serverless" version has always been extremely confusing to me. I expect serverless to mean the absence of a server. In SQLite's particular case, it's subverting the expectation that has persisted since the beginning of time (of databases) that a database must be managed by a server.
rumanator 7y ago
> Well, it's bending the overall consensus defining serverless as a managed / and or stateless service. That assertion is quite the stretch because there is no consensus on what serverless actually means. The only thing that exists is that the concept of function-as-a-service is being forced as a placeholder for serverless, but some vendors try to manipulate the definition to include their managed services offerings.
masklinn 7y ago
The SQLite page dates back to 2007 at least: https://web.archive.org/web/20071115173112/https://www.sqlite.org/serverless.html https://web.archive.org/web/20071115173112/https://www.sqlit... > Using this definition, anything and everything that is not requiring a specific server to be served / distributed can be described as serverless. The distinction is useful to make when similar systems traditionally rely on a client/server model. Which many DBMS do, to say nothing of RDBMS. That it is server-less is a distinctive feature of SQLite as an RDBMS.

deleted 7y ago

[deleted]

jankotek 7y ago

I think better name (and coined since 1990ies) is embedded db engine. MySQL, Firebird (InterBase) etc can also be linked into binary and run in the same process, without any type of sockets.

tiku 7y ago

CSV is also serverless..

hestefisk 7y ago
Very serverless. The new thing.
threatripper 7y ago
But usually not multi-user-write with transaction support.
deleted 7y ago
[deleted]
bob1029 7y ago
CSV doesn't have a write-ahead log.
- inopinatus 7y ago
  The file system it is stored on does.
- masklinn 7y ago
  I don't think sqlite had WAL when that page was created.
  - threatripper 7y ago
    I think bob means transactions.
    - bob1029 7y ago
      Transactions, durability and acceptable performance with concurrent access.
masklinn 7y ago
CSV does not traditionally involve a server. RDBMS do.

oldgradstudent 7y ago

> Of those that are serverless, SQLite is the only one known to this author that allows multiple applications to access the same database at the same time. IIRC, MS Access allowed that, which explained a lot of its popularity.

EGreg 7y ago
Waaaaait wouldn’t that mean the file system is the server, with some binary API and responsible for handling concurrent access and locks for the entire file? LOL.
- simias 7y ago
  Well you effectively report some of the workload normally handled by the server to the OS's filesystem layer, that's true. In particular you rely heavily on the FS locking working correctly. Calling the FS a "server" is a bit of a stretch though.
- m-p-3 7y ago
  Serverless in this situation means that you don't really have to provision or setup an actual server to handle the database, the client itself just need the ability to read and write an SQLite file.
  - EGreg 7y ago
    Because many of the locking things are outsourced to the file system which acts as server for concurrent threads. A server is something that listens for commands from various clients and executes them.
- idclip 7y ago
  That just circle jerk. There is an agreed upon def of serverless. If we reduce things to the absurd we stop being able to reason about things. https://en.m.wikipedia.org/wiki/Serverless_computing https://en.m.wikipedia.org/wiki/Serverless_computing

geokon 7y ago

a bit tangential, but when do you move form using in-application data structures (maps, trees, vector/arrays) to using a database? Is it basically when the data doesn't fit in memory? I've been programming for almost a decade and I've never come across needing a database... (for context, it's ten years without anything web related) I'm interested in them and I'd love to learn SQL but I can't even think of a use case outside of managing user in some web based application

manigandham 7y ago
Persistence, portability, and durability. SQLite is an actual file on disk, can be easily read by multiple programs/languages, and has write-ahead logging to keep your data safe from crashes.
dragontamer 7y ago
> when do you move form using in-application data structures (maps, trees, vector/arrays) to using a database? The killer feature of databases, including no-sql or older ones like BerkeleyDB, is multiprocess synchronization. If two programs (or two copies of the same program) need to coordinate their data, a database is often far easier to use than writing your own transaction layer through files, mutexes, and other primitives. Web applications, such as forums or discussion boards, have many users and processes adding data simultaneously. That's why databases are so popular in web backends.
typon 7y ago
Any medium size or greater application either uses SQL or recreates a shitty version of it. You just can't write complex code without relations.
d0mine 7y ago
Anytime you want persistency: sqlite is a replacement for fopen() https://www.sqlite.org/whentouse.html https://www.sqlite.org/whentouse.html
shakna 7y ago
Structured transactions between applications is one place. You can get by with files, but they're slower. A DB is the right choice. Structured data anytime you're hitting multiuser - web, networked games, collab programs, live maps, etc. Anything that's massively stateful. A DB hands you a lot of guarantees. The filesystem has some of them, but is much slower.

eb0la 7y ago

Serverless, like all embedded databases ;-) In the sense you don't need to provision an additional piece of infrastructure to power your application :-)

ashildr 7y ago

I find this article valuable. I made a mental note to check if SQLite is enough every time I consider one of the “serverless” options.

k__ 7y ago

Two definitions of serverless? I heard many, but what they're writing isn't one of them. As far as I can tell SQLite was always called an embedded database, and never a serverless one.

masklinn 7y ago
The page is more that 10 years old. And it uses "serverless" in the sence of server-less aka not having a server. Which is pretty damn sensible terminology, as opposed to the newer "serverless" meaning" runs on some utility server outside of your care or control".
- k__ 7y ago
  I know, but have you anyone calling "embedded databases" "serverless databases" in the last 10 years? It's like someone found this page and felt very smart about it, because they stick it to the serverless crowd.
  - SQLite 7y ago
    "embedded database" does not imply "serverless database". Other embedded RDBMSes run servers; they just run the server as a separate thread rather than a separate process. SQLite is different in that there is no separate thread of control. SQLite runs in the same thread as the application that calls it. There is no separate thread hanging around to clean up or handle background tasks after an SQLite function call returns.
    - deleted 7y ago
      [deleted]

imafish 7y ago

My understanding of serverless = easily scalable, managed service. But somehow the word annoys people. Maybe we should find a better word?

afroboy 7y ago
serverless it mean no server but everything use servers, yep we should find a better expression.
- threatripper 7y ago
  Peer-to-peer networking in its most pure form does not need a dedicated/central server. Each machine hosts a client/server instance locally.
vbezhenar 7y ago
My understanding was that cloud = easily scalable, managed service.
- iampims 7y ago
  That was mine too. It’s possible that this previously applied to primitives (cpu, memory, network, storage) and now refers to applications (keyvalue store, SQL database, message queues, etc.)
rumanator 7y ago
> My understanding of serverless = easily scalable, managed service. There is already a term for that concept: managed services. There is no such thing as a managed service that's designed not to be scalable. Some implementations may be better at scaling than others, but that's it. The serverless buzzword is pure marketing.
- nck4222 7y ago
  On the one hand, the term serverless is really frustrating to me. On the other hand, there's a certain amount of amusement I get from seeing "the cloud" become a marketing buzzword in the mid-late '00's, and now seeing the same thing happen with "serverless." When you look at the implementation of the two "technologies", they're about 95% the same. Yet somehow they're pitched as these big revolutions. In another decade when terminals or p2p become popular again we'll be hearing about some new buzzword like "Terran" computing, or "social" architecture or something.

JDiculous 7y ago

This is not new or unknown information, why is this on the front page?

klez 7y ago
Because other people deemed it interesting.
- JDiculous 7y ago
  The fact that SQLite is serverless is common knowledge and not interesting unless you're brand new to databases

threatripper 7y ago

In case SQLite is not enough and you need redundant servers or clustering, there's also database servers that use SQLite as storage engine: http://rqlite.com/ http://rqlite.com/ https://dqlite.io/ https://dqlite.io/

Gys 7y ago
So you run an app with sqlite on one server and sync to sqlite on another server? What would be the benefit to using a separate db server like in the 'neo-serveless' setup?
- threatripper 7y ago
  No, in this case you always have to use rqlite/dqlite because they manage the network synchronization. They use SQLite as storage engine (one SQLite database per server instance).
  - Gys 7y ago
    I understand that in those cases rqlite/dqlite is used. But that it just a technical detail. My point is that I am running two servers: one with the app and Xqlite and another one with Xqlite. In case of a neo-serveless setup, I also have two servers: one with the app, the other one with the db server. So what are the benefits of the Xqlite setup? I looked into that before and for one thing, Xqlite is slower (obviously) then just sqlite. So speed is not a key benefit. I also will have to manage both servers myself. At least for a seperate db server I have the benefit that I can buy that as a service, incl management, backups and such.
    - tracker1 7y ago
      Not the author, or knowing of all the technical details... simplistic replication structure and redundancy/failover without an expensive or more complex RDBMS solution while still self-hosting the service. There are still a lot of instances where you cannot use a cloud provider for your app or database. To be honest, I'd probably lean more towards a nosql database that has in the box, relatively easy replication strategy, though that might mean 3+ db servers for good performance. (RethinkDB, MongoDB, Cassandra/ScyllaDB, Cockroach). Just depends on the budget and resources.

Gys 7y ago

The main problem that I see for using sqlite is exactly for it being 'Classic Serverless'. Because how does one keep an up te date backup? Deploying to Heroku, Dokku, AWS Lambda and such means the sqlite file will be lost on a crash or new deploy. Even a VM can crash. Export to S3 on every write? Maybe if changes do not happen often, so only for specific use cases (and actually I think you should just generate static html in that case). I use sqlite for local tests for cases where the live app has a 'neo-serverless' database. It is very, very fast so the tests run almost instantly.

coleifer 7y ago
>how do you back up a file
- wongarsu 7y ago
  It's a valid concern in so far as that any standalone database setup you would use at a serverless provider has backups already taken care of (for example AWS Lambda with AWS RDS has backups built into RDS). If I deploy SQLite instead I have to take care of backups myself, and that might not be trivial to get right.
hashhar 7y ago
It's just a single file. Easier to backup than any other database. Also I don't think any cloud provider/database provides an always up to date backup other than a standby replica (which isn't also a backup exactly).
- wongarsu 7y ago
  If I remember the "ways to corrupt SQLite databases" page discussed a few days ago, backing up SQLite isn't entirely trivial. By default there are up to three files that have to be copied simultaneously, else you risk corruption. The optimum is to run the backup command to create a copy, but that requires realizing this exists.
  - 7y ago

greatjack613 7y ago

This document is golden, serverless means an application reading and writing to the database in the process directly

dana321 7y ago

It does allow multiple applications to access the same database at the same time, but when you do so it really hurts performance. I noticed this when i wrote a web crawler in go and used sqlite as the backend. As soon as i connect using the command line interface, it slows down significantly. Just something to bear in mind if you want to use it with multiple processes!

iampims 7y ago
Even with WAL enabled?
- FpUser 7y ago
  Let's say there are concurrent writes to a single table with few indexes. Process of updating said table and indices eventually boils down to update of a single B+ tree from a multiple threads. In general until serialized it is physically impossible without corrupting the tree. Sure there might be other specially crafted implementation of table more friendly to concurrent writes but there will be trade-offs. There are no miracles in the world. As for particular case with WAL, all it does in this particular scenario is act as a queue. If your database load is spiky then it can even out load and make an impression of faster response. Under constant load it will not speed up the things and will internally serialize all actual updates
  - catalogia 7y ago
    If you need concurrent/parallel writes, SQLite is not the right tool for you. You may as well lament that hammers are no good for driving screws.
    - FpUser 7y ago
      Can you please point me to where did I say anything about SQLite being right tool for concurrent write? My point was that true concurrent writes to a generic database table with indexes are are physically impossible in a general case disregarding what database it is.

bullen 7y ago

I integrated my database with the app server, but I still open a socket internally to access it. So my system is also "serverless" in that meaning. Acronyms used to be confusing, this is just ridiculous. http://root.rupy.se http://root.rupy.se

WolfOliver 7y ago

I like the definition of classical- and neo- serverless. Does anyone knows more articles about serverless in terms of classical serverless?

peterwwillis 7y ago

"Microsoft Azure Cosmo DB and Amazon S3 are examples of a neo-serverless databases." Can we back away a bit from the bandwagoning of misused terminology? Serverless literally means "running your apps on somebody else's server". S3 is not a server you run your apps on, it is SaaS that you manipulate through an API - you don't put your apps on it. If S3 is serverless, then literally every network service of any kind is serverless.

tracker1 7y ago
I always preferred XaaS nomenclature myself. "XaaS with autoscaling"... is probably better than "Serverless"

mettamage 7y ago

Does this mean that you can hack some database storage (w/ sqlite) together on frontend only hosting platforms like Github or Netlify? I think not, but I wonder if some hack is available by virtue of it simply being a file that you can read (and somehow) write to. The best I came up with: let's say you have a toy project, and you call the Github API and replace the file upon every write. Implementing a read is easier as you know where the file is located. This hack shows that you somehow need write access to get any form of performance out of it, because this hack is super slow.

aikah 7y ago
> Does this mean that you can hack some database storage (w/ sqlite) together on frontend only hosting platforms like Github or Netlify? You can run SQLlite in the client via webassembly therefore open a SQLite file in the browser to query it yes, you just can't write anything in it and expect it to persist somehow on the static hosting service itself. > The best I came up with: let's say you have a toy project, and you call the Github API and replace the file upon every write. Implementing a read is easier as you know where the file is located. This hack shows that you somehow need write access to get any form of performance out of it, because this hack is super slow. No you'll need a server for that, you can't make random HTTP requests to any server in the browser, because of CORS/SAME ORIGIN policies. "SQLite is serverless" is meaningless buzzword. It just means that SQLite is equivalent a flat file where you'd shove some data, just that you can use SQL to query that file instead of having to index data in it yourself.
- giancarlostoro 7y ago
  You could make it persist if you have it uploaded somewhere. But then I guess thats not serverless.
  - Frost1x 7y ago
    "Serverless" is a garbage marketing term in general. It sounds sexy to nontechnical management folks who are used to hearing a bunch of expensive costs and the word "server" associated with them in some fashion. From that view, anything that gets rid of those pesky "servers" sounds like a win.

m3kw9 7y ago

I don’t get it, then any db can be classic serverless if they run in the same computer as the app?

flatline 7y ago
No, there’s still some form of IPC going on then. SQLite is an embedded database - everything happens in the memory space of the same application. Of course you can still throw a web front-end in front of your application but there’s still no extra hop to the database, per se.
jasode 7y ago
>any db can be classic serverless if they run in the _same computer_ as the app? Not just the "same computer" ... it's the same process id (PID). Extract of relevant text from that webpage: >Classic Serverless: The database engine runs within the same process, thread, and address space as the application. There is no message passing or network activity. In other words, when you compile and link "sqlite.c" into your own executable, the same PID (process) that handles text input and paints pixels on the screen -- is the same PID that writes to the sqlite database file. It's all the same process. That's what they mean by "classic serverless". In contrast, if you make a Go executable that writes to MySQL/PostgreSQL db and make them both run on the same physical computer, that's not "classic serverless". It's because when you enter "ps -aux" to list all running processes, you see separate PIDs for the Go executable and the MySQL db engine. Other jargon used might be "in-process" vs "out-of-process" or "embedded" vs "external". SQLite is sometimes characterized as "in-process embedded database" but MySQL is an "out-of-process" db.
l34ch 7y ago
Nope, only SQLite doesn't use another process as an intermediary to read/write to the DB.
manigandham 7y ago
Yes, technically, although they make the distinction is that it's in the same process. It's a bad definition. I would stick with the terms "embedded" or "in-process" which have been around for decades and are well-understood.

loukrazy 7y ago

Isn’t it easier to say “embedded” and “severless”

manigandham 7y ago
Not in this thread apparently. I guarantee nobody here as ever used the term "serverless" over "embedded" or "in-process" in their entire career but apparently the purity and nostalgia of SQLite overrides it.
- filleduchaos 7y ago
  This article was written in 2007, so yes, people have been using the word "serverless" before managed service providers decided to take it up as a buzzword.
  - manigandham 7y ago
    Nobody used it before. Even the about page says: "SQLite is an embedded SQL database engine." "serverless" is a marketing term, then and now. Not sure why there's such a big defense of it. If you have to argue this much over the provenance of a term that clearly isn't used the same way today then it's a good sign that it's not very useful.
    - catalogia 7y ago
      > Nobody used it before. The archive.org links already provided to you prove that you're flat wrong about that. SQLite used the term many years before it was a webshit buzzword. They used it in the intuitive straight forward english sense; somebody who has no hat is hatless, somebody who has no home is homeless, and a database system that has no server process is serverless. In 2007 when it was written, nobody would have batted at eye at this term, the meaning would have been immediately clear to anybody who had any familiarity with databases.
      - manigandham 7y ago

bane 7y ago

I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/write methods. SQLite is not really any different, except the read/write work is already done, and you use SQL to format the data values and encode the read/write logic instead of the language you are working in. The file format has some cool extensions like text indexing, geospatial, etc. But it's no more a RDBMS than reading and writing a JSON file is. "But there's indexes!" Yes, just liked you might build an index on your JSON file and read that before reading the JSON file to know where things are faster -- and then you have to write all the code to do that. SQLite is just a file format, where you can also build indexes and all the code for that is already done for you. It's just a file format with an API that's similar to the ones you'd use for any regular old RDBMS and uses SQL as the domain language to read/write data. It's just a file format. Anything you can do/can't do with a file format you do with SQLite. It's just a file format. A nice convenient one that you are probably better off using than most other things for most purposes. It's just a file format. edit I'm glad this comment is getting such a response. I'm not trying to be mean, just help clarify thinking. Here's two thought experiments: 1) If SQLite didn't require you to use SQL as the read/write logic and was called "datalite", and instead just forced you to use C function calls, exactly like you would if you were working with literally any other file format on the planet, would you still be confused as to what it is? 2) Do you consider reading and writing to any other file format anywhere in the hierarchy of RDBMSs? Consider Python's csv module. Is that an RDBMS? Let's move away from tabular data, how about python-docx?

lucideer 7y ago
I'm not sure I understand your comment. What's the signifigance of it being "just a file format"? What are the aspects of "approaching it from a classic RDBMS angle" that are incompatible with it being "just a file format"? I've always seen it as "just a file format" myself, and I think I've always approached it "from a classic RDBMS angle", but I've never felt confused, and I don't see what I'm overcomplicating.

pm90 7y ago

What I really want: SQLite storage backend driver for s3/gcs. No need for disks then. I haven’t been able to find such a solution though; and am not technically proficient enough in C (the Lang SQLite is written in) to do so myself.

bshipp 7y ago
that would be cool! as an interim step, if your data is small enough, perhaps you could run an in-memory sqlite db and periodically backup to a permanent s3 file? APSW exposes the sqlite backup api so you could do them online without shutting down the database.
tn1 7y ago
The challenge with this is that S3 and friends are object stores, meaning you upload or download the whole file each time. As you can imagine, this will cost you tremendous bandwidth even to insert one row. Furthermore, it doesn't solve the multiple writers problem, because (afaik) there's no way to lock a file on S3.
- pm90 7y ago
  > As you can imagine, this will cost you tremendous bandwidth even to insert one row. Is this the cost of network transfers changed by AWS or GCP? What if I'm hosting my app on e.g. EKS or GKE respectively?
  - kondro 7y ago
    It costs roughly $5/million PUTs and $0.40/million GETs on S3 in addition to the bandwidth and storage you use. S3 objects are also immutable. Once they’re written, they can’t be updated. A read-only version of this might be useful, but probably wouldn’t work in-place. Something that might be of interest is S3 SELECT support that lets you query a single (optionally compressed) CSV, JSON or Parquet file server-side at the same cost of a regular S3 GET. https://docs.aws.amazon.com/AmazonS3/latest/dev/selecting-content-from-objects.html https://docs.aws.amazon.com/AmazonS3/latest/dev/selecting-co... And if you really want relational (i.e. JOINs, aggregations and sub-queries) semantics on a bucket full of CSV, JSON, Parquet, ORC or regular-expression-describable files in a cost effective way that has great performance on buckets containing 100’s of TBs of data, definitely look at Athena which is only $5/TB of data scanned during a query.

deleted 7y ago

[deleted]

nicholaides 7y ago

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are just files that are easy to copy, store, version, etc and don't require a centralized service). I'm not saying ALWAYS use SQLite for these cases, but in the right scenario it can simplify things significantly. Another similar use case would be AI/ML models that require a bunch of data to operate (e.g. large random forests). If you store that data in Postgres, Mongo or Redis, it becomes hard to ship your model alongside with updated data sets. If you store the data in memory (e.g. if you just serialize your model after training it), it can be too large to fit in memory. SQLite (or other embedded database, like BerkleyDB) can give the best of both worlds-- fast random access, low memory usage, and easy to ship.

at_a_remove 7y ago
Using SQLite in my ETL processes is something I have done for over a decade. It's just so convenient and, at the end, I have this file that can be examined and queried to see where something might have gone wrong. All of my "temporary" tables are right there for me to look at. It is wonderful!
- chrisweekly 7y ago
  Yes! Along these lines I heartily recommend `lnav` ^1, a fantastic, lightweight, scriptable CLI mini-ETL tool w embedded sqlite engine, ideally suited for working with moderately-sized data sets (ie, millions of rows not billions) ... so useful! 1. https://lnav.org https://lnav.org
  - tenken 7y ago
    I have used it to inspect say the history of a users' requests on a load-balanced server. I like to permanently store the results of the logfile excerpt to a DB table for posterity and future reporting. Figuring out how to enter "sql" mode in lnav, generate a logfile table, and then persist it from an in-memory sqlite db to a saved-to-disk sqlite db .... was frustratingly annoying. It boils down to: :create-logline-table custom_log ;ATTACH DATABASE `test02.db` AS bkup; ;create table bkup.custom_log as select * from custom_log; ;detach database bkup; if i recall you cannot call sqlite commands ".backup" or similar in lnavs sql mode. So lnavs interjection into the sqlite command processing is annoying (I'm actually very familiar with sqlite).

cle 7y ago

> It is important to understand these two different definitions for "serverless". When a database claims to be "serverless", be sure to discern whether they mean "classic serverless" or "neo-serverless". It's really not important to understand that distinction, because this author seems to be the only one making it. Everyone knows what "serverless" means at this point, and it's not an embedded DB.

sgt 7y ago
After this article has been uncovered by HN, that may just be about to change.
0xff00ffee 7y ago
"neo-serverless" LOL GTFOUTTAHERE! Such an arrogant programmer. I see this on HN every now and then: a programmer discovers something and thinks "I bet I'm first to notice this!", then proceeds to invent new lexicon to feel special.
hinkley 7y ago
Meh. If we allow serverless to make REST calls, is accessing a file system any different? I had more trouble with the assertion that the embedded DB would be maintenance-free. I started a top level comment asking for someone to explain how that would work. The part of my brain that protects me from scams is screaming “something for nothing”.
- catalogia 7y ago
  > "I had more trouble with the assertion that the embedded DB would be maintenance-free. I started a top level comment asking for someone to explain how that would work." When is the last time a certed-up DBA had to do any maintenance on your Firefox install's places.sqlite database? That's what's meant by maintenance free; you can reasonably employ sqlite databases on users' computers, without users having the foggiest idea of what a database even is.
skrebbel 7y ago
I agree! Also the cryptography people should stop calling their hobby "crypto", everybody knows crypto means bitcoin and stuff. Snobs. (here's SQLite's "Serverless" page the way it was in 2007: https://web.archive.org/web/20071115173112/https://www.sqlite.org/serverless.html https://web.archive.org/web/20071115173112/https://www.sqlit...)

hinkley 7y ago

> Neo-Serverless: The database engine runs in a separate namespace from the application, probably on a separate machine, but the database is provided as a turn-key service by the hosting provider, requires no management or administration by the application owners, and is so easy to use that the developers can think of the database as being serverless even if it really does use a server under the covers. I’m having a failure of imagination here. What would I use a database for that I could reasonably assert to my peers and superiors that no maintenance whatsoever is required? Backups and restores count as maintenance. Multi region is now common, if not pervasive. Are there public datasets that are so common that it would be worth it to provide it as a service? What other service would behave like S3 but look like SQLite? I get that one might be safe to assume that “serverless” isn’t just pure functions. It could reach out to other services that are not serverless and still not consume (further) resources on a set of machines while not in use. But a severless database... I’d have to have something aggressively read-mostly, written to S3 at intervals and read from serverless processes. But is that a new thing or reading data from S3?

macdice 7y ago

"There's no server process" vs "there's no server on your itemised bill".

dacort 7y ago

SQLite is amazingly prevalent as well. Your own phone probably has hundreds of SQLite databases on it. One challenge with SQLite, though, is you have to download the whole thing to make use of it. Shameless plug, but I made a fun side project that allows Amazon Athena to read SQLite databases from S3. https://github.com/dacort/athena-sqlite https://github.com/dacort/athena-sqlite

chmaynard 7y ago

I don't think we fully appreciate just how pervasive SQLite is in our computing devices. Apple recognized its power and utility and embedded it in macOS and iOS, and Google apparently followed suit in Android. The universality of SQLite is due in no small measure to its stellar quality and the trustworthiness of its author, D. Richard Hipp.

pachico 7y ago

I currently use it to build a read-only rest service for GeoNames gazetteer. Each instance contains a copy of the database which means its really linearly scalable.

cluoma 7y ago

I use SQLite as a backend for my website and absolutely love it. Not having to setup a full dbms just for a small blog is a dream, and backups are as easy as copying a file.

zubairq 7y ago

Yep SQLite is in fact serverless. We based our open source web based IDE on SQLite and it works fantastically. The best part is that you can even take a SQLite database and use sql.js and run it offline in the browser!