5 ms·

It is an intrinsic property of database engines. Database engines access the fs. That is the slowest part of memory in a computer. The web app is stateless, it'

It is an intrinsic property of database engines. Database engines access the fs. That is the slowest part of memory in a computer. The web app is stateless, it's never suppose to touch the fs. Additionally database engines usually exist as separate processes or servers necessitating need for ipc which is another intrinsic bottleneck. If an AWS vm can support 10 million writes/sec to disk storage then a stateless web app on the same vm should be even faster doing the same reads and writes to in memory storage. I can agree with your latter statement that io can become the bigger bottleneck these days.

jandrewrogers 3y ago
You do not understand how modern databases actually work. Databases don't need to access the filesystem, fast ones work with the block devices directly. The slowest part of modern hardware is the network, not the storage. IPC performance isn't a real issue. Databases do their own I/O and execution scheduling, that is how they achieve extremely high write throughput. Unless your web app is designed this way (it isn't), you cannot achieve anything close to the throughput of a modern database engine. Performance is a product of software architecture. Being stateless does not make your software fast in any absolute sense, many stateless in-memory systems have mediocre performance. A disk-backed design with superior architecture can out-perform an in-memory design, both in theory and practice.
- corethree 3y ago
  DBMSs do not bypass filesystem. If this was the case, the table names would not be case-insensitive under Windows and case-sensitive under Linux (in MySQL). What they do is to allocate large space on the file system (by the way, the data is still visible as a file / set of files in the underlying operating system) and manage internal data structure. This lower the fragmentation and the overall overhead. In a similar way cache systems works - Varnish allocates entire memory it needs with a single call to operating system, then maintains internal data structure. >Unless your web app is designed this way (it isn't), you cannot achieve anything close to the throughput of a modern database engine. Your saying if I write to a section of memory on the heap or stack 4000 times with some different random byte every time it will be slower than doing the same thing to a db? Let's be real. >Performance is a product of software architecture. Being stateless does not make your software fast in any absolute sense, many stateless in-memory systems have mediocre performance. A disk-backed design with superior architecture can out-perform an in-memory design, both in theory and practice. Uh fundamental performance of software is absolutely limited by hardware. Stateless removes some fundamental limits. Sure architecture can change things but not not absolutely. Given the best architecture for all cases the best possible performance is in the end determined by which one is stateless. The absolute speed in the end is determined by hardware. SW architecture is changeable and thus NOT absolute. I would know I work in HPC. We don't have any databases in any of our processing pipelines for obvious reasons.

jandrewrogers 3y ago

You do not understand how modern databases actually work. Databases don't need to access the filesystem, fast ones work with the block devices directly. The slowest part of modern hardware is the network, not the storage. IPC performance isn't a real issue. Databases do their own I/O and execution scheduling, that is how they achieve extremely high write throughput. Unless your web app is designed this way (it isn't), you cannot achieve anything close to the throughput of a modern database engine. Performance is a product of software architecture. Being stateless does not make your software fast in any absolute sense, many stateless in-memory systems have mediocre performance. A disk-backed design with superior architecture can out-perform an in-memory design, both in theory and practice.

corethree 3y ago
DBMSs do not bypass filesystem. If this was the case, the table names would not be case-insensitive under Windows and case-sensitive under Linux (in MySQL). What they do is to allocate large space on the file system (by the way, the data is still visible as a file / set of files in the underlying operating system) and manage internal data structure. This lower the fragmentation and the overall overhead. In a similar way cache systems works - Varnish allocates entire memory it needs with a single call to operating system, then maintains internal data structure. >Unless your web app is designed this way (it isn't), you cannot achieve anything close to the throughput of a modern database engine. Your saying if I write to a section of memory on the heap or stack 4000 times with some different random byte every time it will be slower than doing the same thing to a db? Let's be real. >Performance is a product of software architecture. Being stateless does not make your software fast in any absolute sense, many stateless in-memory systems have mediocre performance. A disk-backed design with superior architecture can out-perform an in-memory design, both in theory and practice. Uh fundamental performance of software is absolutely limited by hardware. Stateless removes some fundamental limits. Sure architecture can change things but not not absolutely. Given the best architecture for all cases the best possible performance is in the end determined by which one is stateless. The absolute speed in the end is determined by hardware. SW architecture is changeable and thus NOT absolute. I would know I work in HPC. We don't have any databases in any of our processing pipelines for obvious reasons.