4 ms·

Can you qualify your statement that this is nothing? How many CPUs and cores are available? Are we talking SSDs? How much data is being inserted? What are th

by ijidak 4y ago

Can you qualify your statement that this is nothing? How many CPUs and cores are available? Are we talking SSDs? How much data is being inserted? What are the keys and index? 15,000 per second seems like plenty much in some contexts.

andrewstuart 4y ago
15K doesn't seem particularly noteworthy. I made a prototype message queue in Rust that processed about 7 million messages/second. That was running from RAM. I also had a prototype that ran from disk and it very rapidly maxed out the random write performance of the SSD. Can't remember the number but it was in the order of 15K to 30K messages a second, bottlenecked by the SSD. If the sqlite test is writing to disk it is likely also bottlenecked at the disk.
- gomezjdaniel 4y ago
  > I made a prototype message queue in Rust that processed about 7 million messages/second. That was running from RAM. Can you share the source code?
  - andrewstuart 4y ago
    It was pure garbage - a failed attempt to learn Rust.
    - re-thc 4y ago
      Need some garbage collection added?
- marginalia_nu 4y ago
  To be fair it's very easy to outperform a DBMS by not offering ACID guarantees.
- osigurdson 4y ago
  Agree. I know for Postgres, we are getting at least 50K / second sustained and not doing anything special - other than using bulk insert.
VWWHFSfQ 4y ago
The schema: CREATE TABLE IF NOT EXISTS users ( id BLOB PRIMARY KEY NOT NULL, created_at TEXT NOT NULL, username TEXT NOT NULL ); The data: let user = User { id: uuid::Uuid::new_v4(), created_at: chrono::Utc::now(), username: String::from("Hello"), }; If you can't insert 15,000 of these records per second then there's something wrong with your database. I'm aware that people are all on the SQLite hype train, but this kind of stuff is table-stakes. I really like SQLite. But this blog isn't any kind of great performance indicator for it.
- snacktaster 4y ago
  This entire post reduces to: insert into users (id, created_at, username) values ("<id>", "<created_at>", "Hello"), ...15,000 times. And we're going to be impressed with sqlite's performance?
  - Aeolun 4y ago
    Well, some people think 3 inserts per second is fast, so maybe it gives some perspective? To be fair, I wouldn’t expect those to hang around hackernews though.
- ZephyrBlu 4y ago
  I think a lot of people have forgotten or don't know how powerful CPUs are. Simple operations should be insanely fast, like easily sub-millisecond if not sub-nanosecond fast.
  - Aeolun 4y ago
    You can double the performance of this whole thing by inserting 2 values at a time.
    - eyegor 4y ago
      I don't see why you wouldn't just bundle them all into a transaction at that point. The idea of 1 insert at a time is to roughly estimate the overhead/perf in a real environment, where you're only adding 1 entry at a time.

by ijidak 4y ago

andrewstuart 4y ago

15K doesn't seem particularly noteworthy. I made a prototype message queue in Rust that processed about 7 million messages/second. That was running from RAM. I also had a prototype that ran from disk and it very rapidly maxed out the random write performance of the SSD. Can't remember the number but it was in the order of 15K to 30K messages a second, bottlenecked by the SSD. If the sqlite test is writing to disk it is likely also bottlenecked at the disk.

gomezjdaniel 4y ago
> I made a prototype message queue in Rust that processed about 7 million messages/second. That was running from RAM. Can you share the source code?
- andrewstuart 4y ago
  It was pure garbage - a failed attempt to learn Rust.
  - re-thc 4y ago
    Need some garbage collection added?
marginalia_nu 4y ago
To be fair it's very easy to outperform a DBMS by not offering ACID guarantees.
osigurdson 4y ago
Agree. I know for Postgres, we are getting at least 50K / second sustained and not doing anything special - other than using bulk insert.

VWWHFSfQ 4y ago

The schema: CREATE TABLE IF NOT EXISTS users ( id BLOB PRIMARY KEY NOT NULL, created_at TEXT NOT NULL, username TEXT NOT NULL ); The data: let user = User { id: uuid::Uuid::new_v4(), created_at: chrono::Utc::now(), username: String::from("Hello"), }; If you can't insert 15,000 of these records per second then there's something wrong with your database. I'm aware that people are all on the SQLite hype train, but this kind of stuff is table-stakes. I really like SQLite. But this blog isn't any kind of great performance indicator for it.

snacktaster 4y ago
This entire post reduces to: insert into users (id, created_at, username) values ("<id>", "<created_at>", "Hello"), ...15,000 times. And we're going to be impressed with sqlite's performance?
- Aeolun 4y ago
  Well, some people think 3 inserts per second is fast, so maybe it gives some perspective? To be fair, I wouldn’t expect those to hang around hackernews though.
ZephyrBlu 4y ago
I think a lot of people have forgotten or don't know how powerful CPUs are. Simple operations should be insanely fast, like easily sub-millisecond if not sub-nanosecond fast.
- Aeolun 4y ago
  You can double the performance of this whole thing by inserting 2 values at a time.
  - eyegor 4y ago
    I don't see why you wouldn't just bundle them all into a transaction at that point. The idea of 1 insert at a time is to roughly estimate the overhead/perf in a real environment, where you're only adding 1 entry at a time.