3 ms·

From a cursory look at your source code… It looks like memory is also not contiguous in your implementation. You don't support concurrency at all (xLock/xUnlo

From a cursory look at your source code… It looks like memory is also not contiguous in your implementation. You don't support concurrency at all (xLock/xUnlock/xCheckReservedLock are noops). I'm not sure how APSW handles journaling (you usually want the journal_mode to be memory in these cases, but to force that, you must return SQLITE_OPEN_MEMORY as an OutFlag from xOpen). Otherwise, you need to deal with files (like journals, etc) for which reads/writes are not page aligned. There is one caveat, though: if you want to “deserialize” a WAL database with journal_mode memory, you must convert it to rollback before opening (but you can do that by editing bytes 18 and 19).

michalc 3y ago
> and the associated sqlite3_serialize and sqlite3_deserialize APIs So... I have to admit I missed the existence of these. So for many cases, using sqlite3_deserialize is probably a very reasonable choice over sqlite-memory-vfs Still, as you say, sqlite-memory-vfs doesn't require the memory to be contiguous, unlike sqlite3_deserialize > You don't support concurrency at all (xLock/xUnlock/xCheckReservedLock are noops). Well, you can read concurrently, just not write
- ncruces 3y ago
  If you're interested, both SQLite's and my memdb VFSes implement safe locking. Depending on your familiarity with Go, mine maybe easier to follow, or not. https://github.com/ncruces/go-sqlite3/blob/f1b00a9944730eaa9da28a7c967658ed2426d48d/vfs/memdb/memdb.go#L181-L257 https://github.com/ncruces/go-sqlite3/blob/f1b00a9944730eaa9...
  - michalc 3y ago
    Ah thanks for this (maybe not so much for the code itself admittedly, but for the push to implement locking) So after a few attempts, I _think_ I have something reasonable: https://github.com/michalc/sqlite-memory-vfs/blob/64a55c349173794f37230f1c101514e8964da669/sqlite_memory_vfs.py#L111-L140 https://github.com/michalc/sqlite-memory-vfs/blob/64a55c3491... (I'm still a but stuck on PENDING I have to admit... not really sure how that fits in)
    - ncruces 3y ago
      Pending exists to prevent writer starvation: if there's always at least one reader, no one can ever write. You take pending before exclusive (which should always be possible because you have reserved by then), and you should check pending before taking a shared lock (no need to check exclusive, because they got pending before exclusive). Shared forces writers to wait; Reserved prevents new readers from upgrading to write; Pending prevents new readers from starting; and exclusive forces everyone else to wait.
      - michalc 3y ago
        Have now implemented a version of this at https://github.com/michalc/sqlite-memory-vfs/blob/74a24ce567557d72eebf73fbdf6d8e609aedafe3/sqlite_memory_vfs.py#L112-L155 https://github.com/michalc/sqlite-memory-vfs/blob/74a24ce567... I think different to yours in two ways: - If doesn't go via PENDING if it doesn't need to. Not sure what's best really - but since RESERVED can already be skipped in some cases (recovering from a hot-journal) then so far happy with "skipping is a thing", and PENDING is private to the VFS anyway from what I can tell, so nothing else should care I think - It doesn't do the blocking in the lock function, but depends on SQLite's core code doing it via the busy handler/timeout

michalc 3y ago

> and the associated sqlite3_serialize and sqlite3_deserialize APIs So... I have to admit I missed the existence of these. So for many cases, using sqlite3_deserialize is probably a very reasonable choice over sqlite-memory-vfs Still, as you say, sqlite-memory-vfs doesn't require the memory to be contiguous, unlike sqlite3_deserialize > You don't support concurrency at all (xLock/xUnlock/xCheckReservedLock are noops). Well, you can read concurrently, just not write

ncruces 3y ago
If you're interested, both SQLite's and my memdb VFSes implement safe locking. Depending on your familiarity with Go, mine maybe easier to follow, or not. https://github.com/ncruces/go-sqlite3/blob/f1b00a9944730eaa9da28a7c967658ed2426d48d/vfs/memdb/memdb.go#L181-L257 https://github.com/ncruces/go-sqlite3/blob/f1b00a9944730eaa9...
- michalc 3y ago
  Ah thanks for this (maybe not so much for the code itself admittedly, but for the push to implement locking) So after a few attempts, I _think_ I have something reasonable: https://github.com/michalc/sqlite-memory-vfs/blob/64a55c349173794f37230f1c101514e8964da669/sqlite_memory_vfs.py#L111-L140 https://github.com/michalc/sqlite-memory-vfs/blob/64a55c3491... (I'm still a but stuck on PENDING I have to admit... not really sure how that fits in)
  - ncruces 3y ago
    Pending exists to prevent writer starvation: if there's always at least one reader, no one can ever write. You take pending before exclusive (which should always be possible because you have reserved by then), and you should check pending before taking a shared lock (no need to check exclusive, because they got pending before exclusive). Shared forces writers to wait; Reserved prevents new readers from upgrading to write; Pending prevents new readers from starting; and exclusive forces everyone else to wait.
    - michalc 3y ago
      Have now implemented a version of this at https://github.com/michalc/sqlite-memory-vfs/blob/74a24ce567557d72eebf73fbdf6d8e609aedafe3/sqlite_memory_vfs.py#L112-L155 https://github.com/michalc/sqlite-memory-vfs/blob/74a24ce567... I think different to yours in two ways: - If doesn't go via PENDING if it doesn't need to. Not sure what's best really - but since RESERVED can already be skipped in some cases (recovering from a hot-journal) then so far happy with "skipping is a thing", and PENDING is private to the VFS anyway from what I can tell, so nothing else should care I think - It doesn't do the blocking in the lock function, but depends on SQLite's core code doing it via the busy handler/timeout