3 ms·

I cannot even ssh into the server after trying to use DuckDB. It is completely dead (with all the ducks, what a misery). The reason is probably that it's using

I cannot even ssh into the server after trying to use DuckDB. It is completely dead (with all the ducks, what a misery). The reason is probably that it's using a full index, in contrast with the sparse index in ClickHouse, and maybe it's trying to build it in memory, going to swap (the server has 32 GB memory).

zX41ZdbW 4y ago
Interesting - the database file looks ok, but the data is lost (the table is empty): ubuntu@ip-172-31-3-138:~$ ls -l total 69561648 -rw-rw-r-- 1 ubuntu ubuntu 17631031296 Dec 16 23:57 my-db.duckdb -rw-rw-r-- 1 ubuntu ubuntu 326 Dec 16 23:53 my-db.duckdb.wal -rw-rw-r-- 1 ubuntu ubuntu 16257755606 Jan 21 2022 pwned-passwords-sha1-ordered-by-hash-v8.7z -rw-rw-r-- 1 ubuntu ubuntu 37342268646 Dec 2 2021 pwned-passwords-sha1-ordered-by-hash-v8.txt ubuntu@ip-172-31-3-138:~$ python3 Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import duckdb >>> con = duckdb.connect(database='my-db.duckdb') >>> con.execute("SELECT count(*) FROM passwords").fetchall() [(0,)]
- ignoramous 4y ago
  Curious: Are you affiliated with ClickHouse or any other Columnar DB project in any way? If so, you may want to add that as a disclosure.
  - deleted 4y ago
    [deleted]
  - zX41ZdbW 4y ago
    Yes, I'm working on ClickHouse, here is my GitHub profile: https://github.com/alexey-milovidov https://github.com/alexey-milovidov I'm also trying to follow every existing technology in the data engineering space :)
- mytherin 4y ago
  Because DuckDB uses ACID [1] data is loaded in an all-or-nothing manner. As the load was interrupted due to the system running out of memory, the table is expected to be empty. [1] https://en.wikipedia.org/wiki/ACID https://en.wikipedia.org/wiki/ACID