4 ms·

We also came around to dumping files into SQLite instead of dealing with the filesystem (and git) for the lix change control system. This article touches on pro

We also came around to dumping files into SQLite instead of dealing with the filesystem (and git) for the lix change control system. This article touches on problems we encountered: https://opral.substack.com/i/150054233/breaking-git-compatibility https://opral.substack.com/i/150054233/breaking-git-compatib.... - File locking, concurrency, etc., are problems that SQLite solves - Using SQLite allows drumroll querying files with SQL instead of platform-dependent fs APIs - Using SQL queries is typesafe with Kysely https://kysely.dev/ https://kysely.dev/ (without the need for an ORM)

hinkley 2y ago
You can add as much metadata to the files as your little heart desires.
egeozcan 2y ago
> Using SQL queries is typesafe with Kysely https://kysely.dev/ https://kysely.dev/ (without the need for an ORM) Wow, this is even better than what I've seen people do with F# type providers. Cool cool cool.
- samuelstros 2y ago
  the types kysely implemented are crazy. even custom where clause like this one [0] are typesafe :O [0] https://github.com/opral/monorepo/blob/99356e577f558f4442a95160ac5c49f6dcc2a6b7/packages/lix-sdk/src/query-utilities/is-in-simulated-branch.ts#L14 https://github.com/opral/monorepo/blob/99356e577f558f4442a95...
  - surrealize 2y ago
    Totally random, but I had a guess about the ts error. I hadn't seen kysely before, very cool! diff --git a/packages/lix-sdk/src/query-utilities/is-in-simulated-branch.ts b/packages/lix-sdk/src/query-utilities/is-in-simulated-branch.ts index 7d677477e..39502f245 100644 --- a/packages/lix-sdk/src/query-utilities/is-in-simulated-branch.ts +++ b/packages/lix-sdk/src/query-utilities/is-in-simulated-branch.ts @@ -21,10 +21,9 @@ export function isInSimulatedCurrentBranch( // change is not in a conflict eb("change.id", "not in", (subquery) => subquery.selectFrom("conflict").select("conflict.change_id").unionAll( - // @ts-expect-error - no idea why subquery .selectFrom("conflict") - .select("conflict.conflicting_change_id"), + .select("conflict.conflicting_change_id as change_id"), ), ), // change is in a conflict that has not been resolved
    - samuelstros 2y ago
      You are now forever in our git history https://github.com/opral/monorepo/commit/58734e11e51d8e20092 https://github.com/opral/monorepo/commit/58734e11e51d8e20092.... the ts-expect-error was indeed fixed by your suggestion
      - ec109685 2y ago
        Isn’t kind just overriding the type checker? Should that as be necessary?
        samuelstros 2y ago
        Yes. The runtime was not affected. Hence, the ts-expect-error. Still nice to have the ts-expect-error fixed :)
        surrealize 2y ago
        In the context of the union query, I think it makes sense. The query takes two different columns (with two different names) from the same table, and combines them together into one column. The TS error happened because the column names didn't match between the two union inputs. But they would never match without the "as".
      - surrealize 2y ago
        Cool! I'm surrealize though, sureglymop wrote a sibling comment! I should have just made a PR, lol.
      - samuelstros 2y ago
        Today i learned that HN has a nesting limit. @surrealize i fixed the username https://github.com/opral/monorepo/commit/7dc1f3c806bd89c6d682a2006e487cb4501ef655 https://github.com/opral/monorepo/commit/7dc1f3c806bd89c6d68...
        surrealize 2y ago
        Haha, thanks!
sureglymop 2y ago
Regarding your lix project, have you looked at Fossil? It seems like it could maybe do what you're trying to do with some small changes.
- samuelstros 2y ago
  Yes. We looked at all version control systems (fossil, pijul, jj, git, darc, sapling). None provide what we need: - target the browser - supports any file format, not just text files - understands changes, not just versioning of files - suited to build web apps on top - ultimately enable 1000's of artists, designers, lawyers, civil engineers, etc. to collaborate we are 2 years into the journey. dec 16, we'll release a public preview
  - deleted 2y ago
    [deleted]
  - sureglymop 2y ago
    Interesting! Definitely looking forward to that and I hope you succeed.
mmerickel 2y ago
I'm a huge proponent of using SQLite as an abstraction over a filesystem. One warning I will note though, is be aware that a SQLite database does not shrink unless you vacuum it (basically copies the data into a separate file and deletes the original). This is a manual operation you have to do at points where it makes sense within your application. So be careful with disk usage when just writing binary data and then deleting it.
- samuelstros 2y ago
  That's a good tip. I was wondering how SQLite avoids page fragmentation. The answer is "it depends," but VACUUM is to the rescue. What are your reasons for advocating for SQLite as a filesystem abstraction?

hinkley 2y ago

You can add as much metadata to the files as your little heart desires.

egeozcan 2y ago

> Using SQL queries is typesafe with Kysely https://kysely.dev/ https://kysely.dev/ (without the need for an ORM) Wow, this is even better than what I've seen people do with F# type providers. Cool cool cool.

samuelstros 2y ago
the types kysely implemented are crazy. even custom where clause like this one [0] are typesafe :O [0] https://github.com/opral/monorepo/blob/99356e577f558f4442a95160ac5c49f6dcc2a6b7/packages/lix-sdk/src/query-utilities/is-in-simulated-branch.ts#L14 https://github.com/opral/monorepo/blob/99356e577f558f4442a95...
- surrealize 2y ago
  Totally random, but I had a guess about the ts error. I hadn't seen kysely before, very cool! diff --git a/packages/lix-sdk/src/query-utilities/is-in-simulated-branch.ts b/packages/lix-sdk/src/query-utilities/is-in-simulated-branch.ts index 7d677477e..39502f245 100644 --- a/packages/lix-sdk/src/query-utilities/is-in-simulated-branch.ts +++ b/packages/lix-sdk/src/query-utilities/is-in-simulated-branch.ts @@ -21,10 +21,9 @@ export function isInSimulatedCurrentBranch( // change is not in a conflict eb("change.id", "not in", (subquery) => subquery.selectFrom("conflict").select("conflict.change_id").unionAll( - // @ts-expect-error - no idea why subquery .selectFrom("conflict") - .select("conflict.conflicting_change_id"), + .select("conflict.conflicting_change_id as change_id"), ), ), // change is in a conflict that has not been resolved
  - samuelstros 2y ago
    You are now forever in our git history https://github.com/opral/monorepo/commit/58734e11e51d8e20092 https://github.com/opral/monorepo/commit/58734e11e51d8e20092.... the ts-expect-error was indeed fixed by your suggestion
    - ec109685 2y ago
      Isn’t kind just overriding the type checker? Should that as be necessary?
      - samuelstros 2y ago
        Yes. The runtime was not affected. Hence, the ts-expect-error. Still nice to have the ts-expect-error fixed :)
      - surrealize 2y ago
        In the context of the union query, I think it makes sense. The query takes two different columns (with two different names) from the same table, and combines them together into one column. The TS error happened because the column names didn't match between the two union inputs. But they would never match without the "as".
    - surrealize 2y ago
      Cool! I'm surrealize though, sureglymop wrote a sibling comment! I should have just made a PR, lol.
    - samuelstros 2y ago
      Today i learned that HN has a nesting limit. @surrealize i fixed the username https://github.com/opral/monorepo/commit/7dc1f3c806bd89c6d682a2006e487cb4501ef655 https://github.com/opral/monorepo/commit/7dc1f3c806bd89c6d68...
      - surrealize 2y ago
        Haha, thanks!

sureglymop 2y ago

Regarding your lix project, have you looked at Fossil? It seems like it could maybe do what you're trying to do with some small changes.

samuelstros 2y ago
Yes. We looked at all version control systems (fossil, pijul, jj, git, darc, sapling). None provide what we need: - target the browser - supports any file format, not just text files - understands changes, not just versioning of files - suited to build web apps on top - ultimately enable 1000's of artists, designers, lawyers, civil engineers, etc. to collaborate we are 2 years into the journey. dec 16, we'll release a public preview
- deleted 2y ago
  [deleted]
- sureglymop 2y ago
  Interesting! Definitely looking forward to that and I hope you succeed.

mmerickel 2y ago

I'm a huge proponent of using SQLite as an abstraction over a filesystem. One warning I will note though, is be aware that a SQLite database does not shrink unless you vacuum it (basically copies the data into a separate file and deletes the original). This is a manual operation you have to do at points where it makes sense within your application. So be careful with disk usage when just writing binary data and then deleting it.

samuelstros 2y ago
That's a good tip. I was wondering how SQLite avoids page fragmentation. The answer is "it depends," but VACUUM is to the rescue. What are your reasons for advocating for SQLite as a filesystem abstraction?