35 ms·

The problem I have with sqlite for being a general purpose file format is that it’s difficult to take a sqlite file in any language and work with it without lin

• Ecco 6y ago
  What's the problem of linking against SQLite? In my book it's much better because: - SQLite is coded in C, so performance and weight are going to be equal or better than any other reimplementation. - Having one code base means no compatibility issue between implementations - Also SQLite's codebase is so heavily tested and its track record is so good that I really don't see the point.
  • pvg 6y ago
    https://www.zdnet.com/article/google-chrome-impacted-by-new-magellan-2-0-vulnerabilities/ https://www.zdnet.com/article/google-chrome-impacted-by-new-...
    • sigzero 6y ago
      He didn't say it never has bugs. It does have a great track record and Dr. Hipp fixes things quickly when found.
    • cowsandmilk 6y ago
      If there were implementations of SQLite in other languages, do you believe chrome would have used a different implementation?
      • pvg 6y ago
        I don't know! But if this sort of thing can happen in Chrome, I think it's a pretty strong counterargument to "let's standardize formats in terms of a C implementation of a thing that parses a complicated language, among many other things".
        • rhencke 6y ago
          XML has been a source of countless vulnerabilities, yet we continue to use it as a standard.
          • pvg 6y ago
            XML has its own problems but 'has a single canonical implementation in a big hairball of C' is not among them.
          • viraptor 6y ago
            Non-C implementations of XML don't have countless vulnerabilities. There's XXE and schema requests which are language independent, but otherwise... it's ok.
  • gwd 6y ago
    FWIW I've used golang linking against sqlite in a couple of my projects. One issue is that linking through cgo is really slow, due to the fact that the runtime has no idea anything about types. Probably OK if you're just doing basic queries at a reasonable rate; but if you implement a custom callback, suddenly all the marshaling back and forth takes a huge amount of time. Not a deal-breaker; but a native implementation of sqlite in golang would be a lot nicer.
    • 0xjnml 6y ago
      > Not a deal-breaker; but a native implementation of sqlite in golang would be a lot nicer. CGo-free SQLite for Linux: https://godoc.org/modernc.org/sqlite https://godoc.org/modernc.org/sqlite
  • sargun 6y ago
    I'll give a couple examples of where I've needed a K/V database, but linking caused problems: * We have a sandboxed Javascript runtime inside of the JVM that can only do restricted work. IT would be nice to be able to use something like a SQLite file to deliver configuration to this Javascript runtime * There are environments where you cannot install any native code -- like serverless providers that require your software to be pure Java / Javascript / Python * Cross-language debugging can be very difficult when something goes wrong * Erlang's FFI model is either "blocking and fast" (it runs the FFI call inline, and halts preemptive multitasking), or "slow and safe" (dirty NIFs, where it runs FFI calls on a dedicated thread.) This can get really ugly, really quickly.
• AtlasBarfed 6y ago
  Yes. Possibly if this was done however, we'd get native sqllite file format read/write libraries in most major languages. Might overall be good. Sqllite-as-file-format isn't necessarily a bad idea if this infrastructure exists.
• an_d_rew 6y ago
  In this particular case though you have to weigh linking against SQLight versus zlib and libxml.
  • rsynnott 6y ago
    Native implementations of both those are available for many languages, though. ~no-one is using libxml with Java, say.
• 7thaccount 6y ago
  Why can't you just have your language call to SQLite via the command line?
  • simonw 6y ago
    One reason SQLite is so useful for this kind of application is that once you've linked it in any SQL call can be treated as a C function call. Many Small Queries Are Efficient In SQLite: https://www.sqlite.org/np1queryprob.html https://www.sqlite.org/np1queryprob.html Round-tripping queries via shelling out to the SQLite binary would lose much of that advantage.
    • 7thaccount 6y ago
      So efficiency?
• simonw 6y ago
  WASM is really interesting here. https://github.com/sql-js/sql.js https://github.com/sql-js/sql.js is SQLite complied to WASM, which means you can run it not just in the browser but also in any WASM environment. https://github.com/wasmerio/wasmer-python https://github.com/wasmerio/wasmer-python lets you execute WASM compiled binaries within Python. https://github.com/wasmerio/wasmer-ruby https://github.com/wasmerio/wasmer-ruby handles Ruby. So it should be possible to take the WASM compiled SQLite and run it in other languages. This is a thing that really excites me about WASM: as a mainly-Python programmer it has the potential to give me access to a much wider range of tools.
  • mcint 6y ago
    Nice links. (Hi again) Bring full SQL select, join, and order in datasette into the browser, perhaps?