3 ms·

898 KB even without compression. Pretty good for what it is. Properly compressed, it’d be well under half that on the wire. (The server seems to be serving wit

898 KB even without compression. Pretty good for what it is. Properly compressed, it’d be well under half that on the wire. (The server seems to be serving with only extremely mild compression, and none on the biggest part, the WASM. Feeding it all through `gzip -9` cuts it to 408 KB, of which 374 KB is the WASM. `brotli` cuts it to 352 KB, of which 323 KB is the WASM. The HTML, CSS and JavaScript are also largely unminified, and even a simple conservative pass roughly halves their gzipped size. No idea of the state of the WASM, I’m not going prodding inside it just now. I know that I’m never impressed with the JavaScript side of the bindings produced by Empscripten, Rust’s wasm-bindgen and any other such tools I’ve seen—with a little manual effort, I’ve very easily reduced 50 KB files—minified!—to under 5 KB. Optimising such things and immediately discarding the results is a strange hobby of mine.) For comparison, the similar parts of https://sql.js.org/examples/GUI/index.html https://sql.js.org/examples/GUI/index.html are a bit under 1.2 MB uncompressed (most notably excluding CodeMirror), around 450 KB on the wire.

sgbeal 4y ago
> 898KB even without compression. Pretty good. gzip'd it's roughly half of that but that particular web server (althttpd) doesn't (yet?) know how to compress output payloads.
- motiejus 4y ago
  It makes sense to pre-compress the static assets. Then the server can load the compressed ones directly, without requiring to link to the compressor, and of course pay the price of online compression. I create two extra files for each static file (for some extensions): file file.gz <-- with zopfli file.br <-- with brotli And the web server, depending on the `Accept-Encoding` header, would serve the right file. The wasm module: $ ls -lh fiddle-module.wasm* -rw-r--r-- 1 motiejus motiejus 779K May 26 15:25 fiddle-module.wasm -rw-r--r-- 1 motiejus motiejus 323K May 26 15:25 fiddle-module.wasm.br -rw-r--r-- 1 motiejus motiejus 357K May 26 18:00 fiddle-module.wasm.gz
  - sgbeal 4y ago
    > And the web server, depending on the `Accept-Encoding` header, would serve the right file. This web server (also part of the sqlite family of projects) doesn't yet know how to do that. https://sqlite.org/althttpd https://sqlite.org/althttpd
  - chrismorgan 4y ago
    Interesting comparing the sizes of different compressors. `gzip -9` gets to 374K, whereas you report zopfli taking it down to 357K. And I had at first run `brotli -9`, being fairly confident in my memory that it had more than nine levels, but finding -h mentioning -# as 1–9, and not seeing --best (level 11) at the other end of the output, and that that’s the default (unlike gzip where I think the default is mostly 6), so that -9 actually makes it do a worse job. Sigh.
deleted 4y ago
[deleted]
sgbeal 4y ago
> I know that I’m never impressed with the JavaScript side of the bindings produced by Empscripten i don't mind saying, having spent much of the past 2 weeks in and around that code, that much of the generated part of the JS/wasm "glue" is... Much of it looks like it was thrown together by someone who half-understood JS and was just glad that it worked, with little or no attention to detail and refinement. It could use some TLC.
- chrismorgan 4y ago
  Yeah, last time I looked Emscripten’s were definitely considerably worse than wasm-bindgen’s, but that was a few years ago. There’s also just a lot of missed opportunity for things like abbreviating identifiers, plus stuff that’s completely normal for optimising compilers like GCC or LLVM like inlining and safe code reordering to eliminate completely unnecessary temporary variables and the likes, but for which absolutely no equivalent tooling exists in JavaScript. And I have no idea why that’s the case. I know of only two even vaguely interesting projects along these general lines: Google’s Closure Compiler (2009– but functionally I don’t think much interesting has happened in the last decade), but it’s too esoteric and requires too many compromises for most people to use it (and it didn’t help that it’s written in Java); and Facebook’s Prepack (2018), which tried doing partial evaluation but they given up on it before it really got anywhere useful. Everything else is just quite hopeless, almost never going beyond very simplistic syntactic transformations that don’t modify semantics.
  - sgbeal 4y ago
    > definitely considerably worse than wasm-bindgen’s, Thank you, wasm-bindgen is a new term for me (wasm as a whole is new to me since about 2 weeks). i'll add that to the list of tools to check out, as we're actively exploring different options and methodologies at this point for wasm/sqlite.
    - chrismorgan 4y ago
      I doubt wasm-bindgen will be your cup of tea as it’s Rust stuff, though perhaps there may be value in looking at what it generates. For myself, I progressively lean in the direction of burning all of these things down (when possible, and it must be admitted that Emscripten’s strength is that it makes legacy stuff work) and writing binding JavaScript manually, also with less of an FFI/dual-sided-bindings/skip-blithely-between-languages flavour and more deliberate, less RPCy techniques.
aidenn0 4y ago
One of the better commentaries on just how light sqlite is was in apenwarr's redo FAQ[1] > ... I think people underestimate how "lite" sqlite really is: root root 573376 2010-10-20 09:55 /usr/lib/libsqlite3.so.0.8.6 > 573k for a complete and very fast and transactional SQL database. For comparison, libdb is: root root 1256548 2008-09-13 03:23 /usr/lib/libdb-4.6.so > ...more than twice as big, and it doesn't even have an SQL parser in it! Or if you want to be really horrified: root root 1995612 2009-02-03 13:54 /usr/lib/libmysqlclient.so.15.0.0 > The mysql client library is two megs, and it doesn't even have a database in it! People who think SQL databases are automatically bloated and gross have not yet actually experienced the joys of sqlite. SQL has a well-deserved bad reputation, but sqlite is another story entirely. It's excellent, and much simpler and better written than you'd expect. 1: https://redo.readthedocs.io/en/latest/FAQImpl/ https://redo.readthedocs.io/en/latest/FAQImpl/
- sgbeal 4y ago
  > root root 573376 2010-10-20 09:55 /usr/lib/libsqlite3.so.0.8.6 In all fairness, though: sqlite3 was much smaller in 2010 ;). Similarly, libdb and libmysqlclient were probably smaller in 2008/2009. My locally-installed copy of today's trunk version of libsqlite3 (pre-3.39), on x86_64, stripped of debug symbols, is 1.27MB.
  - chasil 4y ago
    You can keep using those. The database format is defined as a long-term storage format by the U.S. Library of Congress. https://www.sqlite.org/locrsf.html https://www.sqlite.org/locrsf.html There have been a lot of new features over the new years, the latest being window functions. Those features do not impact the database file format. Embedded systems that don't need these features probably haven't upgraded.
    - masklinn 4y ago
      > There have been a lot of new features over the new years, the latest being window functions. Window functions were added in 3.25, 4 years ago. Since then sqlite added a bunch of stuff like RETURNING clauses, upserts, json operators (and enabled json functions by default though they’d been there a while), update from, generated columns, …

sgbeal 4y ago

> 898KB even without compression. Pretty good. gzip'd it's roughly half of that but that particular web server (althttpd) doesn't (yet?) know how to compress output payloads.

motiejus 4y ago
It makes sense to pre-compress the static assets. Then the server can load the compressed ones directly, without requiring to link to the compressor, and of course pay the price of online compression. I create two extra files for each static file (for some extensions): file file.gz <-- with zopfli file.br <-- with brotli And the web server, depending on the `Accept-Encoding` header, would serve the right file. The wasm module: $ ls -lh fiddle-module.wasm* -rw-r--r-- 1 motiejus motiejus 779K May 26 15:25 fiddle-module.wasm -rw-r--r-- 1 motiejus motiejus 323K May 26 15:25 fiddle-module.wasm.br -rw-r--r-- 1 motiejus motiejus 357K May 26 18:00 fiddle-module.wasm.gz
- sgbeal 4y ago
  > And the web server, depending on the `Accept-Encoding` header, would serve the right file. This web server (also part of the sqlite family of projects) doesn't yet know how to do that. https://sqlite.org/althttpd https://sqlite.org/althttpd
- chrismorgan 4y ago
  Interesting comparing the sizes of different compressors. `gzip -9` gets to 374K, whereas you report zopfli taking it down to 357K. And I had at first run `brotli -9`, being fairly confident in my memory that it had more than nine levels, but finding -h mentioning -# as 1–9, and not seeing --best (level 11) at the other end of the output, and that that’s the default (unlike gzip where I think the default is mostly 6), so that -9 actually makes it do a worse job. Sigh.

deleted 4y ago

[deleted]

sgbeal 4y ago

> I know that I’m never impressed with the JavaScript side of the bindings produced by Empscripten i don't mind saying, having spent much of the past 2 weeks in and around that code, that much of the generated part of the JS/wasm "glue" is... Much of it looks like it was thrown together by someone who half-understood JS and was just glad that it worked, with little or no attention to detail and refinement. It could use some TLC.

chrismorgan 4y ago
Yeah, last time I looked Emscripten’s were definitely considerably worse than wasm-bindgen’s, but that was a few years ago. There’s also just a lot of missed opportunity for things like abbreviating identifiers, plus stuff that’s completely normal for optimising compilers like GCC or LLVM like inlining and safe code reordering to eliminate completely unnecessary temporary variables and the likes, but for which absolutely no equivalent tooling exists in JavaScript. And I have no idea why that’s the case. I know of only two even vaguely interesting projects along these general lines: Google’s Closure Compiler (2009– but functionally I don’t think much interesting has happened in the last decade), but it’s too esoteric and requires too many compromises for most people to use it (and it didn’t help that it’s written in Java); and Facebook’s Prepack (2018), which tried doing partial evaluation but they given up on it before it really got anywhere useful. Everything else is just quite hopeless, almost never going beyond very simplistic syntactic transformations that don’t modify semantics.
- sgbeal 4y ago
  > definitely considerably worse than wasm-bindgen’s, Thank you, wasm-bindgen is a new term for me (wasm as a whole is new to me since about 2 weeks). i'll add that to the list of tools to check out, as we're actively exploring different options and methodologies at this point for wasm/sqlite.
  - chrismorgan 4y ago
    I doubt wasm-bindgen will be your cup of tea as it’s Rust stuff, though perhaps there may be value in looking at what it generates. For myself, I progressively lean in the direction of burning all of these things down (when possible, and it must be admitted that Emscripten’s strength is that it makes legacy stuff work) and writing binding JavaScript manually, also with less of an FFI/dual-sided-bindings/skip-blithely-between-languages flavour and more deliberate, less RPCy techniques.

aidenn0 4y ago

One of the better commentaries on just how light sqlite is was in apenwarr's redo FAQ[1] > ... I think people underestimate how "lite" sqlite really is: root root 573376 2010-10-20 09:55 /usr/lib/libsqlite3.so.0.8.6 > 573k for a complete and very fast and transactional SQL database. For comparison, libdb is: root root 1256548 2008-09-13 03:23 /usr/lib/libdb-4.6.so > ...more than twice as big, and it doesn't even have an SQL parser in it! Or if you want to be really horrified: root root 1995612 2009-02-03 13:54 /usr/lib/libmysqlclient.so.15.0.0 > The mysql client library is two megs, and it doesn't even have a database in it! People who think SQL databases are automatically bloated and gross have not yet actually experienced the joys of sqlite. SQL has a well-deserved bad reputation, but sqlite is another story entirely. It's excellent, and much simpler and better written than you'd expect. 1: https://redo.readthedocs.io/en/latest/FAQImpl/ https://redo.readthedocs.io/en/latest/FAQImpl/

sgbeal 4y ago
> root root 573376 2010-10-20 09:55 /usr/lib/libsqlite3.so.0.8.6 In all fairness, though: sqlite3 was much smaller in 2010 ;). Similarly, libdb and libmysqlclient were probably smaller in 2008/2009. My locally-installed copy of today's trunk version of libsqlite3 (pre-3.39), on x86_64, stripped of debug symbols, is 1.27MB.
- chasil 4y ago
  You can keep using those. The database format is defined as a long-term storage format by the U.S. Library of Congress. https://www.sqlite.org/locrsf.html https://www.sqlite.org/locrsf.html There have been a lot of new features over the new years, the latest being window functions. Those features do not impact the database file format. Embedded systems that don't need these features probably haven't upgraded.
  - masklinn 4y ago
    > There have been a lot of new features over the new years, the latest being window functions. Window functions were added in 3.25, 4 years ago. Since then sqlite added a bunch of stuff like RETURNING clauses, upserts, json operators (and enabled json functions by default though they’d been there a while), update from, generated columns, …