6 ms·

SQLite’s Use of Tcl (2017)

138 points by fanf2 2y ago

weinzierl 2y ago
This is from the SQLite creator D. Richard Hipp who is always worth reading, but, I'd like to recommend reading what TCL's creator John Ousterhout has to say. His article on threads from 1995 was highly influential on me, and I remember it to this day. More recently (2018, revised and expanded in 2021), he published a book on software engineering practices called A Philosophy of Software Design which is, in my opinion, the best in its category.
- dilawar 2y ago
  +100 for his book. TIL he created TCL also. The book is excellent and one of the very resources that talks about architecture independent of language or tech-stack used.
  - notRobot 2y ago
    He also gave a talk that greatly influenced how I look at relationships: https://gist.github.com/gtallen1187/27a585fcf36d6e657db2 https://gist.github.com/gtallen1187/27a585fcf36d6e657db2
  - mananaysiempre 2y ago
    Like Lamport, who is more widely known for (what was originally) his side project LaTeX than for his seminal distributed systems research including Paxos, Ousterhout tends to be more widely known by name for (what was originally) his side project Tcl than for his seminal distributed systems research including Raft.
    - awesomeMilou 2y ago
      What an amusing coincidence, I didn't know Lamport wrote LaTeX, rather I knew of him only in connection with Lamport clocks.
      - abhgh 2y ago
        "(La)Tex" is believed to expand to "(Lamport)TeX" although I haven't seen an official source for that trivia :-)
    - p_l 2y ago
      Ousterhout also I think had outsized presence in early VLSI tooling, which is why TCL happened and why is it still common in tooling (pun intended)
- pjot 2y ago
  This book lives on my desk. And I’ve proudly filled its pages with annotations and sticky notes. It’s really well done.
- leoh 2y ago
  >His article on threads from 1995 was highly influential on me, and I remember it to this day. More recently (2018, revised and expanded in 2021), he published a book on software engineering practices called A Philosophy of Software Design which is, in my opinion, the best in its category. Would you have a link?
deleted 2y ago
[deleted]
graemep 2y ago
> Early versions of SQLite (prior to 2004) operated on the classic TCL principal that "everything is a string". Beginning with SQLite3 (2004-06-18), SQLite also supports binary data. TCL can handle binary data. It is just not a separate type: https://wiki.tcl-lang.org/page/Working+with+binary+data https://wiki.tcl-lang.org/page/Working+with+binary+data SQLite also always had a null type, surely? > However, types are still very flexible in SQLite, just as they are in TCL. SQLite treats the datatypes on column names in a CREATE TABLE statement as suggestions rather than hard requirements This is something I do not much like. Its not compulsory (you can create "strict" tables). It works well in TCL which is an entire language designed around the idea. Less so in SQL. One of the advantages of RDBMSes is that not accepting obviously wrong data makes life easier for developers. You can debug an issue that happens on inserting the data, not when you find the wrong type or other bad much later on.
- tyingq 2y ago
  There was a time when TCL could not handle binary data at all. The support came in TCL 8.0: "Binary data is now supported in Tcl." https://www.tcl.tk/software/tcltk/relnotes/tcl8.0.txt https://www.tcl.tk/software/tcltk/relnotes/tcl8.0.txt Well before 2004, but worth mentioning because you'll find a fair amount of old posts complaining about it.
  - gcr 2y ago
    What do you mean, that TCL strings weren’t 8-bit-clean?
    - pavlov 2y ago
      Maybe they were zero-terminated C strings, which of course can’t represent arbitrary binary data.
      - tyingq 2y ago
        Yes, prior to 8.0, tcl strings mapped to null-terminated C strings. "Binary I/O. The new object system in Tcl 8.0 supports binary strings (internally, strings are counted in addition to being null terminated). There is a new "binary" command for inserting and extracting data to/from binary strings. Commands such as "puts", "gets", and "read" commands now operate correctly on binary data. There is a new variable tcl_platform(byteOrder) to identify the native byte order for the current host."
zvr 2y ago
Back in the day, the Tcl conferences (and the EuroTcl in Europe) were great sources of information. Only about half of the presentations were for Tcl internals and extensions. The rest were about other projects that were using Tcl in some way or other, and it was fascinating to learn about completely different areas of software.
- wduquette 2y ago
  They were. I attended for many years, including the conference where Richard gave this talk.
  - cmacleod4 2y ago
    The US Tcl conference has not happened for a few years now, but EuroTcl continues, I gave a talk there myself this year - https://openacs.org/conf2024/info/ https://openacs.org/conf2024/info/ .
    - wduquette 2y ago
      Hi, Colin! I'm sorry to hear that the US conference is dead, but glad to hear that EuroTcl carries on.
postepowanieadm 2y ago
Anyone has used the mysterious "e" editor?
- cess11 2y ago
  There's a manual and it influenced Stallman in the creation of Emacs so some people sure did. http://i.stanford.edu/pub/cstr/reports/cs/tr/80/796/CS-TR-80-796.pdf http://i.stanford.edu/pub/cstr/reports/cs/tr/80/796/CS-TR-80...
  - postepowanieadm 2y ago
    That's a treasure!
  - coliveira 2y ago
    That's not the E editor he talked about. It is the editor created by Richard Hipp himself. You can find a reference here: https://wiki.tcl-lang.org/page/Tcl+Editors https://wiki.tcl-lang.org/page/Tcl+Editors And you can find a version here: http://grumbeer.dyndns.org/ftp/cdroms/freebsd/freebsd-2.2.1-2/ports/distfiles/e93-1.2r6X.tar.gz http://grumbeer.dyndns.org/ftp/cdroms/freebsd/freebsd-2.2.1-...
    - pdw 2y ago
      No, that's Todd Squires's e93 editor, the next entry on the wiki :)
smitty1e 2y ago
> For example, the byte-code engine used to evaluate SQL statements inside of SQLite is implemented as a large "switch" statement inside a "for" loop, with a separate "case" for each opcode, all in the "vdbe.c" source file. Duff's Device[1] for the win! [1] https://en.m.wikipedia.org/wiki/Duff's_device https://en.m.wikipedia.org/wiki/Duff's_device
- wizzwizz4 2y ago
  Duff's Device is a for loop inside a switch statement. Totally different.
  - 1oooqooq 2y ago
    And here I was thinking I found a fancy term for my "lazy parser"... Well "Lazy Parser" it will remain named.
  - smitty1e 2y ago
    I stand clue-batted, thank you.
rwalle 2y ago
I wonder -- if they were to restart from scratch today, would they do the same thing? If not, which stack would they choose?
- pstuart 2y ago
  Rust would probably be the language of choice -- it appears to be the default for software in that domain.
1vuio0pswjnm7 2y ago
"The sqlite3.c and sqlite3.h source files are build products, and the source tree used to build those files is over 50% TCL code." Always feels silly to have to install Tcl to compile sqlite3, I use the --disable-tcl configuration option before compiling. After "make" I have a one-liner to statically-link the sqlite3 binary. It takes a relatively long time to link; sqlite3.c is a whopping 8.7M. I end up with a 1.7M sqlite3 binary. I never do "make install". I do not want to install all the Tcl stuff.
omoikane 2y ago
There is also a video recording of this talk: https://youtu.be/kHmwv3I1Kxk?si=CrZW_VVnA_UqcXCU https://youtu.be/kHmwv3I1Kxk?si=CrZW_VVnA_UqcXCU I was looking for this because Figure 1 (Breakdown Of SQLite Source Code By Language) is missing, and archive.org doesn't seem to have it. That chart can be found near 11:56 of the video above.
novoreorx 2y ago
I wonder is it possible to use Tcl as a testing tool for modern languages such as Python and JavaScript, the way SQLite use Tcl to do testing looks really promising

138 points by fanf2 2y ago

This is from the SQLite creator D. Richard Hipp who is always worth reading, but, I'd like to recommend reading what TCL's creator John Ousterhout has to say. His article on threads from 1995 was highly influential on me, and I remember it to this day. More recently (2018, revised and expanded in 2021), he published a book on software engineering practices called A Philosophy of Software Design which is, in my opinion, the best in its category.

dilawar 2y ago
+100 for his book. TIL he created TCL also. The book is excellent and one of the very resources that talks about architecture independent of language or tech-stack used.
- notRobot 2y ago
  He also gave a talk that greatly influenced how I look at relationships: https://gist.github.com/gtallen1187/27a585fcf36d6e657db2 https://gist.github.com/gtallen1187/27a585fcf36d6e657db2
- mananaysiempre 2y ago
  Like Lamport, who is more widely known for (what was originally) his side project LaTeX than for his seminal distributed systems research including Paxos, Ousterhout tends to be more widely known by name for (what was originally) his side project Tcl than for his seminal distributed systems research including Raft.
  - awesomeMilou 2y ago
    What an amusing coincidence, I didn't know Lamport wrote LaTeX, rather I knew of him only in connection with Lamport clocks.
    - abhgh 2y ago
      "(La)Tex" is believed to expand to "(Lamport)TeX" although I haven't seen an official source for that trivia :-)
  - p_l 2y ago
    Ousterhout also I think had outsized presence in early VLSI tooling, which is why TCL happened and why is it still common in tooling (pun intended)
pjot 2y ago
This book lives on my desk. And I’ve proudly filled its pages with annotations and sticky notes. It’s really well done.
leoh 2y ago
>His article on threads from 1995 was highly influential on me, and I remember it to this day. More recently (2018, revised and expanded in 2021), he published a book on software engineering practices called A Philosophy of Software Design which is, in my opinion, the best in its category. Would you have a link?

deleted 2y ago

[deleted]

graemep 2y ago

> Early versions of SQLite (prior to 2004) operated on the classic TCL principal that "everything is a string". Beginning with SQLite3 (2004-06-18), SQLite also supports binary data. TCL can handle binary data. It is just not a separate type: https://wiki.tcl-lang.org/page/Working+with+binary+data https://wiki.tcl-lang.org/page/Working+with+binary+data SQLite also always had a null type, surely? > However, types are still very flexible in SQLite, just as they are in TCL. SQLite treats the datatypes on column names in a CREATE TABLE statement as suggestions rather than hard requirements This is something I do not much like. Its not compulsory (you can create "strict" tables). It works well in TCL which is an entire language designed around the idea. Less so in SQL. One of the advantages of RDBMSes is that not accepting obviously wrong data makes life easier for developers. You can debug an issue that happens on inserting the data, not when you find the wrong type or other bad much later on.

tyingq 2y ago
There was a time when TCL could not handle binary data at all. The support came in TCL 8.0: "Binary data is now supported in Tcl." https://www.tcl.tk/software/tcltk/relnotes/tcl8.0.txt https://www.tcl.tk/software/tcltk/relnotes/tcl8.0.txt Well before 2004, but worth mentioning because you'll find a fair amount of old posts complaining about it.
- gcr 2y ago
  What do you mean, that TCL strings weren’t 8-bit-clean?
  - pavlov 2y ago
    Maybe they were zero-terminated C strings, which of course can’t represent arbitrary binary data.
    - tyingq 2y ago
      Yes, prior to 8.0, tcl strings mapped to null-terminated C strings. "Binary I/O. The new object system in Tcl 8.0 supports binary strings (internally, strings are counted in addition to being null terminated). There is a new "binary" command for inserting and extracting data to/from binary strings. Commands such as "puts", "gets", and "read" commands now operate correctly on binary data. There is a new variable tcl_platform(byteOrder) to identify the native byte order for the current host."

zvr 2y ago

Back in the day, the Tcl conferences (and the EuroTcl in Europe) were great sources of information. Only about half of the presentations were for Tcl internals and extensions. The rest were about other projects that were using Tcl in some way or other, and it was fascinating to learn about completely different areas of software.

wduquette 2y ago
They were. I attended for many years, including the conference where Richard gave this talk.
- cmacleod4 2y ago
  The US Tcl conference has not happened for a few years now, but EuroTcl continues, I gave a talk there myself this year - https://openacs.org/conf2024/info/ https://openacs.org/conf2024/info/ .
  - wduquette 2y ago
    Hi, Colin! I'm sorry to hear that the US conference is dead, but glad to hear that EuroTcl carries on.

postepowanieadm 2y ago

Anyone has used the mysterious "e" editor?

cess11 2y ago
There's a manual and it influenced Stallman in the creation of Emacs so some people sure did. http://i.stanford.edu/pub/cstr/reports/cs/tr/80/796/CS-TR-80-796.pdf http://i.stanford.edu/pub/cstr/reports/cs/tr/80/796/CS-TR-80...
- postepowanieadm 2y ago
  That's a treasure!
- coliveira 2y ago
  That's not the E editor he talked about. It is the editor created by Richard Hipp himself. You can find a reference here: https://wiki.tcl-lang.org/page/Tcl+Editors https://wiki.tcl-lang.org/page/Tcl+Editors And you can find a version here: http://grumbeer.dyndns.org/ftp/cdroms/freebsd/freebsd-2.2.1-2/ports/distfiles/e93-1.2r6X.tar.gz http://grumbeer.dyndns.org/ftp/cdroms/freebsd/freebsd-2.2.1-...
  - pdw 2y ago
    No, that's Todd Squires's e93 editor, the next entry on the wiki :)

smitty1e 2y ago

> For example, the byte-code engine used to evaluate SQL statements inside of SQLite is implemented as a large "switch" statement inside a "for" loop, with a separate "case" for each opcode, all in the "vdbe.c" source file. Duff's Device[1] for the win! [1] https://en.m.wikipedia.org/wiki/Duff's_device https://en.m.wikipedia.org/wiki/Duff's_device

wizzwizz4 2y ago
Duff's Device is a for loop inside a switch statement. Totally different.
- 1oooqooq 2y ago
  And here I was thinking I found a fancy term for my "lazy parser"... Well "Lazy Parser" it will remain named.
- smitty1e 2y ago
  I stand clue-batted, thank you.

rwalle 2y ago

I wonder -- if they were to restart from scratch today, would they do the same thing? If not, which stack would they choose?

pstuart 2y ago
Rust would probably be the language of choice -- it appears to be the default for software in that domain.

1vuio0pswjnm7 2y ago

"The sqlite3.c and sqlite3.h source files are build products, and the source tree used to build those files is over 50% TCL code." Always feels silly to have to install Tcl to compile sqlite3, I use the --disable-tcl configuration option before compiling. After "make" I have a one-liner to statically-link the sqlite3 binary. It takes a relatively long time to link; sqlite3.c is a whopping 8.7M. I end up with a 1.7M sqlite3 binary. I never do "make install". I do not want to install all the Tcl stuff.

omoikane 2y ago

There is also a video recording of this talk: https://youtu.be/kHmwv3I1Kxk?si=CrZW_VVnA_UqcXCU https://youtu.be/kHmwv3I1Kxk?si=CrZW_VVnA_UqcXCU I was looking for this because Figure 1 (Breakdown Of SQLite Source Code By Language) is missing, and archive.org doesn't seem to have it. That chart can be found near 11:56 of the video above.

novoreorx 2y ago

I wonder is it possible to use Tcl as a testing tool for modern languages such as Python and JavaScript, the way SQLite use Tcl to do testing looks really promising