31 ms·

Why Is SQLite Coded in C? (2017)

372 points by jeffreyrogers 9y ago

strkek 9y ago
My problem with C isn't even the language; it's the ecosystem around it. Glibc, auto{conf,tools,*}, CMake, etc. Writing Makefiles by hand works for small projects, but for bigger ones you're forced to choose the lesser of all evils. And if you're on linux you'll still have to deal with glibc most of the time. And of course you can only choose between compilers that do too much (as in, not "C compilers" but "compilers that happen to support C as well"), compilers that don't even aim for standards compliance, or proprietary compilers.
- oneplane 9y ago
  Well, as with C, the ecosystem around it is mostly just misunderstood. Autotools isn't all that hard once you actually know what you are doing. While having a build and configuration system that has the learning curve of a hockey stick isn't the nicest thing, spending some time to know to to use it gets you a lot of knowledge and access that is hard to come by otherwise. I have spent about three weeks going deep in the autotools rabbit hole (mostly converting FOSS projects that used other build systems and then ran into trouble) but coming out the other side I can say it's not as bad as it looks, and compared to other systems it could be much, much worse. As was posted earlier, calling C a "portable assembly language" seems right, and having a portable system to go with it to build and have common functionality isn't all that strange when you think about it that way. While other systems might be easier for a lot of cases, not much out there can get you where the classic stack of C + Autotools + (g)make + glibc/musl gets you. As with everything, it's a tradeoff between quality, cost (As in time/knowledge) and options.
- rwbt 9y ago
  > And of course you can only choose between compilers that do too much (as in, not "C compilers" but "compilers that happen to support C as well"), compilers that don't even aim for standards compliance, or proprietary compilers. I'm not sure if I understand, but almost all mainstream compilers treat 'C' as a first class citizen. Even MSVC, notorious for only supporting C89 finally caught up and now supports most of C11.
  - strkek 9y ago
    What I mean is more like they're not "C compilers", but "C/C++/Objective-C/Java/Rust/Perl/Bash/Fortran compilers".
    - jcelerier 9y ago
      and ? what would you gain if the other languages were not supported ?
    - deckard1 9y ago
      > C/C++/Objective-C/Java/Rust/Perl/Bash/Fortran compilers huh? What compiler supports C and Perl and Bash? WTF? Sure, most C compilers are also C++ compilers. I'm not sure why this would be surprising to anyone. I don't even know how you would create a C++ compiler that didn't also compile C.
- skohan 9y ago
  I can't disagree with you as tools for managing builds, but isn't clang a pretty good C compiler which does everything you need and works pretty much everywhere?
  - strkek 9y ago
    Yup, but I'd like to remove the Objective-C{,++} support I'm almost definitely not gonna use in the foreseeable future, for example. Take it as my obsession for minimalism.
    - saagarjha 9y ago
      Since Objective-C is so similar to C, I don't see why you would write a compiler for one and not do the amount of work to get it working with C and not do the comparatively smaller amount to get Objective-C to work.
- camgunz 9y ago
  Mostly just get a CMakeLists.txt file and keep tweaking it. Here, have one of mine: https://github.com/camgunz/libd2k/blob/master/CMakeLists.txt https://github.com/camgunz/libd2k/blob/master/CMakeLists.txt
  - mcintyre1994 9y ago
    404 on that file and the repo https://github.com/camgunz/libd2k https://github.com/camgunz/libd2k - is it private?
- Cyph0n 9y ago
  I tried using CMake for one of my course projects recently and it worked like a charm. Of course, I have no idea whether or not CMake can handle larger projects as effectively. Maybe someone with more experience can chime in.
  - jcelerier 9y ago
    > I have no idea whether or not CMake can handle larger projects as effectively. well, it's able to handle whole operating systems so... https://github.com/reactos/reactos https://github.com/reactos/reactos
  - blueline 9y ago
    both times i inherited a cmake build system in two (small-ish) teams with moderately sized projects it seemed to suffer pretty hard from a "last mile" problem. 90% of what we wanted out of it was easy and the last 10% ranged from "easy but quirky" to "how is something this simple this hard" i don't have a solid gauge for how much of that was because of my own inexperience or the tool itself, although my gut feeling is that it's probably a bit of both.
- NelsonMinar 9y ago
  Worth noting sqlite has its own unique ecosystem of tools. Most people who compile it are compiling "the amalgamation", a unique thing where they smash all the sqlite source code together into one standalone .c file you can just compile and link with, no build tools needed. I've never worked with the actual source code but the docs talk about a simple Makefile, nothing more complex. https://www.sqlite.org/howtocompile.html https://www.sqlite.org/howtocompile.html
- pjc50 9y ago
  > compilers that do too much Why is this a problem?
  - strkek 9y ago
    I don't like the many-compilers-in-one approach for the same reason I wouldn't like an `ls` program with bundled support for LDAP, SSH, Git, SVN and CVS, for example. Sure, some way or another all those can fall under the category "list directory contents", but I'd rather use separate programs for every one of those. Programs that I will install if and when I need them.
- lukaszsamson 9y ago
  C tools are not more complex than Javascript tools and they don't change that often. However, after few years in JS land I feel the rust.
- fsloth 9y ago
  Try meson. http://mesonbuild.com/ http://mesonbuild.com/ It seems to be actually a simple build configuration tool. A ton of projects are using it, including systemd. http://mesonbuild.com/Users.html http://mesonbuild.com/Users.html
chungy 9y ago
I strongly suspect this article was written as a response against C++, Java, or C#. Especially given SQLite being started in 2000, C really was undeniably the best choice, and the existing code in C is a strong argument in favor against rewrites. That being said, Rust and possibly even Go would be strong contenders to make a new SQLite-like library/program today. At least on the Rust side, the C bindings are excellent too.
- Sir_Cmpwn 9y ago
  They are both weak contenders, for the reasons laid out in the article. Go and Rust have large runtimes with lots of dependencies, and Go is not as performant as C. Literally every system has a C compiler, except maybe for a very small number of very old and niche systems. Assuming it cound fit on the ROM, sqlite could probably be ported to my z80 calculator, which has an 8 bit processor and 128 kiolobytes of RAM - trivially! C runs literally everywhere, which is something no other programming language can lay claim to. For a tool like sqlite there is no other choice, period.
  - steveklabnik 9y ago
    Rust has an equivalent amount of runtime to C. Our platform support is limited by LLVM though, this is a good reason for sure.
    - Sir_Cmpwn 9y ago
      Fair, I forgot that Rust has good freestanding support. I still don't think it'd be a good candidate for sqlite, though, for one reason in particular: Rust is not competitive with C on portability.
      - dom96 9y ago
        Which is why Nim would be a good candidate. Compiles to C so has the benefits of its portability, interoperability and speed.
commandlinefan 9y ago
"Why is SQLite Coded in C? Because 'C is best'". And you know what? I can't really say I disagree.
- oneplane 9y ago
  When taking the portability and the 'C is portable assembly'-statement into account I can't disagree either. While there is plenty of hard things and things you can do wrong with C, there is a lot that you can't do in pretty much any other language.
rwbt 9y ago
My favorite aspect of programming in C is how little it gets in the way of the programmer. Once you become familiar with the language there is not much to lookup in the docs because it's such a small language. I might end up writing more verbose code, but it's usually very clear to me. You still can write spaghetti code but that's besides the point.
- deleted 9y ago
  [deleted]
- ethelward 9y ago
  I'm not sure I get your point. You say that C is a tiny language, hence leading to small docs, but that can be said of every language? Except that they may have larger standard libraries, but nothing stop you from using a subset of them.
  - pentagonpapers 9y ago
    If you think C++ is not overly complicated, just what is a protected abstract virtual base pure virtual private destructor and when was the last time you needed one?
    - fao_ 9y ago
      C++ =/= C
      - deleted 9y ago
        [deleted]
    - jcelerier 9y ago
      If you think C is not overly complicated, just what is a static const __declspec((dllimport)) __restrict volatile unsigned long long int* and when was the last time you needed one?
      - 9y ago
zanethomas 9y ago
It's always tempting to use c++ because of, for example, the availability of libraries such as STL and Boost. However doing so greatly increases the complexity of the code, when considered as a whole, and we all know that complexity spins off bugs even in well-tested libs such as mentioned above. The great thing about C is that when someone shoots you in the foot you know who it was.
- rwbt 9y ago
  Boost is a technical masterpiece and very high quality C++. It's also battle tested and very reliable. I use it for my professional projects but I loathe using it. Not just the complexity, but the error messages are really hard to decipher and most IDE's autocomplete systems completely choke when using Boost. Even if I want to see how something works 'under the hood' it's almost impossible to figure out. In the end, I came to the conclusion that it's far more productive (for me atleast) just to stick with plain C and using a 'helper' library like Apache Portable Runtime library.
  - zanethomas 9y ago
    I wrote some code that was handling upwards of 2 billion qps and every few days it would mysteriously blow up. Fortunately my gut instinct, replacing stl::map with a boost::equivalent saved the day. It's not that I don't have faith in using boost, it's just that I prefer to avoid complication and dependencies when it makes sense.
    - rwbt 9y ago
      Yep. I experienced many such situations. Like I said, I still use boost because the benefits outweigh the baggage, but I try to avoid it as much as I can.
    - zanethomas 9y ago
      btw, 2 billion qpDAY not s :)
- jcelerier 9y ago
linkmotif 9y ago
Woah: > In its minimum configuration, SQLite requires only the following routines from the standard C library: memcmp() memcpy() memmove() memset() strcmp() strlen() strnc
- 3chelon 9y ago
  Surely something from <stdio.h> as well?
  - tzs 9y ago
    Minimal configuration only supports in-memory databases.
    - 3chelon 9y ago
      Of course. Makes sense.
  - boomlinde 9y ago
    Not necessarily, with an in-memory database or callback hooks for reading/writing
israrkhan 9y ago
C is perhaps the most portable language. Most platforms support C. I once used sqlite in a embedded system project that was based on a VLIW processor(Trimedia). The only compiler available was a C compiler. The integration worked like a charm.
qwerty456127 9y ago
C is great and SQLite seems doing amazing leveraging its power. Yet I bloody wish there were alternative (yet 100% compatible on the file format and SQL dialect levels) SQLite implementations native to specific languages/platforms (e.g. .Net). In just so many cases the DB speed is the least priority while simplicity and portability are valued much higher (in these particular cases). Every time I build a humble desktop .Net app featuring an SQLite database it turns into in a ridiculous "try this, google a lot, try that, repeat, .... oh wow, it works!... oh, it doesn't" nightmare and even if it actually starts working nobody really knows how to deploy such an app on another computer (especially if it runs a different kind of OS) correctly. SQLite is arguably the best of what has happened to the world of desktop/embedded databases ever yet lack of alternative (not necessarily this fast) implementations in high-level languages is its main problem IMHO.
- laumars 9y ago
  Ive used sqlite a lot in recent years and found the issue you describe to be largely one that only affects Windows deployments. On Linux, FreeBSD, Solaris and OS X builds it's all just worked for me. But the same build scripts will fail for a multitude of subtle ways on Windows. IIRC (I've not targeted Windows much recently) I've found the easiest way to get it working was just to have mingw (or similar POSIX layer) and build sqlite from within that.
  - WorldMaker 9y ago
    As with most things, your mileage varies a lot based on what you are used to. The previous poster mentioned "cross-platform" and I think that is where the most headaches lie because you might know one permutation, but not all of them. For instance, Windows binary deployments are quite predictable (and often dead simple; so easy a lot of Windows installers get it wrong and yet still work), but on *nix you may have to fight distro differences in libc or get a dozen different answers on what you should statically link in your compile versus what you "must" dynamically link and a half-dozen different installers/bundlers/distro tools to deal with making sure the shared libraries are both installed and correctly linked to. What "just works" is often relative and the magic is very easily dispelled in that terrible state of "why isn't this working?", especially if it "just works" like magic for other people. For what it is worth, the Windows UWP platform even bundles a shared SQLite install (as like Android and iOS) and using it is quite easy in a bunch of languages (just not easily portable cross-platform).
jerf 9y ago
I've commented several times before that I consider "C" and "C with analysis backing" to be in practice two different languages. SQLite is the latter: https://sqlite.org/testing.html https://sqlite.org/testing.html Writing SQLite in "plain C" without all that would, well, simply not work. I agree that "C with analysis backing" is the best language for SQLite right now. However, it should not be used as an example of "How C is a great language for programming in" unless you are also planning on skillfully using a very significant subset of the tools listed in that document. SQLite doesn't prove C is great; it demonstrates how much additional scaffolding is necessary to wrap around C to get that level of quality, and given the quantity and diversity of tools we are talking about, it is not particularly complimentary to "plain C".
- pcwalton 9y ago
  I agree. SQLite gets away with using C because it literally uses military-grade levels of verification. As John Regehr pointed out, SQLite is quite possibly the only piece of software that goes to that level of validation and testing without being required to by law. It's not just a matter of skill, either. The cost in terms of money and time needed to develop software in that way is completely impractical in almost any commercial scenario. Aside from some very specific situations, it's not an economically viable way to produce software.
  - IloveHN84 9y ago
    In Germany there are strict regulations if you're writing secure software, especially under the lens of BSI and of the Bundesdruckerei. Softwares which deal with eID are verified and certified with strict rules
    - signa11 9y ago
      > In Germany there are strict regulations if you're writing secure software, especially under the lens of BSI and of the Bundesdruckerei. Softwares which deal with eID are verified and certified with strict rules just curious, does this extend to dependent libraries as well ?
kbumsik 9y ago
I love C for system-level coding. As mentioned in OP, you don't need to study much about language implementation e.g. how GC works, the internal structure of objects. You have the control of every espect of system and that's why Linus prefer C over C++. But I have to admit that the biggest downside of programming in C is people make their projects complicated. I saw some projects that have compilcated Makefile build systems and have hidden definitions. The worst part is that they leave these hidden stuff UNDOCUMENTED. This is often very frustrating when contributing open source projects.
- pcwalton 9y ago
  You absolutely do have to study a lot about the C language to understand when you slip into undefined behavior. There are even circumstances in which compiler developers don't agree on whether some code has defined behavior or not.
  - kbumsik 9y ago
    You are correct. But most cases are explicitly mentioned in compiler's documentations. I would say it's less efforts than getting to know about other language internals.
    - pcwalton 9y ago
      I disagree. It is significantly easier to determine the semantics of arbitrary Java programs than C programs.
alricb 9y ago
Ada wouldn't fit because of compatibility and availability, but it's old, fast, safe and relatively simple, as compared to C with extensive scaffolding.
- irundebian 9y ago
  I don't understand this. Ada is still being developed. Other question: Why not using SPARK?
  - jeffreyrogers 9y ago
    It's much easier to include C source code in an application than Ada.
    - irundebian 9y ago
      Yes, but you don't want C if you choose Ada, no?
      - jeffreyrogers 9y ago
        But sqlite is a dependency for many other applications, lots on platforms that probably don't have good Ada compilers, so sqlite wouldn't be as useful if it were written in Ada.
  - deleted 9y ago
    [deleted]
elchief 9y ago
The browser you're using to read this is written in C++, on your C++ or C OS, talking to a web server written in C.
- WorldMaker 9y ago
  Unless you are using the last couple versions of Firefox which has entire subsystems written in Rust now, as the easiest counter-example. Also, the important bits of the HN "web server" are the business logic that makes HN HN, and that is written in a Lisp dialect. Even if there weren't counter-examples: are the languages that browsers, OSes, and web servers written in somehow more important than any other languages? Are browsers/OSes/web servers the best emblems of reliability engineering to you? There are a lot of interesting assumptions at play here.
  - saagarjha 9y ago
    > Are browsers/OSes/web servers the best emblems of reliability engineering to you? Since so many people depend on them, I'd say yes. It's extremely obvious to users when they don't work, so I'd assume there's a lot of work put into making them reliable.
- noblethrasher 9y ago
  > talking to a web server written in C The HN web server and application are written in a dialect of Lisp, unless something has changed.
  - elchief 9y ago
    I meant the SQLite.org server
- ekr 9y ago
  Other than HaikuOS, I can't think of any other real-world kernel written in C++. Linux, all BSDs, Windows NT, Hurd are all C. I'm not sure but I think XNU is C as well (the wiki lists it as C/C++).
tzs 9y ago
This is kind of cheating, but Lua looks like it would have worked. I consider that a bit of cheat, because it replaces the problem of embedding one C program (SQLite) with the problem of embedding a different C program (Lua). But for those whose objection is that writing the database logic in C is risky because C is too low level, this would put the database logic in a higher level language than C.
pier25 9y ago
https://sqlite.org/fasterthanfs.html https://sqlite.org/fasterthanfs.html This page shows that SQLlite is 5 times faster than Win10 filesystem, but the performance gains are much small in other systems. Can anyone comment on this?
- deleted 9y ago
  [deleted]
- ekingr 9y ago
  See previous discussion: https://news.ycombinator.com/item?id=14550060 https://news.ycombinator.com/item?id=14550060
andrewchambers 9y ago
rust panics when out of memory, that would not be acceptable to sqlite3.
- steveklabnik 9y ago
  The Rust language knows nothing of dynamic allocation. You're speaking of the behavior of the standard library, which is trivial to drop. You're free to have whatever allocator semantics you want, including robustness on OOM.
  - pornel 9y ago
    Steve, you usually give better answers. Invoking the distinction between Rust-as-defined and Rust-as-commonly-used feels like language-lawyery dodging of a real issue. Suggesting to drop all of std just to customize just one feature of the allocator it is not a good solution. You should know harsh OOM handling is a problem for Rust users, and there's work being done to improve it.
    - steveklabnik 9y ago
      Well, suggesting that Rust inherently has this problem is mis-representing the situation. And while it's true that lots of people use std, many people also don't, and especially the people that care most about this aspect of Rust. > You should know harsh OOM handling is a problem for Rust users A small number of people have this issue. It barely even applies on entire operating systems, for example. That said, custom allocators will be nice.
  - andrewchambers 9y ago
    Fair enough, I guess for a project like sqlite rewriting some custom data types is no big deal.
maztheman 9y ago
There is a spelling mistake... Libraries written in C doe not have a huge run-time dependency.
odammit 9y ago
That’s impressive: > Libraries written in C doe not have a huge run-time dependency. In its minimum configuration, SQLite requires only the following routines from the standard C library: memcmp() memcpy() memmove() memset() strcmp() strlen()
- saagarjha 9y ago
  I think this statement is slightly misleading, because nobody uses SQLite in its minimum configuration. See the line below: > In a more complete build, SQLite also uses library routines like malloc() and free() and operating system interfaces for opening, reading, writing, and closing files.
  - andrewmcwatters 9y ago
    A little bit of an overstatement. It's not uncommon to see minimum configurations in many embedded scenarios.
JepZ 9y ago
Yes, SQLite is coded in C and that is okay. But there are situations, when you wish it would be different. For example if you want to cross-compile a Go project and you have to find out that the Go lib sqlite3 just provides bindings for the C library (probably due to the high quality of the C lib) and therefore breaks the easy to use Go compiler :-/ I don't know an easy solution to this as most C compilers seem to be platform specific. In general, I like C and even more SQLite. Nevertheless, it is quite unfortunate if you can't make use of the modern cross platform compilers. Maybe one day we will live on a world where cross compiling isn't an issue anymore.
- saagarjha 9y ago
  > the Go lib sqlite3 just provides bindings for the C library So compile SQLite for your target platform and use the bindings?
  - JepZ 9y ago
    Yeah sure, but in order to do that you need an extra C compiler for the target platform. With pure Go applications it is as easy as changing a parameter for the standard compiler.
camgunz 9y ago
There are a lot of comments about "well, if you use a bunch of tools and write a million tests, C is just fine". But do you _enjoy_ doing that? I know I don't, and anything we can do to achieve whatever we're aiming for with C (speed, resource usage, startup time, ubiqity) while avoiding that is great.
ralfjung 9y ago
I can perfectly agree to much of what they say, but here... > The C language is old and boring. It is a well-known and well-understood language. ...I think they are very fundamentally mistaken. C is a horribly complicated language. It is one of the least-understood languages out there. Experienced programmers and compiler authors can debate for hours about whether C code of less than 50 lines has defined behavior or not, and still not come to a conclusion. People can write an entire PhD thesis <https://robbertkrebbers.nl/thesis.html> https://robbertkrebbers.nl/thesis.html> studying the semantics of C, and still leave many open question (chapter 2 of that thesis does not require any academic background to be understandable, and it comes with tons of links to tickets/questions filed against the C standard). Consistently writing safe C/C++ is near impossible <http://robert.ocallahan.org/2017/07/confession-of-cc-programmer.html> http://robert.ocallahan.org/2017/07/confession-of-cc-program..., and judging from <https://sqlite.org/testing.html> https://sqlite.org/testing.html> the SQLite team agrees. C is old, yes -- and C has boring and well-understood fragments. But full C is very, very poorly understood.
- SQLite 9y ago
  Just to be clear: The SQLite project strives to fix UB whenever any is discovered (which is to say, "rarely"). But SQLite also focuses on testing at the machine-code level, not just at the source code level. The machine-code generated by GCC, CLANG, and MSVC is all tested to 100% branch coverage and beyond. So even if one were to find some new UB in the SQLite source code, all the usual compilers are known to be doing something sane with it, not something goofy or harmful, and so it is not really a problem.
domenukk 9y ago
Hello, our friends at MITRE have assigned CVE-2018-8740 to an issue in SQLite3 that was discovered by OSS-Fuzz working on GDAL. http://seclists.org/oss-sec/2018/q1/244 http://seclists.org/oss-sec/2018/q1/244

372 points by jeffreyrogers 9y ago

My problem with C isn't even the language; it's the ecosystem around it. Glibc, auto{conf,tools,*}, CMake, etc. Writing Makefiles by hand works for small projects, but for bigger ones you're forced to choose the lesser of all evils. And if you're on linux you'll still have to deal with glibc most of the time. And of course you can only choose between compilers that do too much (as in, not "C compilers" but "compilers that happen to support C as well"), compilers that don't even aim for standards compliance, or proprietary compilers.

oneplane 9y ago
Well, as with C, the ecosystem around it is mostly just misunderstood. Autotools isn't all that hard once you actually know what you are doing. While having a build and configuration system that has the learning curve of a hockey stick isn't the nicest thing, spending some time to know to to use it gets you a lot of knowledge and access that is hard to come by otherwise. I have spent about three weeks going deep in the autotools rabbit hole (mostly converting FOSS projects that used other build systems and then ran into trouble) but coming out the other side I can say it's not as bad as it looks, and compared to other systems it could be much, much worse. As was posted earlier, calling C a "portable assembly language" seems right, and having a portable system to go with it to build and have common functionality isn't all that strange when you think about it that way. While other systems might be easier for a lot of cases, not much out there can get you where the classic stack of C + Autotools + (g)make + glibc/musl gets you. As with everything, it's a tradeoff between quality, cost (As in time/knowledge) and options.
rwbt 9y ago
> And of course you can only choose between compilers that do too much (as in, not "C compilers" but "compilers that happen to support C as well"), compilers that don't even aim for standards compliance, or proprietary compilers. I'm not sure if I understand, but almost all mainstream compilers treat 'C' as a first class citizen. Even MSVC, notorious for only supporting C89 finally caught up and now supports most of C11.
- strkek 9y ago
  What I mean is more like they're not "C compilers", but "C/C++/Objective-C/Java/Rust/Perl/Bash/Fortran compilers".
  - jcelerier 9y ago
    and ? what would you gain if the other languages were not supported ?
  - deckard1 9y ago
    > C/C++/Objective-C/Java/Rust/Perl/Bash/Fortran compilers huh? What compiler supports C and Perl and Bash? WTF? Sure, most C compilers are also C++ compilers. I'm not sure why this would be surprising to anyone. I don't even know how you would create a C++ compiler that didn't also compile C.
skohan 9y ago
I can't disagree with you as tools for managing builds, but isn't clang a pretty good C compiler which does everything you need and works pretty much everywhere?
- strkek 9y ago
  Yup, but I'd like to remove the Objective-C{,++} support I'm almost definitely not gonna use in the foreseeable future, for example. Take it as my obsession for minimalism.
  - saagarjha 9y ago
    Since Objective-C is so similar to C, I don't see why you would write a compiler for one and not do the amount of work to get it working with C and not do the comparatively smaller amount to get Objective-C to work.
camgunz 9y ago
Mostly just get a CMakeLists.txt file and keep tweaking it. Here, have one of mine: https://github.com/camgunz/libd2k/blob/master/CMakeLists.txt https://github.com/camgunz/libd2k/blob/master/CMakeLists.txt
- mcintyre1994 9y ago
  404 on that file and the repo https://github.com/camgunz/libd2k https://github.com/camgunz/libd2k - is it private?
Cyph0n 9y ago
I tried using CMake for one of my course projects recently and it worked like a charm. Of course, I have no idea whether or not CMake can handle larger projects as effectively. Maybe someone with more experience can chime in.
- jcelerier 9y ago
  > I have no idea whether or not CMake can handle larger projects as effectively. well, it's able to handle whole operating systems so... https://github.com/reactos/reactos https://github.com/reactos/reactos
- blueline 9y ago
  both times i inherited a cmake build system in two (small-ish) teams with moderately sized projects it seemed to suffer pretty hard from a "last mile" problem. 90% of what we wanted out of it was easy and the last 10% ranged from "easy but quirky" to "how is something this simple this hard" i don't have a solid gauge for how much of that was because of my own inexperience or the tool itself, although my gut feeling is that it's probably a bit of both.
NelsonMinar 9y ago
Worth noting sqlite has its own unique ecosystem of tools. Most people who compile it are compiling "the amalgamation", a unique thing where they smash all the sqlite source code together into one standalone .c file you can just compile and link with, no build tools needed. I've never worked with the actual source code but the docs talk about a simple Makefile, nothing more complex. https://www.sqlite.org/howtocompile.html https://www.sqlite.org/howtocompile.html
pjc50 9y ago
> compilers that do too much Why is this a problem?
- strkek 9y ago
  I don't like the many-compilers-in-one approach for the same reason I wouldn't like an `ls` program with bundled support for LDAP, SSH, Git, SVN and CVS, for example. Sure, some way or another all those can fall under the category "list directory contents", but I'd rather use separate programs for every one of those. Programs that I will install if and when I need them.
lukaszsamson 9y ago
C tools are not more complex than Javascript tools and they don't change that often. However, after few years in JS land I feel the rust.
fsloth 9y ago
Try meson. http://mesonbuild.com/ http://mesonbuild.com/ It seems to be actually a simple build configuration tool. A ton of projects are using it, including systemd. http://mesonbuild.com/Users.html http://mesonbuild.com/Users.html

chungy 9y ago

I strongly suspect this article was written as a response against C++, Java, or C#. Especially given SQLite being started in 2000, C really was undeniably the best choice, and the existing code in C is a strong argument in favor against rewrites. That being said, Rust and possibly even Go would be strong contenders to make a new SQLite-like library/program today. At least on the Rust side, the C bindings are excellent too.

Sir_Cmpwn 9y ago
They are both weak contenders, for the reasons laid out in the article. Go and Rust have large runtimes with lots of dependencies, and Go is not as performant as C. Literally every system has a C compiler, except maybe for a very small number of very old and niche systems. Assuming it cound fit on the ROM, sqlite could probably be ported to my z80 calculator, which has an 8 bit processor and 128 kiolobytes of RAM - trivially! C runs literally everywhere, which is something no other programming language can lay claim to. For a tool like sqlite there is no other choice, period.
- steveklabnik 9y ago
  Rust has an equivalent amount of runtime to C. Our platform support is limited by LLVM though, this is a good reason for sure.
  - Sir_Cmpwn 9y ago
    Fair, I forgot that Rust has good freestanding support. I still don't think it'd be a good candidate for sqlite, though, for one reason in particular: Rust is not competitive with C on portability.
    - dom96 9y ago
      Which is why Nim would be a good candidate. Compiles to C so has the benefits of its portability, interoperability and speed.

commandlinefan 9y ago

"Why is SQLite Coded in C? Because 'C is best'". And you know what? I can't really say I disagree.

oneplane 9y ago
When taking the portability and the 'C is portable assembly'-statement into account I can't disagree either. While there is plenty of hard things and things you can do wrong with C, there is a lot that you can't do in pretty much any other language.

rwbt 9y ago

My favorite aspect of programming in C is how little it gets in the way of the programmer. Once you become familiar with the language there is not much to lookup in the docs because it's such a small language. I might end up writing more verbose code, but it's usually very clear to me. You still can write spaghetti code but that's besides the point.

deleted 9y ago
[deleted]
ethelward 9y ago
I'm not sure I get your point. You say that C is a tiny language, hence leading to small docs, but that can be said of every language? Except that they may have larger standard libraries, but nothing stop you from using a subset of them.
- pentagonpapers 9y ago
  If you think C++ is not overly complicated, just what is a protected abstract virtual base pure virtual private destructor and when was the last time you needed one?
  - fao_ 9y ago
    C++ =/= C
    - deleted 9y ago
      [deleted]
  - jcelerier 9y ago
    If you think C is not overly complicated, just what is a static const __declspec((dllimport)) __restrict volatile unsigned long long int* and when was the last time you needed one?
    - 9y ago

zanethomas 9y ago

It's always tempting to use c++ because of, for example, the availability of libraries such as STL and Boost. However doing so greatly increases the complexity of the code, when considered as a whole, and we all know that complexity spins off bugs even in well-tested libs such as mentioned above. The great thing about C is that when someone shoots you in the foot you know who it was.

rwbt 9y ago
Boost is a technical masterpiece and very high quality C++. It's also battle tested and very reliable. I use it for my professional projects but I loathe using it. Not just the complexity, but the error messages are really hard to decipher and most IDE's autocomplete systems completely choke when using Boost. Even if I want to see how something works 'under the hood' it's almost impossible to figure out. In the end, I came to the conclusion that it's far more productive (for me atleast) just to stick with plain C and using a 'helper' library like Apache Portable Runtime library.
- zanethomas 9y ago
  I wrote some code that was handling upwards of 2 billion qps and every few days it would mysteriously blow up. Fortunately my gut instinct, replacing stl::map with a boost::equivalent saved the day. It's not that I don't have faith in using boost, it's just that I prefer to avoid complication and dependencies when it makes sense.
  - rwbt 9y ago
    Yep. I experienced many such situations. Like I said, I still use boost because the benefits outweigh the baggage, but I try to avoid it as much as I can.
  - zanethomas 9y ago
    btw, 2 billion qpDAY not s :)
jcelerier 9y ago

linkmotif 9y ago

Woah: > In its minimum configuration, SQLite requires only the following routines from the standard C library: memcmp() memcpy() memmove() memset() strcmp() strlen() strnc

3chelon 9y ago
Surely something from <stdio.h> as well?
- tzs 9y ago
  Minimal configuration only supports in-memory databases.
  - 3chelon 9y ago
    Of course. Makes sense.
- boomlinde 9y ago
  Not necessarily, with an in-memory database or callback hooks for reading/writing

israrkhan 9y ago

C is perhaps the most portable language. Most platforms support C. I once used sqlite in a embedded system project that was based on a VLIW processor(Trimedia). The only compiler available was a C compiler. The integration worked like a charm.

qwerty456127 9y ago

C is great and SQLite seems doing amazing leveraging its power. Yet I bloody wish there were alternative (yet 100% compatible on the file format and SQL dialect levels) SQLite implementations native to specific languages/platforms (e.g. .Net). In just so many cases the DB speed is the least priority while simplicity and portability are valued much higher (in these particular cases). Every time I build a humble desktop .Net app featuring an SQLite database it turns into in a ridiculous "try this, google a lot, try that, repeat, .... oh wow, it works!... oh, it doesn't" nightmare and even if it actually starts working nobody really knows how to deploy such an app on another computer (especially if it runs a different kind of OS) correctly. SQLite is arguably the best of what has happened to the world of desktop/embedded databases ever yet lack of alternative (not necessarily this fast) implementations in high-level languages is its main problem IMHO.

laumars 9y ago
Ive used sqlite a lot in recent years and found the issue you describe to be largely one that only affects Windows deployments. On Linux, FreeBSD, Solaris and OS X builds it's all just worked for me. But the same build scripts will fail for a multitude of subtle ways on Windows. IIRC (I've not targeted Windows much recently) I've found the easiest way to get it working was just to have mingw (or similar POSIX layer) and build sqlite from within that.
- WorldMaker 9y ago
  As with most things, your mileage varies a lot based on what you are used to. The previous poster mentioned "cross-platform" and I think that is where the most headaches lie because you might know one permutation, but not all of them. For instance, Windows binary deployments are quite predictable (and often dead simple; so easy a lot of Windows installers get it wrong and yet still work), but on *nix you may have to fight distro differences in libc or get a dozen different answers on what you should statically link in your compile versus what you "must" dynamically link and a half-dozen different installers/bundlers/distro tools to deal with making sure the shared libraries are both installed and correctly linked to. What "just works" is often relative and the magic is very easily dispelled in that terrible state of "why isn't this working?", especially if it "just works" like magic for other people. For what it is worth, the Windows UWP platform even bundles a shared SQLite install (as like Android and iOS) and using it is quite easy in a bunch of languages (just not easily portable cross-platform).

jerf 9y ago

I've commented several times before that I consider "C" and "C with analysis backing" to be in practice two different languages. SQLite is the latter: https://sqlite.org/testing.html https://sqlite.org/testing.html Writing SQLite in "plain C" without all that would, well, simply not work. I agree that "C with analysis backing" is the best language for SQLite right now. However, it should not be used as an example of "How C is a great language for programming in" unless you are also planning on skillfully using a very significant subset of the tools listed in that document. SQLite doesn't prove C is great; it demonstrates how much additional scaffolding is necessary to wrap around C to get that level of quality, and given the quantity and diversity of tools we are talking about, it is not particularly complimentary to "plain C".

pcwalton 9y ago
I agree. SQLite gets away with using C because it literally uses military-grade levels of verification. As John Regehr pointed out, SQLite is quite possibly the only piece of software that goes to that level of validation and testing without being required to by law. It's not just a matter of skill, either. The cost in terms of money and time needed to develop software in that way is completely impractical in almost any commercial scenario. Aside from some very specific situations, it's not an economically viable way to produce software.
- IloveHN84 9y ago
  In Germany there are strict regulations if you're writing secure software, especially under the lens of BSI and of the Bundesdruckerei. Softwares which deal with eID are verified and certified with strict rules
  - signa11 9y ago
    > In Germany there are strict regulations if you're writing secure software, especially under the lens of BSI and of the Bundesdruckerei. Softwares which deal with eID are verified and certified with strict rules just curious, does this extend to dependent libraries as well ?

kbumsik 9y ago

I love C for system-level coding. As mentioned in OP, you don't need to study much about language implementation e.g. how GC works, the internal structure of objects. You have the control of every espect of system and that's why Linus prefer C over C++. But I have to admit that the biggest downside of programming in C is people make their projects complicated. I saw some projects that have compilcated Makefile build systems and have hidden definitions. The worst part is that they leave these hidden stuff UNDOCUMENTED. This is often very frustrating when contributing open source projects.

pcwalton 9y ago
You absolutely do have to study a lot about the C language to understand when you slip into undefined behavior. There are even circumstances in which compiler developers don't agree on whether some code has defined behavior or not.
- kbumsik 9y ago
  You are correct. But most cases are explicitly mentioned in compiler's documentations. I would say it's less efforts than getting to know about other language internals.
  - pcwalton 9y ago
    I disagree. It is significantly easier to determine the semantics of arbitrary Java programs than C programs.

alricb 9y ago

Ada wouldn't fit because of compatibility and availability, but it's old, fast, safe and relatively simple, as compared to C with extensive scaffolding.

irundebian 9y ago
I don't understand this. Ada is still being developed. Other question: Why not using SPARK?
- jeffreyrogers 9y ago
  It's much easier to include C source code in an application than Ada.
  - irundebian 9y ago
    Yes, but you don't want C if you choose Ada, no?
    - jeffreyrogers 9y ago
      But sqlite is a dependency for many other applications, lots on platforms that probably don't have good Ada compilers, so sqlite wouldn't be as useful if it were written in Ada.
- deleted 9y ago
  [deleted]

elchief 9y ago

The browser you're using to read this is written in C++, on your C++ or C OS, talking to a web server written in C.

WorldMaker 9y ago
Unless you are using the last couple versions of Firefox which has entire subsystems written in Rust now, as the easiest counter-example. Also, the important bits of the HN "web server" are the business logic that makes HN HN, and that is written in a Lisp dialect. Even if there weren't counter-examples: are the languages that browsers, OSes, and web servers written in somehow more important than any other languages? Are browsers/OSes/web servers the best emblems of reliability engineering to you? There are a lot of interesting assumptions at play here.
- saagarjha 9y ago
  > Are browsers/OSes/web servers the best emblems of reliability engineering to you? Since so many people depend on them, I'd say yes. It's extremely obvious to users when they don't work, so I'd assume there's a lot of work put into making them reliable.
noblethrasher 9y ago
> talking to a web server written in C The HN web server and application are written in a dialect of Lisp, unless something has changed.
- elchief 9y ago
  I meant the SQLite.org server
ekr 9y ago
Other than HaikuOS, I can't think of any other real-world kernel written in C++. Linux, all BSDs, Windows NT, Hurd are all C. I'm not sure but I think XNU is C as well (the wiki lists it as C/C++).

tzs 9y ago

This is kind of cheating, but Lua looks like it would have worked. I consider that a bit of cheat, because it replaces the problem of embedding one C program (SQLite) with the problem of embedding a different C program (Lua). But for those whose objection is that writing the database logic in C is risky because C is too low level, this would put the database logic in a higher level language than C.

pier25 9y ago

https://sqlite.org/fasterthanfs.html https://sqlite.org/fasterthanfs.html This page shows that SQLlite is 5 times faster than Win10 filesystem, but the performance gains are much small in other systems. Can anyone comment on this?

deleted 9y ago
[deleted]
ekingr 9y ago
See previous discussion: https://news.ycombinator.com/item?id=14550060 https://news.ycombinator.com/item?id=14550060

andrewchambers 9y ago

rust panics when out of memory, that would not be acceptable to sqlite3.

steveklabnik 9y ago
The Rust language knows nothing of dynamic allocation. You're speaking of the behavior of the standard library, which is trivial to drop. You're free to have whatever allocator semantics you want, including robustness on OOM.
- pornel 9y ago
  Steve, you usually give better answers. Invoking the distinction between Rust-as-defined and Rust-as-commonly-used feels like language-lawyery dodging of a real issue. Suggesting to drop all of std just to customize just one feature of the allocator it is not a good solution. You should know harsh OOM handling is a problem for Rust users, and there's work being done to improve it.
  - steveklabnik 9y ago
    Well, suggesting that Rust inherently has this problem is mis-representing the situation. And while it's true that lots of people use std, many people also don't, and especially the people that care most about this aspect of Rust. > You should know harsh OOM handling is a problem for Rust users A small number of people have this issue. It barely even applies on entire operating systems, for example. That said, custom allocators will be nice.
- andrewchambers 9y ago
  Fair enough, I guess for a project like sqlite rewriting some custom data types is no big deal.

maztheman 9y ago

There is a spelling mistake... Libraries written in C doe not have a huge run-time dependency.

odammit 9y ago

That’s impressive: > Libraries written in C doe not have a huge run-time dependency. In its minimum configuration, SQLite requires only the following routines from the standard C library: memcmp() memcpy() memmove() memset() strcmp() strlen()

saagarjha 9y ago
I think this statement is slightly misleading, because nobody uses SQLite in its minimum configuration. See the line below: > In a more complete build, SQLite also uses library routines like malloc() and free() and operating system interfaces for opening, reading, writing, and closing files.
- andrewmcwatters 9y ago
  A little bit of an overstatement. It's not uncommon to see minimum configurations in many embedded scenarios.

JepZ 9y ago

Yes, SQLite is coded in C and that is okay. But there are situations, when you wish it would be different. For example if you want to cross-compile a Go project and you have to find out that the Go lib sqlite3 just provides bindings for the C library (probably due to the high quality of the C lib) and therefore breaks the easy to use Go compiler :-/ I don't know an easy solution to this as most C compilers seem to be platform specific. In general, I like C and even more SQLite. Nevertheless, it is quite unfortunate if you can't make use of the modern cross platform compilers. Maybe one day we will live on a world where cross compiling isn't an issue anymore.

saagarjha 9y ago
> the Go lib sqlite3 just provides bindings for the C library So compile SQLite for your target platform and use the bindings?
- JepZ 9y ago
  Yeah sure, but in order to do that you need an extra C compiler for the target platform. With pure Go applications it is as easy as changing a parameter for the standard compiler.

camgunz 9y ago

There are a lot of comments about "well, if you use a bunch of tools and write a million tests, C is just fine". But do you _enjoy_ doing that? I know I don't, and anything we can do to achieve whatever we're aiming for with C (speed, resource usage, startup time, ubiqity) while avoiding that is great.

ralfjung 9y ago

I can perfectly agree to much of what they say, but here... > The C language is old and boring. It is a well-known and well-understood language. ...I think they are very fundamentally mistaken. C is a horribly complicated language. It is one of the least-understood languages out there. Experienced programmers and compiler authors can debate for hours about whether C code of less than 50 lines has defined behavior or not, and still not come to a conclusion. People can write an entire PhD thesis <https://robbertkrebbers.nl/thesis.html> https://robbertkrebbers.nl/thesis.html> studying the semantics of C, and still leave many open question (chapter 2 of that thesis does not require any academic background to be understandable, and it comes with tons of links to tickets/questions filed against the C standard). Consistently writing safe C/C++ is near impossible <http://robert.ocallahan.org/2017/07/confession-of-cc-programmer.html> http://robert.ocallahan.org/2017/07/confession-of-cc-program..., and judging from <https://sqlite.org/testing.html> https://sqlite.org/testing.html> the SQLite team agrees. C is old, yes -- and C has boring and well-understood fragments. But full C is very, very poorly understood.

SQLite 9y ago
Just to be clear: The SQLite project strives to fix UB whenever any is discovered (which is to say, "rarely"). But SQLite also focuses on testing at the machine-code level, not just at the source code level. The machine-code generated by GCC, CLANG, and MSVC is all tested to 100% branch coverage and beyond. So even if one were to find some new UB in the SQLite source code, all the usual compilers are known to be doing something sane with it, not something goofy or harmful, and so it is not really a problem.

domenukk 9y ago

Hello, our friends at MITRE have assigned CVE-2018-8740 to an issue in SQLite3 that was discovered by OSS-Fuzz working on GDAL. http://seclists.org/oss-sec/2018/q1/244 http://seclists.org/oss-sec/2018/q1/244