8 ms·

“None of the safe programming languages existed for the first 10 years of SQLite's existence. SQLite could be recoded in Go or Rust, but doing so would probably

“None of the safe programming languages existed for the first 10 years of SQLite's existence. SQLite could be recoded in Go or Rust, but doing so would probably introduce far more bugs than would be fixed, and it may also result in slower code.” Modern languages might do more than C to prevent programmers from writing buggy code, but if you already have bug-free code due to massive time, attention, and testing, and the rate of change is low (or zero), it doesn’t really matter what the language is. SQLIte could be assembly language for all it would matter.

oconnor663 1y ago
> and the rate of change is low (or zero) This jives with a point that the Google Security Blog made last year: "The [memory safety] problem is overwhelmingly with new code...Code matures and gets safer with time." https://security.googleblog.com/2024/09/eliminating-memory-safety-vulnerabilities-Android.html https://security.googleblog.com/2024/09/eliminating-memory-s...
- miohtama 1y ago
  You can find historical SQLite CVEs here https://www.sqlite.org/cves.html https://www.sqlite.org/cves.html Note that although code matures the chances of C Human error bugs will never go to zero. We have some bad incidents like Heartbleed to show this.
  - ziotom78 1y ago
    Right, but I believe nobody can claim that Human error bugs go to zero for Rust code.
    - john_the_writer 1y ago
      Agreed. I rather dislike the idea of "safe" coding languages. Fighting with a memory leak in an elixir app, for the past week. I never viewed c or c++ as unsafe. Writing code is hard, always has been, always will be. It is never safe.
      - simonask 1y ago
        This is a bit of a misunderstanding. Safe code is just code that cannot have Undefined Behavior. C and C++ have the concept of "soundness" just like Rust, just no way to statically guard against it.
        gcr 1y ago
        Modern compilers like clang and GCC both have static analysis for some of this. Check out the undefined behavior sanitizer.
        humanrebar 1y ago
        Sanitizers are technically dynamic analysis. They instrument built programs and analyze them as they run.
        deleted 1y ago
        [deleted]
        simonask 1y ago
        As the other person pointed out, these are two different things. Sanitizers add runtime checks (with zero consideration for performance - don’t use these in productions). Static analysis runs at compile time, and while both GCC and Clang are doing amazing jobs of it, it’s still very easy to run into trouble. The mostly catch the low-hanging fruit. The technical reason is that Rust-the-language gives the compiler much more information to work with, and it doesn’t look like it is possible to add this information to the C or C++ languages.
        rileymat2 1y ago
        There is more than undefined behavior, if I was to define using an uninitialized variable as yielding whatever data was previously in that location, it is well defined but unsafe.
        oconnor663 1y ago
        That could be well defined for POD types like arrays of bytes (I think in that case they call it "freezing"), but you'd still have undefined behavior if the type in question contained pointers. Also at least in Rust it's UB to create illegal values of certain types, like a bool that's anything other than 0 or 1. Which is all kind of to say, all of Rust's different safety rules end up being surprisingly densely interconnected. It's really hard to guarantee one of them (say "no use-after-free") without ultimately requiring all of them ("no uninitialized variables", "no mutable aliasing", etc).
  - hnlmorg 1y ago
    Heartbleed was a great demonstration of critical systems that were under appreciated. Too few maintainers, too few security researchers and too little funding. When writing systems as complicated and as sensitive as the leading encryption suite used globally, no language choice will save you from under resourcing.
nabhasablue 1y ago
there is already an sqlite port in Go :) https://gitlab.com/cznic/sqlite https://gitlab.com/cznic/sqlite
- ncruces 1y ago
  That's not a port. That's an extremely impressive machine translation of C to Go. The output is a non-portable half-a-million LoC Go file for each platform.
  - cratermoon 1y ago
    also unmaintainable and full of unsafe
  - lanstin 1y ago
    And it works very well both as SQLite and as a pure go entity. I have used it for few years to do async backups of an in memory counting database (read from only at start up, written to by worker go routine that batches writes) without incident. Doesn’t really show up in the profiler.
weinzierl 1y ago
"SQLite could be recoded in Go or Rust, but doing so would probably introduce far more bugs than would be fixed, and it may also result in slower code." We will see. On the Rust side there is Turso which is pretty active. https://turso.tech/ https://turso.tech/
- Sammi 1y ago
  The Sqlite team in stuck in the classic dilemma. They are stuck with their existing thing because it is so big, that you can't just stop the world for you existing users and redo it. Meanwhile some small innovator comes along and builds the next thing because why not, they don't have anything holding them back. This is classic Innovator's Dilemma and Creative Destruction. This has of course not happened yet and we have to wait and see if Turso can actually deliver, but the Turso team is extremely talented and their git repo history is on fire, so it is definitely a possible scenario.
  - saalweachter 1y ago
    You say that like SQLite is a for profit company competing for market share or one of several rival projects in a corporation trying to not be canceled. It's an open sourced project! It's public domain! If you make an open source project that is heavily used and widely lauded for a quarter century before being supplanted by a newer solution that's better, do you know what that is? A success! You did it! You made a thing that was useful and loved! Nothing lasts forever; there's nothing wrong with a project filling a niche and then gracefully fading away when that niche goes away.
    - steveklabnik 1y ago
      I agree with your broad point, but it's worth pointing out that SQLite very much is a product that is being sold and that aspect of things is not open source.
    - Sammi 1y ago
      Funny you should say so because I actually made an effort _not_ to use corporate business terms, because open source project definitely do compete with each other both for developer and user attention and for general prestige, which in turn may be leveraged to get access to funding and other development resources in various ways. And in an even funnier turn of events Sqlite development is actually funded by a for-profit company that sells Sqlite support contracts: https://www.sqlite.org/consortium.html https://www.sqlite.org/consortium.html
pizza234 1y ago
> you already have bug-free code due to massive time, attention, and testing, and the rate of change is low (or zero), it doesn’t really matter what the language is. SQLIte could be assembly language for all it would matter. This is the C/++ delusion - "if one puts enough effort, a [complex] memory unsafe program can be made memory safe"; the year after this page was published, the Magellan series of RCEs was released. Keeping SQLite in C is certainly a valid design choice, but it's important to be aware of the practical implications of the language.
ahoka 1y ago
Sure they did exist, almost no one cared though.
devjab 1y ago
I quite like that Zig works a drop in for C in a few use cases. It's been very nice to utilize it along with our Python and regular C binaries. We attempted to move into Go because we really like the philosophy and opinions it force upon it's developers, but similar to interpreted languages it can be rather hard to optimize it. I'm sure people more talented than us would have an easy time with it, but they don't work for us. So it was easier to just go with Python and a few parts of it handled by C (and in even fewer cases Zig). I guess we could use Rust and I might be wrong on this, but it seemed like it would be a lot of work to utilize it compared to just continuing with C and gradually incorprating Zig, and we certainly don't write bug free C.
- Hendrikto 1y ago
  > We attempted to move into Go […], but similar to interpreted languages it can be rather hard to optimize it. […] So it was easier to just go with Python I don’t get that. You had trouble optimizing Go, so you went with Python?
  - actionfromafar 1y ago
    Python + C. Probably C for the optimized parts.
    - Hendrikto 1y ago
      Why not Go + C then?
      - actionfromafar 1y ago
        I'm not them but it seems they already had Python.
      - just6979 1y ago
        Because why bother if you're keeping the C? Part of the reason for moving to Go was safety by replacing the C, not just to move away from Python. I'd say the mistake was thinking Python programmers would enjoy moving to Go. I've done it, and it was not enjoyable. I wouldn't mind doing just the tight peformance things in Go instead of C... But using Go for the high-level things that Python is great at, and where the performance is not an issue, is just silly.
ChrisRR 1y ago
This is the argument against re-writing the linux base utils in rust. When they've had such widespread use for decades, a hell of a lot of bugs have been ironed out
- bombcar 1y ago
  Especially since memory bugs are only a subset of all bugs, and (perhaps) not even the most important subset. Memory bugs are often implicated in security issues, but other bugs are more likely to cause data loss, corruption, etc.
pjmlp 1y ago
The author cleverly leaves out all the safer alternatives that have existed outside UNIX, and what was happening with computers outside Bell Labs during the 1970's. Not only was Apple was able to launch the Mac Classic with zero lines of C code, their Pascal dialect lives on in Delphi and Free Pascal. As one example.
- wat10000 1y ago
  Isn’t Pascal just as problematic as C in this respect? And the original Mac was mostly (all?) assembly, not Pascal. They couldn’t have used a higher level language anyway. The system was just too small for that. SQLite wouldn’t fit on it.
  - pjmlp 1y ago
    Not at all, because Pascal is more strongly typed, and has features that C is still yet to acquire in type safety. Non exaustive list: - proper strings with bounds checking - proper arrays with bounds checking - no pointer decays, you need to be explicit about getting pointers to arrays - less use cases of implicit conversions, requires more typecasts - reference parameters reduce the need of pointers - variant records directly support tags - enumerations are stronger typed without implicit conversions - modules with better control what gets exposed - range types - set types - arenas There was plenty of Pascal code on Mac OS including a Smalltalk like OOP framework, until C++ took over Object Pascal's role at Apple, which again it isn't C. I love the "but it has used Assembly!" usual rebutal, as if OSes written in C weren't full of Assembly, or inline Assembly and language extensions that certainly aren't C (ISO C proper) either. If you prefer, Zig is a modern taken on what Modula-2 in 1978, and Object Pascal in the 1980's already offered, with the addition of nullable types and comptime as key differentor in 40 years, packaged in a more appealing syntax for current generations.
    - wat10000 1y ago
      My point is that the original Mac used little to no Pascal. The assembly isn’t a “rebuttal,” it’s just what was actually used.
1vuio0pswjnm7 1y ago
This begs the question of why Rust evangelists keep targeting existing projects instead of focusing writing new, better software. In theory these languages should allow software developers to write programs that they would not, or could not, attempt using languages without automatic memory management Instead what I see _mostly_ is re-writes and proposed re-writes of existing software, often software that has no networking functions, and/or relatively small, easily audited software that IMHO poses little risk of memory-related bugs This is concerning to me as an end-user who builds their software from source because the effects on compilation, e.g., increased resource requirements, increased interdependencies, increased program size, decreased compilation speed, are significant
- MangoToupe 1y ago
  > This begs the question of why Rust evangelists keep targeting existing projects instead of focusing writing new, better software. Designing new software is orders of magnitude more difficult than iterating on existing software
  - itopaloglu83 1y ago
    Yes, of course, but if a project owner wants to stay with a non-rust programming language, maybe we should just let them be, instead of nagging them multiple times a day about why they haven't switched, in that case writing your own project with rust to prove that it can be done better is the way to go.
    - burntsushi 1y ago
      And when they do that, they'll be vilified for "rewriting in Rust" and creating nothing new. Who actually is nagging people multiple times per day to do free labor for them and rewrite a project in a completely different programming language?
    - MangoToupe 1y ago
      Hey if it increases participation, why not? I certainly don't blame anyone for avoiding the adhoc mess that is C
throwaway81523 1y ago
> Go or Rust [Cries in Ada]
masfoobar 1y ago
Seems we live in a "cool" or "flashy" era or rewriting everything in Rust. Dont get me wrong, despite not taking time to learn Rust at this time, I am aware that memory safety is the thing at this time. Yes... some software might do better being rewritten from C to Rust. However, there are other projects that have been worked on for years and years. Sqlite is an example of this. That quote above is 100% Various GNU tools are being replaced. Dont just expect them to be 100% despite being "memory safe" -- they will have to fo through various tweaks to boost performance. With Rust likely (still) to go through some further changes, I am sure Linus will get frustrated at some points within the Linux Kernel. We shall see. Point is - some things are just better left with their mature state. Though.. on the flip side, we are have to think about the younger generation. Will SQlite survive if it continues to use C? It's likely to be a language that each new generation will not bother, and focus on Rust, Zig.. or whatever comes out in the future. I am just waiting for rewrite of Doom or Quake (I am sure they already exist if I can be bothered to search.. in Rust)

oconnor663 1y ago

> and the rate of change is low (or zero) This jives with a point that the Google Security Blog made last year: "The [memory safety] problem is overwhelmingly with new code...Code matures and gets safer with time." https://security.googleblog.com/2024/09/eliminating-memory-safety-vulnerabilities-Android.html https://security.googleblog.com/2024/09/eliminating-memory-s...

miohtama 1y ago
You can find historical SQLite CVEs here https://www.sqlite.org/cves.html https://www.sqlite.org/cves.html Note that although code matures the chances of C Human error bugs will never go to zero. We have some bad incidents like Heartbleed to show this.
- ziotom78 1y ago
  Right, but I believe nobody can claim that Human error bugs go to zero for Rust code.
  - john_the_writer 1y ago
    Agreed. I rather dislike the idea of "safe" coding languages. Fighting with a memory leak in an elixir app, for the past week. I never viewed c or c++ as unsafe. Writing code is hard, always has been, always will be. It is never safe.
    - simonask 1y ago
      This is a bit of a misunderstanding. Safe code is just code that cannot have Undefined Behavior. C and C++ have the concept of "soundness" just like Rust, just no way to statically guard against it.
      - gcr 1y ago
        Modern compilers like clang and GCC both have static analysis for some of this. Check out the undefined behavior sanitizer.
        humanrebar 1y ago
        Sanitizers are technically dynamic analysis. They instrument built programs and analyze them as they run.
        deleted 1y ago
        [deleted]
        simonask 1y ago
        As the other person pointed out, these are two different things. Sanitizers add runtime checks (with zero consideration for performance - don’t use these in productions). Static analysis runs at compile time, and while both GCC and Clang are doing amazing jobs of it, it’s still very easy to run into trouble. The mostly catch the low-hanging fruit. The technical reason is that Rust-the-language gives the compiler much more information to work with, and it doesn’t look like it is possible to add this information to the C or C++ languages.
      - rileymat2 1y ago
        There is more than undefined behavior, if I was to define using an uninitialized variable as yielding whatever data was previously in that location, it is well defined but unsafe.
        oconnor663 1y ago
        That could be well defined for POD types like arrays of bytes (I think in that case they call it "freezing"), but you'd still have undefined behavior if the type in question contained pointers. Also at least in Rust it's UB to create illegal values of certain types, like a bool that's anything other than 0 or 1. Which is all kind of to say, all of Rust's different safety rules end up being surprisingly densely interconnected. It's really hard to guarantee one of them (say "no use-after-free") without ultimately requiring all of them ("no uninitialized variables", "no mutable aliasing", etc).
- hnlmorg 1y ago
  Heartbleed was a great demonstration of critical systems that were under appreciated. Too few maintainers, too few security researchers and too little funding. When writing systems as complicated and as sensitive as the leading encryption suite used globally, no language choice will save you from under resourcing.

nabhasablue 1y ago

there is already an sqlite port in Go :) https://gitlab.com/cznic/sqlite https://gitlab.com/cznic/sqlite

ncruces 1y ago
That's not a port. That's an extremely impressive machine translation of C to Go. The output is a non-portable half-a-million LoC Go file for each platform.
- cratermoon 1y ago
  also unmaintainable and full of unsafe
- lanstin 1y ago
  And it works very well both as SQLite and as a pure go entity. I have used it for few years to do async backups of an in memory counting database (read from only at start up, written to by worker go routine that batches writes) without incident. Doesn’t really show up in the profiler.

weinzierl 1y ago

"SQLite could be recoded in Go or Rust, but doing so would probably introduce far more bugs than would be fixed, and it may also result in slower code." We will see. On the Rust side there is Turso which is pretty active. https://turso.tech/ https://turso.tech/

Sammi 1y ago
The Sqlite team in stuck in the classic dilemma. They are stuck with their existing thing because it is so big, that you can't just stop the world for you existing users and redo it. Meanwhile some small innovator comes along and builds the next thing because why not, they don't have anything holding them back. This is classic Innovator's Dilemma and Creative Destruction. This has of course not happened yet and we have to wait and see if Turso can actually deliver, but the Turso team is extremely talented and their git repo history is on fire, so it is definitely a possible scenario.
- saalweachter 1y ago
  You say that like SQLite is a for profit company competing for market share or one of several rival projects in a corporation trying to not be canceled. It's an open sourced project! It's public domain! If you make an open source project that is heavily used and widely lauded for a quarter century before being supplanted by a newer solution that's better, do you know what that is? A success! You did it! You made a thing that was useful and loved! Nothing lasts forever; there's nothing wrong with a project filling a niche and then gracefully fading away when that niche goes away.
  - steveklabnik 1y ago
    I agree with your broad point, but it's worth pointing out that SQLite very much is a product that is being sold and that aspect of things is not open source.
  - Sammi 1y ago
    Funny you should say so because I actually made an effort _not_ to use corporate business terms, because open source project definitely do compete with each other both for developer and user attention and for general prestige, which in turn may be leveraged to get access to funding and other development resources in various ways. And in an even funnier turn of events Sqlite development is actually funded by a for-profit company that sells Sqlite support contracts: https://www.sqlite.org/consortium.html https://www.sqlite.org/consortium.html

pizza234 1y ago

> you already have bug-free code due to massive time, attention, and testing, and the rate of change is low (or zero), it doesn’t really matter what the language is. SQLIte could be assembly language for all it would matter. This is the C/++ delusion - "if one puts enough effort, a [complex] memory unsafe program can be made memory safe"; the year after this page was published, the Magellan series of RCEs was released. Keeping SQLite in C is certainly a valid design choice, but it's important to be aware of the practical implications of the language.

ahoka 1y ago

Sure they did exist, almost no one cared though.

devjab 1y ago

I quite like that Zig works a drop in for C in a few use cases. It's been very nice to utilize it along with our Python and regular C binaries. We attempted to move into Go because we really like the philosophy and opinions it force upon it's developers, but similar to interpreted languages it can be rather hard to optimize it. I'm sure people more talented than us would have an easy time with it, but they don't work for us. So it was easier to just go with Python and a few parts of it handled by C (and in even fewer cases Zig). I guess we could use Rust and I might be wrong on this, but it seemed like it would be a lot of work to utilize it compared to just continuing with C and gradually incorprating Zig, and we certainly don't write bug free C.

Hendrikto 1y ago
> We attempted to move into Go […], but similar to interpreted languages it can be rather hard to optimize it. […] So it was easier to just go with Python I don’t get that. You had trouble optimizing Go, so you went with Python?
- actionfromafar 1y ago
  Python + C. Probably C for the optimized parts.
  - Hendrikto 1y ago
    Why not Go + C then?
    - actionfromafar 1y ago
      I'm not them but it seems they already had Python.
    - just6979 1y ago
      Because why bother if you're keeping the C? Part of the reason for moving to Go was safety by replacing the C, not just to move away from Python. I'd say the mistake was thinking Python programmers would enjoy moving to Go. I've done it, and it was not enjoyable. I wouldn't mind doing just the tight peformance things in Go instead of C... But using Go for the high-level things that Python is great at, and where the performance is not an issue, is just silly.

ChrisRR 1y ago

This is the argument against re-writing the linux base utils in rust. When they've had such widespread use for decades, a hell of a lot of bugs have been ironed out

bombcar 1y ago
Especially since memory bugs are only a subset of all bugs, and (perhaps) not even the most important subset. Memory bugs are often implicated in security issues, but other bugs are more likely to cause data loss, corruption, etc.

pjmlp 1y ago

The author cleverly leaves out all the safer alternatives that have existed outside UNIX, and what was happening with computers outside Bell Labs during the 1970's. Not only was Apple was able to launch the Mac Classic with zero lines of C code, their Pascal dialect lives on in Delphi and Free Pascal. As one example.

wat10000 1y ago
Isn’t Pascal just as problematic as C in this respect? And the original Mac was mostly (all?) assembly, not Pascal. They couldn’t have used a higher level language anyway. The system was just too small for that. SQLite wouldn’t fit on it.
- pjmlp 1y ago
  Not at all, because Pascal is more strongly typed, and has features that C is still yet to acquire in type safety. Non exaustive list: - proper strings with bounds checking - proper arrays with bounds checking - no pointer decays, you need to be explicit about getting pointers to arrays - less use cases of implicit conversions, requires more typecasts - reference parameters reduce the need of pointers - variant records directly support tags - enumerations are stronger typed without implicit conversions - modules with better control what gets exposed - range types - set types - arenas There was plenty of Pascal code on Mac OS including a Smalltalk like OOP framework, until C++ took over Object Pascal's role at Apple, which again it isn't C. I love the "but it has used Assembly!" usual rebutal, as if OSes written in C weren't full of Assembly, or inline Assembly and language extensions that certainly aren't C (ISO C proper) either. If you prefer, Zig is a modern taken on what Modula-2 in 1978, and Object Pascal in the 1980's already offered, with the addition of nullable types and comptime as key differentor in 40 years, packaged in a more appealing syntax for current generations.
  - wat10000 1y ago
    My point is that the original Mac used little to no Pascal. The assembly isn’t a “rebuttal,” it’s just what was actually used.

1vuio0pswjnm7 1y ago

This begs the question of why Rust evangelists keep targeting existing projects instead of focusing writing new, better software. In theory these languages should allow software developers to write programs that they would not, or could not, attempt using languages without automatic memory management Instead what I see _mostly_ is re-writes and proposed re-writes of existing software, often software that has no networking functions, and/or relatively small, easily audited software that IMHO poses little risk of memory-related bugs This is concerning to me as an end-user who builds their software from source because the effects on compilation, e.g., increased resource requirements, increased interdependencies, increased program size, decreased compilation speed, are significant

MangoToupe 1y ago
> This begs the question of why Rust evangelists keep targeting existing projects instead of focusing writing new, better software. Designing new software is orders of magnitude more difficult than iterating on existing software
- itopaloglu83 1y ago
  Yes, of course, but if a project owner wants to stay with a non-rust programming language, maybe we should just let them be, instead of nagging them multiple times a day about why they haven't switched, in that case writing your own project with rust to prove that it can be done better is the way to go.
  - burntsushi 1y ago
    And when they do that, they'll be vilified for "rewriting in Rust" and creating nothing new. Who actually is nagging people multiple times per day to do free labor for them and rewrite a project in a completely different programming language?
  - MangoToupe 1y ago
    Hey if it increases participation, why not? I certainly don't blame anyone for avoiding the adhoc mess that is C

throwaway81523 1y ago

> Go or Rust [Cries in Ada]

masfoobar 1y ago

Seems we live in a "cool" or "flashy" era or rewriting everything in Rust. Dont get me wrong, despite not taking time to learn Rust at this time, I am aware that memory safety is the thing at this time. Yes... some software might do better being rewritten from C to Rust. However, there are other projects that have been worked on for years and years. Sqlite is an example of this. That quote above is 100% Various GNU tools are being replaced. Dont just expect them to be 100% despite being "memory safe" -- they will have to fo through various tweaks to boost performance. With Rust likely (still) to go through some further changes, I am sure Linus will get frustrated at some points within the Linux Kernel. We shall see. Point is - some things are just better left with their mature state. Though.. on the flip side, we are have to think about the younger generation. Will SQlite survive if it continues to use C? It's likely to be a language that each new generation will not bother, and focus on Rust, Zig.. or whatever comes out in the future. I am just waiting for rewrite of Doom or Quake (I am sure they already exist if I can be bothered to search.. in Rust)