7 ms·

Is the size of SQLite really a problem? Is its reliability something that hardware devs are willing to sacrifice? (In other words, how big is SQLite and what a

• pfalcon 11y ago
  LittleD is designed to run on devices the size of (higher-end) Arduinos. So, it's difference of not being able to use a database and being able to use it. Reliability is a different problem, which is addressed for deeply embedded devices is slightly different ways than for bloated servers (e.g. by employing flash memory properties, or monitoring power supply in realtime).
• pbreit 11y ago
  In the project's defense, I didn't see the size comparison anywhere other than this submission.
  • pfalcon 11y ago
    Yes, currently you need to build it yourself to do size comparisons. Generally, this HN post is in argument to LittleD's author that people care about his project, and that he should care about its potential users (and contributors!) and provide better disclosure of LittleD's capabilities and traits, not requiring everyone to dig deep to get basic facts.
    • geedy 11y ago
      You've been very good at nudging me pfalcon. Thank you!
• teraflop 11y ago
  Although SQLite's memory overhead is tiny by the standards of modern desktops/servers/phones, it's still much too big for 8-bit microcontrollers like the Arduino. If you want to build something that costs tens of cents instead of tens of dollars, SQLite is way too big. For example, https://www.sqlite.org/about.html https://www.sqlite.org/about.html claims that SQLite can run in "very little heap (100KiB)"; this project targets an Atmel ATmega2560 which has only 8KiB of RAM. (I doubt "5 times smaller than SQLite" is a useful comparison; the limiting factor is probably RAM consumption, as opposed to executable code which can be stored in cheaper non-volatile memory.)
  • pfalcon 11y ago
    I thought hard what to put in the subject (limited length, you know) - code size or working RAM size, but figured if I write "can run queries in 1KB", people won't get it ;-).
    • rattray 11y ago
      Consider putting some comparisons on the readme!
      • pfalcon 11y ago
        That's what I'm telling LittleD's author too! I posted here on HN just to show him that it's not my whim, and other people think the same ;-).
    • dang 11y ago
      Since the size comparison to SQLite in the title has made the thread mostly be an argument about size comparisons and SQLite, let's try the other phrase and see if we fare better. (Submitted title was "LittleD – SQL database for IoT, 5 times smaller than SQLite").
      • collinmanderson 11y ago
        I actually saw the new title without SQLite, and I my first question was still how it compared to SQLite. :)
  • tkinom 11y ago
    IMHO, for that kind of 8bit device/size, the app storage requirements is most likely limited. define a few C structs with standard read() write() function call are most likely be enough and it is easy to design/test/debug. One has a lot more control on RAM/ROM/Code space with that approach. You can easily find out where/how every single byte/bit of memory is used.
    • 11y ago
• raimue 11y ago
  LittleD targets devices the size of an Arduino, while SQLite requires at least some sort of operating system offering a filesystem interface. The feature set of LittleD is also reduced as it supports only simple queries. The corresponding paper [1] claims "SQLite requires at least 200 KB of code space and cannot run on popular platforms such as the Arduino." For comparison, Arduinos based on ATmega microcontrollers only offer up to 256 KB of flash memory at maximum. [1] https://dl.acm.org/citation.cfm?id=2554891 https://dl.acm.org/citation.cfm?id=2554891
  • jly 11y ago
    SQLite definitely does not require a filesystem interface, or even standard OS interface. It does require some kind of storage, though, but that could be direct flash, RAM, etc. I have spent a lot of time with it on ARM Cortex-M-class devices on small RTOSes or even bare metal, and you can adapt it to almost anything by plugging in new VFS interfaces (http://www.sqlite.org/vfs.html http://www.sqlite.org/vfs.html). It does require at least 10s of kb of RAM as a heap to do pretty much anything, as others have stated.
    • to3m 11y ago
      The ATmegas only go up to 16KBytes RAM, with 4KBytes or less being usual. As a Harvard architecture device, none of your binary is memory-mapped, which can make porting existing code a bit annoying. sizeof(int)==2, which doesn't help either (however sizeof(void *)==2 as well, so there is at least that). It's rather unusual by today's standards. Certainly quite different from an ARM.
      • listic 11y ago
        ...though still the latest and sanest of that crop of micro's, IMHO. Designed in 1996, it was once marketed as "the first new 8-bit architecture designed in 20 years", I believe :-)
    • pfalcon 11y ago
      You can adapt, or have you adapted? I looked at SQLite internals, and decided that I'll skip that adaptation for now, seeing if someone else would do that. LittleD's author says that he looked into adapting SQLite, and that's how he decided to write LittleD.
• trymas 11y ago
  +1 SQLite is probably on of the most well tested piece of software on this globe. I do not know if this space saving is worth it.
  • pfalcon 11y ago
    No. But running SQL where you never could run it before (where SQLite couldn't fit even remotely) is definitely interesting.
  • nickpsecurity 11y ago
    It is if the space savings gets your product sales due to lower unit price with big $$$ resulting. That's why there's a market for 8-16bit processors. Most developers don't want it so much as need it because getting job done for $1 a CPU is better than using $10-$100. ;) EDIT to add two links by Ganssle to explain it better: http://www.ganssle.com/rants/8bitsisdead.htm http://www.ganssle.com/rants/8bitsisdead.htm http://www.ganssle.com/articles/8and16bit.htm http://www.ganssle.com/articles/8and16bit.htm I think Microchip's 8-bitters doing $1+ billion in business with better profits and dividends vs troubles of many in 32-64 bit markets says a lot.
    • sitkack 11y ago
      Those articles are over a decade old. The size of silicon between an 8 or 16 or 32 bit ALU contained in an MCU is miniscule. Most of the space is taken up by peripherals and this is where Microchip has historically thrived but this is no longer so. There are a number of 32 bit MCUs that are cost competitive with 8 bit parts.
      • nickpsecurity 11y ago
        How about this month with a market survey? ;) http://www.edn.com/electronics-blogs/embedded-insights/4440832/8-bit-isn-t-dying--it-s-growing http://www.edn.com/electronics-blogs/embedded-insights/44408... Interesting details on how 8-bit companies are doing that from John Donovan the year before: https://www.digikey.com/en/articles/techzone/2014/feb/is-there-a-future-for-8-bit-mcus https://www.digikey.com/en/articles/techzone/2014/feb/is-the... We'd need a price list for the other thing. Can you get 32's with decent memory & low watts for a few bucks a CPU in low volume? Aside from that, you still think they're dead and useless with the new articles? I'll add that many chips from old nodes survived for decades due to stability. The new process nodes have all kinds of issues and break faster. So, there is that to consider if the application is long-term, safety-critical, or security-critical. Many in high assurance design stayed with old stuff (esp on SOI) because manufacturing bugs were less and interference/wear issues are lower.
• pilif 11y ago
  Back in 2004, I wrote code for the Intermec ck1 barcode scanner. It ran uClinux and had a stack size limited to 4K. That thing could only very barely run sqlite 2 (after applying a few patches to it works with the small stack size) and I could never get SQLite 3 to work. Granted. That was 10 years ago, but it was also quite a beefy device for its time. So while naturally the devices have become quite more powerful in the last 10 years, current IoT devices are also much more compact than the old ck1. As such I would say it's conceivable that you don't have the memory to easily run current SQLite on any of these devices.
  • pfalcon 11y ago
    Also mind that software doesn't stay the same either. As an example, LMDB (smaller, key-value DB) in its initial docs and slides bragged 30K of object size. Build current version, you'll get 60K, and there doesn't seem to be config options to just make it smaller. So either use older version, which is of course scary (who knows how many bugs was fixed later), or spend quite a lot of time tearing it apart, with unclear maintenance perspectives. Having something ROM and RAM optimized from the beginning seems like pretty good solution (if someone grew brave to do it).
    • geedy 11y ago
      In all honesty, keeping the ROM size small is the challenge.
• geedy 11y ago
  As others have noted, the resource constraints associated with many of the most cost-effective 8/16/32-bit AVR or ARM devices simply cannot run on SQLite. I even looked at making it smaller before starting LittleD, but I deemed that a Herculean task. SQLite itself requires at least 100KiB of Heap space, sometimes many times more memory than already available: http://www.sqlite.org/about.html http://www.sqlite.org/about.html
• revelation 11y ago
  It's not a tradeoff you are making, the fact is that your Arduino only has program space measured in the dozens KiB. If it doesn't fit, it doesn't fit. But that's not the biggest problem. SQLite, like most programs, relies on having a C library like glibc available. This is pretty much the intermediary person to the operating system - you can ask it for memory, it will allocate memory for you, you can ask it to open a file, it will facilitate that. Most programs assume it's always present and so the dependency isn't properly abstracted. This is a problem when you don't have an operating system. Notice also that you can't just go and implement or stub everything that glibc does. To do that, you would just end up having to add an operating system. (This particular rabbit hole goes much deeper. A lot of programs sadly don't depend on a C library, they depend on glibc in particular. This causes massive problems for systems that run Linux yet can't accomodate glibc, like OpenWRT. They use an alternative C library that works fine 99% of the time, and for the remaining 1% it will cause programs to silently misbehave or crash because of an implicit dependency on whatever glibc does. I had this problem with the hwclock utility which started segfaulting when OpenWRT switched to musl libc)