4 ms·

Off topic; It looks like you linked the linux binary completely statically. I am curious why you decided to do this and how you achieved it. Did you use somethi

Off topic; It looks like you linked the linux binary completely statically. I am curious why you decided to do this and how you achieved it. Did you use something other than glibc to compile these binaries? Also did you roll the SQL engine from scratch or does it built on some existing parser/runtime? Would love to learn more.

kcl 11y ago
Linux linking: I wasn't doing anything special with the linking. The Linux binary was compiled on a VirtualBox instance of Mint and stripped and compressed, so it's possible your tools are misreporting, or I did something I wasn't expecting. At any rate, I haven't made that as a conscious design choice, and it's easy to change (fix?) compilation styles in a future build. SQL engine: all from scratch
- paulasmuth 11y ago
  Awesome stuff, I am actually hacking on something similar, so I would love to hear your thoughts RE: supporting "windowed" aggregations. (E.g. a moving average over a timeseries). Are you planning on adding that and which syntax are you planning to use? Have you looked at postgres "OVER PARTITION"? While it seems powerful it's also fairly unintuitive IMHO. I was experimenting with adding a GROUP BY clause that allows each input row to appear in more than one group in the result set. Something like: SELECT time, mean(value) FROM mymetric GROUP OVER TIMEWINDOW(time, 60);
  - electrum 11y ago
    You can do that like this: GROUP BY date_trunc('minute', time)
    - paulasmuth 11y ago
      The snippet you posted will compute an aggregation based on a fixed time interval. I.e. it will put every row into a "bucket" of per-minute granularity and then compute an aggregate function for each of those buckets bucket, taking into account only the rows that ended up in that specific bucket (i.e. only rows from that specific minute). To put it another way this is asking the question "Please give me the aggregate of some value per minute". To make my question more precise; I was trying to ask specifically about a "moving window aggregation" (e.g. a moving average over a timeseries). This is more like asking the question "Please give me every minute an aggregate based on all values in the last N minutes". To do that you need each input row to end up in more than one bucket (or have a special type of aggregation function like postgres does). For example, if you were doing a moving aggregation with a 1-minute interval ("bucket size") and a 5 minute window ("lookback"), you would need to place each row into 5 buckets: The bucket into which it belongs based on it's timestamp and the 4 previous buckets. And a vanilla SQL GROUP BY can't do that. Hope that makes sense.
vezzy-fnord 11y ago
The executable appears to be packed using vanilla UPX. Decompressing it and grepping the symbol table reveals the presence /lib/libc.so.6, /lib/libm.so.6 and /lib/libpthread.so.0. By method of exclusion, it cannot be musl because musl unifies everything into a single /lib/libc.so. It most likely is not uClibc because uClibc is typically symlinked to /lib/libc.so.0 from /lib/libuClibc. Considering that the soname of /lib/libc.so.6 has been reserved by glibc since the 2.x days (to differentiate it from the Linux libc days of libc4 and libc5), it is most likely indeed glibc in use. It's obvious this program isn't making use of NSS or anything like that, so it's doable to use glibc for static linking, if frustrating.
- paulasmuth 11y ago
  Ah turns out this binary actually isn't statically linked at all. It's just the UPX packer that you mentioned that adds a trampoline/stub (the decompression routine I suppose) around the actual binary [and that trampoline code is what's completely static]: ldd kerf-unpacked linux-vdso.so.1 => (0x00007fff5e7f1000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff1833aa000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff1830a4000) libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007ff182e5d000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff182a99000) /lib64/ld-linux-x86-64.so.2 (0x00007ff1835d1000) libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007ff182870000) So using NSS shouldn't be an issue; and the binary actually does use it open("/lib/x86_64-linux-gnu/libnss_nis.so.2", O_RDONLY|O_CLOEXEC) = 3
  - vezzy-fnord 11y ago
    Right, the reference to ld-linux(8) should have been a giveaway.
  - jpollock 11y ago
    libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007ff182e5d000) libreadline is GPL'ed, not LGPLed, Kerf is therefore in violation. https://cnswww.cns.cwru.edu/php/chet/readline/rltop.html https://cnswww.cns.cwru.edu/php/chet/readline/rltop.html Edit: The GPL doesn't allow any linking into non-GPL compatible software, dynamic or static. Readline has this license for exactly this reason, and they have forced software into the public with it before (ncftp comes to mind) http://www.gnu.org/licenses/gpl-faq.en.html#GPLStaticVsDynamic http://www.gnu.org/licenses/gpl-faq.en.html#GPLStaticVsDynam... Edit2: https://en.wikipedia.org/wiki/GNU_Readline#Implications_of_GNU_Readline.27s_GPL_license https://en.wikipedia.org/wiki/GNU_Readline#Implications_of_G...

kcl 11y ago

Linux linking: I wasn't doing anything special with the linking. The Linux binary was compiled on a VirtualBox instance of Mint and stripped and compressed, so it's possible your tools are misreporting, or I did something I wasn't expecting. At any rate, I haven't made that as a conscious design choice, and it's easy to change (fix?) compilation styles in a future build. SQL engine: all from scratch

paulasmuth 11y ago
Awesome stuff, I am actually hacking on something similar, so I would love to hear your thoughts RE: supporting "windowed" aggregations. (E.g. a moving average over a timeseries). Are you planning on adding that and which syntax are you planning to use? Have you looked at postgres "OVER PARTITION"? While it seems powerful it's also fairly unintuitive IMHO. I was experimenting with adding a GROUP BY clause that allows each input row to appear in more than one group in the result set. Something like: SELECT time, mean(value) FROM mymetric GROUP OVER TIMEWINDOW(time, 60);
- electrum 11y ago
  You can do that like this: GROUP BY date_trunc('minute', time)
  - paulasmuth 11y ago
    The snippet you posted will compute an aggregation based on a fixed time interval. I.e. it will put every row into a "bucket" of per-minute granularity and then compute an aggregate function for each of those buckets bucket, taking into account only the rows that ended up in that specific bucket (i.e. only rows from that specific minute). To put it another way this is asking the question "Please give me the aggregate of some value per minute". To make my question more precise; I was trying to ask specifically about a "moving window aggregation" (e.g. a moving average over a timeseries). This is more like asking the question "Please give me every minute an aggregate based on all values in the last N minutes". To do that you need each input row to end up in more than one bucket (or have a special type of aggregation function like postgres does). For example, if you were doing a moving aggregation with a 1-minute interval ("bucket size") and a 5 minute window ("lookback"), you would need to place each row into 5 buckets: The bucket into which it belongs based on it's timestamp and the 4 previous buckets. And a vanilla SQL GROUP BY can't do that. Hope that makes sense.

vezzy-fnord 11y ago

The executable appears to be packed using vanilla UPX. Decompressing it and grepping the symbol table reveals the presence /lib/libc.so.6, /lib/libm.so.6 and /lib/libpthread.so.0. By method of exclusion, it cannot be musl because musl unifies everything into a single /lib/libc.so. It most likely is not uClibc because uClibc is typically symlinked to /lib/libc.so.0 from /lib/libuClibc. Considering that the soname of /lib/libc.so.6 has been reserved by glibc since the 2.x days (to differentiate it from the Linux libc days of libc4 and libc5), it is most likely indeed glibc in use. It's obvious this program isn't making use of NSS or anything like that, so it's doable to use glibc for static linking, if frustrating.

paulasmuth 11y ago
Ah turns out this binary actually isn't statically linked at all. It's just the UPX packer that you mentioned that adds a trampoline/stub (the decompression routine I suppose) around the actual binary [and that trampoline code is what's completely static]: ldd kerf-unpacked linux-vdso.so.1 => (0x00007fff5e7f1000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff1833aa000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff1830a4000) libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007ff182e5d000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff182a99000) /lib64/ld-linux-x86-64.so.2 (0x00007ff1835d1000) libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007ff182870000) So using NSS shouldn't be an issue; and the binary actually does use it open("/lib/x86_64-linux-gnu/libnss_nis.so.2", O_RDONLY|O_CLOEXEC) = 3
- vezzy-fnord 11y ago
  Right, the reference to ld-linux(8) should have been a giveaway.
- jpollock 11y ago
  libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007ff182e5d000) libreadline is GPL'ed, not LGPLed, Kerf is therefore in violation. https://cnswww.cns.cwru.edu/php/chet/readline/rltop.html https://cnswww.cns.cwru.edu/php/chet/readline/rltop.html Edit: The GPL doesn't allow any linking into non-GPL compatible software, dynamic or static. Readline has this license for exactly this reason, and they have forced software into the public with it before (ncftp comes to mind) http://www.gnu.org/licenses/gpl-faq.en.html#GPLStaticVsDynamic http://www.gnu.org/licenses/gpl-faq.en.html#GPLStaticVsDynam... Edit2: https://en.wikipedia.org/wiki/GNU_Readline#Implications_of_GNU_Readline.27s_GPL_license https://en.wikipedia.org/wiki/GNU_Readline#Implications_of_G...