4 ms·

Show HN: Litelink – local-first, embedded stream capture into Iceberg tables

9 points by jnbh 1mo ago

Hi HN! I just wanted to share litelink a local-first, embedded capture library I built in python (code is heavily AI generated but designed and reviewed by yours truly). I've been using this for point-and-shoot WebSocket capture but I imagine it could also be useful for observability/metrics ingestion as well. Litelink supports a single writer per stream. I've been doing a lot of development and deployments on tiny VMs (2 vCPU, 8GB, 50-100GB disk) and didn't want the complexity or cost of managing central brokers (Kafka), databases (Postgres), and CDC/connectors just to get queryable WebSocket stream capture running. With litelink, you configure a log in code, and end-to-end setup takes <5 minutes (see the example scripts in the repo). The log is itself an Iceberg table (actually two: a local and archive table), so there's no second copy of your data to keep in sync or connector to manage. I'm sure there are still bugs, but I recently migrated all the capture feeds for a personal research project to litelink, and the experience has been night and day. Before that, I'd hand-rolled a capture system and was dealing with all the issues you'd expect (e.g. small file problem). I'll post some before/after stats in a comment below. I tried to channel the same ethos as LanceDB/Iceberg/SQLite. Everything runs local first without a network connection required. I've tried to abstract the complexity of stream/data lifecycle maintenance away behind a few public library methods. Hopefully someone else finds this useful! Let me know what you think. repo: https://github.com/nhobin219/litelink https://github.com/nhobin219/litelink spec: https://github.com/nhobin219/litelink/blob/main/docs/SPEC.md https://github.com/nhobin219/litelink/blob/main/docs/SPEC.md pypi: `pip install litelink`

jnbh 1mo ago
-- legacy -- - litelink - file byte stream rows files GB files GB ratio ratio ------------------------------------------------------------------------ md/asset-stats-1m 283,283,881 1,285 9.97 872 12.58 1x 0.79x md/trades-b 189,207,469 29,598 13.00 1,400 12.62 21x 1.03x md/trades-a 58,928,828 39,195 5.20 398 4.63 98x 1.12x md/depth 57,568,622 39,026 17.65 1,519 17.07 26x 1.03x md/instruments-a 41,384,403 200,360 9.27 1,514 1.19 132x 7.82x md/asset-stats 9,036,011 76,274 1.64 51 0.73 1,496x 2.25x md/instruments-b † 7,164,452 3,081 5.46 1,246 9.33 2x 0.59x md/depth-archive 6,326,501 2,002 0.53 20 0.58 100x 0.91x md/funding-pred 4,921,010 15,351 0.15 13 0.04 1,181x 3.44x md/candles-1h 1,216,025 9,668 0.14 4 0.07 2,417x 1.91x md/events 1,031,573 35,935 0.75 17 0.39 2,114x 1.91x soc/forum 778,702 19,945 0.86 32 0.03 623x 28.3x md/pools 433,499 7,513 0.27 11 0.14 683x 1.91x soc/posts-b 262,575 35,634 0.42 6 0.15 5,939x 2.75x soc/posts-a 249,136 51,914 0.77 24 0.01 2,163x 57.0x md/token-profiles 234,029 15,595 0.19 3 0.01 5,198x 21.7x md/token-boosts 231,719 15,441 0.21 3 0.00 5,147x 53.3x soc/counts 103,058 164 0.00 0 0.00 all 2.42x md/trending 39,149 5,207 0.10 1 0.03 5,207x 3.49x soc/messages 30,218 9,568 0.10 0 0.00 all 25.1x soc/posts-c 29,617 13,192 0.24 1 0.00 13,192x 86.7x md/holders 9,746 96 0.08 3 0.07 32x 1.11x md/authorities 9,644 137 0.00 0 0.00 all 2.23x md/universe 6,760 29 0.00 0 0.00 all 6.00x soc/stats 6,690 6,690 0.03 0 0.00 all 68.4x md/funding 4,486 1,298 0.01 0 0.00 all 96.0x soc/polls 2,050 2,114 0.01 0 0.00 all 199x md/candles-1d 191 1,597 0.03 0 0.00 all 2,363x soc/filter-a 24 24 0.00 0 0.00 all 7.39x soc/filter-b 1 23 0.00 0 0.00 all 490x ------------------------------------------------------------------------ TOTAL (30 streams) 662,500,069 637,956 67.05 7,138 59.70 89x 1.12x NOTE: litelink auto-generates a `litelink_offset` column (unique, doesn't compress). This column dominates bytes/row in rows where the the byte ratio is < 1 (litelink data is larger). The old, hand-rolled capture lacked any monotonically increasing offset. Also, for the rows that say byte ratio is"all" (reporting zero litelink files), this is because all of the data fits in litestream replicated WAL and there are not yet any compacted files
jnbh 1mo ago
P.S.: my hot (and quite possibly wrong) take on the near future of software development is that we are going to see a shift toward local first tooling since agents can super easily deploy and use these libraries in isolated sandboxes (pods/containers/VMs). Obviously, this doesn't apply to everything and there are many use cases that will still require distributed, zero-downtime deployments. But, for most applications, the cost of deploying code/infra can be made so cheap with an ergonomic library that I think a lot of the appeal of centralized services/servers is decaying (especially if remote object storage can be plugged into the storage layer of a library/tool). Anyhow, those are just my two cents.

9 points by jnbh 1mo ago

jnbh 1mo ago

-- legacy -- - litelink - file byte stream rows files GB files GB ratio ratio ------------------------------------------------------------------------ md/asset-stats-1m 283,283,881 1,285 9.97 872 12.58 1x 0.79x md/trades-b 189,207,469 29,598 13.00 1,400 12.62 21x 1.03x md/trades-a 58,928,828 39,195 5.20 398 4.63 98x 1.12x md/depth 57,568,622 39,026 17.65 1,519 17.07 26x 1.03x md/instruments-a 41,384,403 200,360 9.27 1,514 1.19 132x 7.82x md/asset-stats 9,036,011 76,274 1.64 51 0.73 1,496x 2.25x md/instruments-b † 7,164,452 3,081 5.46 1,246 9.33 2x 0.59x md/depth-archive 6,326,501 2,002 0.53 20 0.58 100x 0.91x md/funding-pred 4,921,010 15,351 0.15 13 0.04 1,181x 3.44x md/candles-1h 1,216,025 9,668 0.14 4 0.07 2,417x 1.91x md/events 1,031,573 35,935 0.75 17 0.39 2,114x 1.91x soc/forum 778,702 19,945 0.86 32 0.03 623x 28.3x md/pools 433,499 7,513 0.27 11 0.14 683x 1.91x soc/posts-b 262,575 35,634 0.42 6 0.15 5,939x 2.75x soc/posts-a 249,136 51,914 0.77 24 0.01 2,163x 57.0x md/token-profiles 234,029 15,595 0.19 3 0.01 5,198x 21.7x md/token-boosts 231,719 15,441 0.21 3 0.00 5,147x 53.3x soc/counts 103,058 164 0.00 0 0.00 all 2.42x md/trending 39,149 5,207 0.10 1 0.03 5,207x 3.49x soc/messages 30,218 9,568 0.10 0 0.00 all 25.1x soc/posts-c 29,617 13,192 0.24 1 0.00 13,192x 86.7x md/holders 9,746 96 0.08 3 0.07 32x 1.11x md/authorities 9,644 137 0.00 0 0.00 all 2.23x md/universe 6,760 29 0.00 0 0.00 all 6.00x soc/stats 6,690 6,690 0.03 0 0.00 all 68.4x md/funding 4,486 1,298 0.01 0 0.00 all 96.0x soc/polls 2,050 2,114 0.01 0 0.00 all 199x md/candles-1d 191 1,597 0.03 0 0.00 all 2,363x soc/filter-a 24 24 0.00 0 0.00 all 7.39x soc/filter-b 1 23 0.00 0 0.00 all 490x ------------------------------------------------------------------------ TOTAL (30 streams) 662,500,069 637,956 67.05 7,138 59.70 89x 1.12x NOTE: litelink auto-generates a `litelink_offset` column (unique, doesn't compress). This column dominates bytes/row in rows where the the byte ratio is < 1 (litelink data is larger). The old, hand-rolled capture lacked any monotonically increasing offset. Also, for the rows that say byte ratio is"all" (reporting zero litelink files), this is because all of the data fits in litestream replicated WAL and there are not yet any compacted files

jnbh 1mo ago

P.S.: my hot (and quite possibly wrong) take on the near future of software development is that we are going to see a shift toward local first tooling since agents can super easily deploy and use these libraries in isolated sandboxes (pods/containers/VMs). Obviously, this doesn't apply to everything and there are many use cases that will still require distributed, zero-downtime deployments. But, for most applications, the cost of deploying code/infra can be made so cheap with an ergonomic library that I think a lot of the appeal of centralized services/servers is decaying (especially if remote object storage can be plugged into the storage layer of a library/tool). Anyhow, those are just my two cents.