dm03514

7 ms·

31.
by dm03514 2y ago
“That single gram of silica gel could have an internal surface area of eight hundred square meters—the size of almost two basketball courts.“ Can someone help explain this? I’m having a hard time wrapping my head around this. The tiny packe
32.
by dm03514 2y ago
In my experiences of living in suburbs for 30 years, I’ve seen the default is to ignore neighbors. I don’t really get this. Our communities have so much in common and so much overlap, we shop at the same stores, go to the same parks, get st
33.
by dm03514 2y ago
Amazing! Thank you for the update, this will be huge
34.
by dm03514 2y ago
What’s the furthest a recipient will be forced to travel? How many offices per population area? What’s the current rate of calls and direct deposit modifications? At that rate what load will it put on the offices? How long is the average in
35.
by dm03514 2y ago
I've mentioend this whenever iceberg comes up. It's wild how immature the ecosystem is still. Duckdb itself lacks the ability to write iceberg.... https://duckdb.org/docs/stable/extensions/iceberg&#x
36.
by dm03514 2y ago
pfsh who needs to write data??? ;p If you have streaming data as a source, I built a side project to write streaming data to s3 in iceberg format: https://sql-flow.com/docs/tutorials/iceberg-sink https:/&#x2
37.
by dm03514 2y ago
Interesting would love to learn more about their methodology. Like are they straight searching “software developer” on indeed? https://fred.stlouisfed.org/series/IHLIDXUSTPSOFTDEVE —- It’s rough out there, I see former
38.
by dm03514 2y ago
First of all I think this article was put together quite well, it was succinct had a great flow and was very easy to follow. The data was laid out to support the case; overall it was great to read. Understanding a problem requires a holisti
39.
by dm03514 2y ago
Fun, I did a similar project some years ago , I always like it when multiple people arrive at the same approach without ever communicating :) https://github.com/dm03514/test-engine/blob/master/tutorials..
40.
by dm03514 2y ago
Very Cool! Which use cases are you targeting? I'm surprised at the relatively sparse state of streaming. I feel like there are: - Flink - Spark - ~Benthos - Arroyo - Then mostly custom / bespoke frameworks Maybe there just isn&#
41.
by dm03514 2y ago
I'm building a stream processing framework using DuckDB. https://github.com/turbolytics/sql-flow The goal is to create a stream processing framework that supports SQL jobs. Apache Flink supports this but is very h
42.
by dm03514 2y ago
We just added Clickhouse sink support to SQLFlow: - https://github.com/turbolytics/sql-flow/issues/100 - https://github.com/turbolytics/sql-flow/pull/116 The Clickhouse python
43.
Show HN: Sink 5000 rows/SEC from Kafka –> Postgres using Python and 250MiB Mem(sql-flow.com)
2 points by dm03514 2y ago|0 comments
44.
by dm03514 2y ago
I think iceberg solves a lot of big data problems, for handling huge amounts of data on blob storage, including partitioning, compaction and ACID semantics. I really like the way the catalog standard can decouple underlying storage as well.
45.
by dm03514 2y ago
Hello! Just released a version of SQLFlow that adds support for streaming data to iceberg tables. https://github.com/turbolytics/sql-flow?tab=readme-ov-file#s... pyiceberg does the heavy lifting. It is a great project
46.
by dm03514 2y ago
Created a tutorial on using SQLFlow to interact with the Bluesky firehose: https://www.linkedin.com/pulse/learning-sqlflow-using-bluesk...
47.
by dm03514 2y ago
I’m thinking of this as a lightweight (single node) alternative in the same way duckdb is focused on data that can be processed by a single node. I think / hope sqlflow will be a viable/lightweight/cost effective/easy to
48.
Show HN: Execute SQL against Bluesky firehose(github.com)
100 points by dm03514 2y ago|5 comments
49.
by dm03514 2y ago
Hello, I’ve been working on a project that embeds duckdb for stream processing. I just added support for websocket sources which enables sql over the Bluesky firehouse. https://github.com/turbolytics/sql-flow?tab=readme
50.
by dm03514 2y ago
This looks really interesting, I instantly found a drawing of my kids to upload because I thought he would love it, but I was unable to even try it once without paying.
51.
Show HN: Librarian a Modern Kafka Connect Toolkit(github.com)
1 points by dm03514 2y ago|0 comments
52.
by dm03514 2y ago
I saw one paragraph on transaction isolation levels hidden in the word salad
53.
by dm03514 2y ago
I agree with the post :) Writing a book was very personally fulfilling. I wrote about my self published book earnings a couple years ago. https://on-systems.tech/blog/130-2022-self-published-book-ea... This quote got m
54.
by dm03514 2y ago
Hello Leo! Are you open for remote for the EM role, or is it on-site? Thank you!
55.
Ask HN: Books on strategic military operations and politics
1 points by dm03514 2y ago|4 comments
56.
by dm03514 2y ago
Hello everyone! I'm researching how to generate guaranteed globally unique IDS, and have been reading about Twitter's snowflake ID: https://en.wikipedia.org/wiki/Snowflake_ID To understand it better I created
57.
Show HN: 128bit flake id generator in go (library and server)(github.com)
2 points by dm03514 2y ago|1 comments
58.
by dm03514 2y ago
This is a n00b question, but how do regulations apply to private human crewed missions in the US? Are there clear guidelines already written? I’m assuming it inherits a lot of rules of general aviation? But since private firms launching int
59.
SQLFlow: DuckDB for Stream Processing – 45k messages/second(github.com)
2 points by dm03514 3y ago|1 comments
60.
by dm03514 3y ago
Hello Everyone, having lots of fun figuring out ways to use DuckDB in common data operations. Already seeing huge savings in traditional data engineering applications replacing data warehouse queries with duckdb over blob storage (parquet).

More ›