mccanne

5 ms·

1.
by mccanne 7mo ago
Hey really cool! Tcpdump co-author here. If you're interested in the origin story of the tcpdump language, bpf, and pcap, I had fun putting together a talk a number of years ago... https://www.youtube.com/watch?v=XHlq
2.
by mccanne 1y ago
For sure... the use of JSON here seems orthogonal to Jamie's point of constructing complex nested values in SQL with a single backend query. Modern SQLs support all this as native types (presuming the result can fit in a homogeneous r
3.
by mccanne 2y ago
Relevant discussion from a few years back https://news.ycombinator.com/item?id=28221654
4.
by mccanne 2y ago
And somewhat ironically here, SuperDB not only implements sum-type decoding of JSON in package unpack, but it also implements native sum types in a superset of JSON that we call Super JSON (with a query language that understands how to rip
5.
by mccanne 2y ago
Nice article! Decoding sum types into Go interface values is obviously tricky stuff, but it gets even harder when you have recursive data structures as in an abstract syntax tree (AST). The article doesn't address this. Since there w
6.
by mccanne 2y ago
Really cool though typing ">" after "|" is a pain https://github.com/brimdata/super/blob/main/docs/language/pi...
7.
by mccanne 3y ago
Necessity is the mother of invention. MapReduce-based systems were developed because the state-of-the-art RDBMS systems of that age could not scale to the needs of the Googles/Yahoos/Facebooks during the phenomenal growth spurt o
8.
by mccanne 3y ago
Yes, the name conflict is unfortunate. We named our project Zed in 2021 before the Zed text editor was a thing.
9.
by mccanne 3y ago
Apologies if our examples in the article weren't rich enough or clear. The beauty of the error type is that you can wrap any value in an error and make the error as rich as you'd like, even stacking errors from different stages o
10.
by mccanne 3y ago
Yes, great point! The idea is that you can fix up a problem with data in place, while you're updating your ingest pipeline to handle whatever is causing the problem. You can do a transform on the errors into clean data, delete the er
11.
by mccanne 3y ago
Hey, thanks for bringing up storage efficiency. This is an often cited objection and a big motivation of the Zed project actually. In Zed, data is organized around a self-describing type system rather than schemas, and unlike the relation
12.
Explore large JSON values with zui and zed(brimdata.io)
4 points by mccanne 4y ago|0 comments
13.
Explore big JSON objects with zui(youtube.com)
3 points by mccanne 4y ago|1 comments
14.
by mccanne 4y ago
A quick look at the forthcoming release of zui-1.0. Explore gnarly JSON data with ease!
15.
Easy Deserialization of Go Interface Values(brimdata.io)
4 points by mccanne 4y ago|0 comments
16.
by mccanne 4y ago
Simon, brilliant observations here and kudos on sqlite-utils. I'm all for layers, a fundamental approach in our field to tame complexity. And the SQL model and SQLite have stood the test of time and are solid foundations. I'm jus
17.
by mccanne 4y ago
Hear, hear!
18.
by mccanne 4y ago
Author here. All good points. Yes, you can build a super-structured type system on top of tables. EdgeDB does this well. And you can put JSON into relational columns. Then you might ask what the "type" of that column is? Wel
19.
by mccanne 4y ago
Author here. Agreed! Validation is important. While I didn't make this point in the article, our thinking is schema validation does not require that the serialization format utilize schemas as the building block and you can always i
20.
by mccanne 4y ago
Author here. This totally makes sense. The challenge here is you need to store the type definitions somewhere (e.g., in the .proto files) and any system that processes protocol buffers needs to know which proto to apply to which messages.
21.
Super-Structured Data: Rethinking the Schema(brimdata.io)
108 points by mccanne 4y ago|46 comments
22.
by mccanne 4y ago
zq good/ -
23.
by mccanne 4y ago
Wow, thanks. Coincidentally, after hearing of a friend's woes dealing with massive amounts of CSV coming from a BPF-instrumental kernel, I played around a bit with integrating Zed and BPF. Just an experimental toy (and the repo is alr
24.
by mccanne 4y ago
That's a cool idea. We've had many collaborators using Zed lakes for search at smallish scale and we are still building the breadth of features needed for a serious search platform, but I think we have a nice architecture that ho
25.
by mccanne 4y ago
Hi, all. Author here. Thanks for all the great feedback. I've learned a lot from your comments and pointers. The Zed project is broader than "a jq alternative" and my bad for trying out this initial positioning. I do know
26.
Zq: An easier and faster alternative to jq(brimdata.io)
462 points by mccanne 4y ago|223 comments
27.
by mccanne 5y ago
There are a few examples in the ZSON spec... https://github.com/brimdata/zed/blob/main/docs/formats/zson.... And you can easily see whatever data you'd like formatted as ZSON using the &qu
28.
by mccanne 5y ago
Also, if any of you find problems with the Zed spec(s), we'd love to hear about them. "Now" would be a good time to make changes / fix flaws.
29.
by mccanne 5y ago
This is a very real problem being addressed here and I am intrigued by all the great comments in this thread. In the Zed project, we've been thinking about and iterating on a better data model for serialization for a few years, and hav
30.
by mccanne 10y ago
I only saw the bits around 230pm, but if by "startup school" they were trying to explain to the inexperienced entrepreneurs how they can be exploited by third-tier investors who enjoy mocking them, cutting them off while they are

More ›