4 ms·

I really like the columns explorer, https://motherduck.com/blog/introducing-column-explorer/ https://motherduck.com/blog/introducing-column-explorer/. Just a f

by stared 2y ago

I really like the columns explorer, https://motherduck.com/blog/introducing-column-explorer/ https://motherduck.com/blog/introducing-column-explorer/. Just a few days ago I have been looking for existing column explorers that look like from Kaggle Dataset, but I was not able to find anything. And this one by DuckDB is better!

thenaturalist 2y ago
I have seen a ton of DB GUI clients/ cloud based data tools for analytics purposes and the fact that MotherDuck's column explorer/ column data distribution is hands down the best I know is puzzling me. It seems nobody else besides them cares. Seeing data distribution, unique values, min/ max/ percentiles is so easy and powerful. Really commend whoever came up with that. It's a bit of a shame this metadata cannot be queried itself, would be immensely useful for automatic data profiling/ QA at scale.
- mritchie712 2y ago
  it's partly bc this would be extremely slow and expensive with many other databases (e.g. it'd be really slow on postgres, very expensive on snowflake).
- stared 2y ago
  Do you know if there is any open source TypeScript component that can be used in a project?
  - thenaturalist 2y ago
    None of the UI is OSS as far as I am aware. :/
    - hamilton 2y ago
      True; at the moment the UI is not open source. We've talked about releasing the Column Explorer as a standalone component, but haven't been able to prioritize it yet. We'd like to!
      - stared 2y ago
        It would be wonderful for the community - as it is the best tool in the class. And foremost - thank you for designing this awesome component!
- aszen 2y ago
  A similar open source column explorer is : https://github.com/manzt/quak https://github.com/manzt/quak See it's demo: https://manzt.github.io/quak/?source=https://pub-2fc10ef67241467c8327dac35d878da9.r2.dev/athletes.parquet https://manzt.github.io/quak/?source=https://pub-2fc10ef6724...
oulipo 2y ago
Seems heavily inspired by the column summary of ObservableHQ, but that's nice!
- hamilton 2y ago
  (I designed and built the Column Explorer feature) Observable's column summary feature is very nice! But I do think there's a very common lineage around these kinds of diagnostics which motivated both Observables and ours. See Jeff Heer's profiler paper[1] for more. I'm very passionate about this area because I think "first mile problems" are underserved by most tools, but they take the longest to work out. We had to do some gnarly things[2] to make this feature work well; and there's a lot of room to make it scale nicely and cover all DuckDB data types. [1] http://vis.stanford.edu/papers/profiler http://vis.stanford.edu/papers/profiler [2] https://motherduck.com/blog/introducing-column-explorer/ https://motherduck.com/blog/introducing-column-explorer/
  - oulipo 2y ago
    Interesting, I guess there's plenty of ideas to grab from their work too! https://observablehq.com/documentation/cells/data-table https://observablehq.com/documentation/cells/data-table
  - thenaturalist 2y ago
    Personally, thank you for this awesome work! I find the ease and intuitiveness of navigating it as well as the clarity of the information presented even for the density of a small window or many columns outstandingly pleasant. Kudos to you!

by stared 2y ago

thenaturalist 2y ago

I have seen a ton of DB GUI clients/ cloud based data tools for analytics purposes and the fact that MotherDuck's column explorer/ column data distribution is hands down the best I know is puzzling me. It seems nobody else besides them cares. Seeing data distribution, unique values, min/ max/ percentiles is so easy and powerful. Really commend whoever came up with that. It's a bit of a shame this metadata cannot be queried itself, would be immensely useful for automatic data profiling/ QA at scale.

mritchie712 2y ago
it's partly bc this would be extremely slow and expensive with many other databases (e.g. it'd be really slow on postgres, very expensive on snowflake).
stared 2y ago
Do you know if there is any open source TypeScript component that can be used in a project?
- thenaturalist 2y ago
  None of the UI is OSS as far as I am aware. :/
  - hamilton 2y ago
    True; at the moment the UI is not open source. We've talked about releasing the Column Explorer as a standalone component, but haven't been able to prioritize it yet. We'd like to!
    - stared 2y ago
      It would be wonderful for the community - as it is the best tool in the class. And foremost - thank you for designing this awesome component!
aszen 2y ago
A similar open source column explorer is : https://github.com/manzt/quak https://github.com/manzt/quak See it's demo: https://manzt.github.io/quak/?source=https://pub-2fc10ef67241467c8327dac35d878da9.r2.dev/athletes.parquet https://manzt.github.io/quak/?source=https://pub-2fc10ef6724...

oulipo 2y ago

Seems heavily inspired by the column summary of ObservableHQ, but that's nice!

hamilton 2y ago
(I designed and built the Column Explorer feature) Observable's column summary feature is very nice! But I do think there's a very common lineage around these kinds of diagnostics which motivated both Observables and ours. See Jeff Heer's profiler paper[1] for more. I'm very passionate about this area because I think "first mile problems" are underserved by most tools, but they take the longest to work out. We had to do some gnarly things[2] to make this feature work well; and there's a lot of room to make it scale nicely and cover all DuckDB data types. [1] http://vis.stanford.edu/papers/profiler http://vis.stanford.edu/papers/profiler [2] https://motherduck.com/blog/introducing-column-explorer/ https://motherduck.com/blog/introducing-column-explorer/
- oulipo 2y ago
  Interesting, I guess there's plenty of ideas to grab from their work too! https://observablehq.com/documentation/cells/data-table https://observablehq.com/documentation/cells/data-table
- thenaturalist 2y ago
  Personally, thank you for this awesome work! I find the ease and intuitiveness of navigating it as well as the clarity of the information presented even for the density of a small window or many columns outstandingly pleasant. Kudos to you!