6 ms·

I've often found it's better to be more verbose in situations like this, but providing the "tree" entries as rows. name,value,color World Asia China

I've often found it's better to be more verbose in situations like this, but providing the "tree" entries as rows. name,value,color World Asia China,1409517397 India,1339180127 Indonesia,263991379 becomes region,subregion,country,value,color World,Asia,China,1409517397 World,Asia,India,1339180127 World,Asia,Indonesia,263991379 This still compresses very nicely, and allows data to be output in any order. The tree form is more of a presentation thing, which should be the last step of a data processing pipeline. As it is, the tree nodes themselves have no names or context, so it makes it harder to consume. Is the second-level entry a region? Easy to tell from this example, but if you're consuming a large data set you're forced to first give names to these columns.

mbostock 8y ago
Your suggestion requires a fixed-depth hierarchy (such as three levels in your example), and then it requires a lot of redundancy. Here, I want to eliminate the redundancy not primarily for performance reasons, but to make it easier to edit the data by hand, for example to cut-and-paste some lines to move them around within the tree without having to change the leading values. To get a sense of what I mean, see this editor which lets you interactively construct a treemap: https://beta.observablehq.com/@mbostock/treemap-o-matic https://beta.observablehq.com/@mbostock/treemap-o-matic
- mpolichette 8y ago
  I'm not arguing against the CST format, but I don't think redundancy is necessarily bad or hard to work with. It can be good because each line is now an independent idea and doesn't require the indentation context, the items can be sorted or moved easily. As for hard to work with... a lot of editors have multi-cursor support these days, which makes editing these things in bulk pretty straight forward for smaller sizes and then there's always sed and awk. ;) Given all that, CST seems like it might be cool for building some quick trees like the given UIs you present. I'd probably keep to to rows for longer storage though.
- lsh 8y ago
  CSV is just a serialisation format like JSON or XML. CST appears to put the emphasis on readability (not a bad thing!). I've taken to using XSV to pretty-print CSV data in bug reports or feature requests for coworkers who give me CSV or spreadsheets to work with. The data doesn't need to go any further, it's already been processed, it just needs to be clearer to understand than raw CSV/JSON/XML: https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv (link has examples of pretty-printed CSV)
- siddboots 8y ago
  Something that I've used in the past: store the hierarchical data as a breadcrumb-like string in an ordinary column. This means the parsing is just ordinary csv parsing, followed by a relatively trivial extra step to parse the breadcrumb columns. You also retain the ability to describe arbitrary depth hierarchies: country,value World/Asia/China, 1409517397 World/Asia/India, 1339180127 World/Asia/Indonesia, 263991379 An added benefit of this approach is that you can have hierarchies in multiple columns. For example: datetime, source_account, destination_account, amount 2016-07-01, Assets/ProjectFunding, Assets/Capital/Delivery, 134.2 2017-07-01, Assets/ProjectFunding, Assets/Capital/Delivery, 72.2 2016-07-01, Assets/ProjectFunding, Assets/Capital/Other, 5.0 2016-07-01, Assets/ProjectFunding, Expenses/Development, 4.7 2017-07-01, Assets/ProjectFunding, Expenses/Development, 1.6 2017-07-01, Assets/Cash, Expenses/OPEX, 0.96 2018-07-01, Assets/Cash, Expenses/OPEX, 1.62
  - kqr 8y ago
    That has the meta-problem of only allowing two different levels of hierarchies. For one more, you'd have to introduce a new separator. I feel like the fixpoint of this function is going back to only one separator, or something, but I am not seeing the reasoning clearly.
breck 8y ago
So far I've only used this format for tiny datasets smaller than 10k root entries (and around ~1M nodes). And even then, before I use Python/R/etc to analyze things, I transform it into a normalized table form exactly like you've shown above. I think the tree form is great for the smaller datasets that need to be edited or reviewed manually, not bulk machine generated data. For one project I have thousands of files, each with data stored in a tree, all stored in a git repo with many editors, sort of like a data wiki. It also works for the "presentation thing", as you say. You can use a table-backed storage system but present to the user a tree, and when they edit a node you propagate the change to the appropriate row. Finally, some editing plugins can give you good highlighting and more context if you create a grammar file. Source: have been tinkering with this stuff for a couple years.
philsnow 8y ago
I like dealing with data that's line-denormalized like this since it's a lot easier to manipulate with simple things like grep and awk. If you do too, and you find yourself dealing with JSON a lot, take a look at gron [0], it takes JSON and denormalizes all the structure into a line-oriented equivalent. % echo '{"root": {"left": [1, 1.4, 2, 2.88], "right": ["a", "b"]}}' | jq . { "root": { "left": [ 1, 1.4, 2, 2.88 ], "right": [ "a", "b" ] } } % echo '{"root": {"left": [1, 1.4, 2, 2.88], "right": ["a", "b"]}}' | gron json = {}; json.root = {}; json.root.left = []; json.root.left[0] = 1; json.root.left[1] = 1.4; json.root.left[2] = 2; json.root.left[3] = 2.88; json.root.right = []; json.root.right[0] = "a"; json.root.right[1] = "b"; [0] https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron
- thechao 8y ago
  AHHA! My brother in arms! I also have a gron-like ML called 'KVIN'[0]. It'd be like this: root.left[0] = 1 .left[1] = 1.4 .left[2] = 2 root.left[3] = 2.88 ..right[0] = "a" .right[1] = "b" [0] https://github.com/jaroslov/kvin https://github.com/jaroslov/kvin
- aasasd 8y ago
  Eurgh, you've just put another blemish on this world's karma with that sad affair of manipulating structured data as indiscriminate mess of letters. If keys in your input contain dots, what will grep and awk do to help you out? Meanwhile, the JQ tool could do your greppings and awkings but preserving and perusing the structure of the document notwithstanding its idiosyncrasies. (A possible problem with JQ, though, is that it may be difficult to query a structure which varies, i.e. use entries on a nesting level that isn't known in advance.)
  - anyfoo 8y ago
    > If keys in your input contain dots, what will grep and awk do to help you out? Without knowing gron, I presume the dots are escaped/quoted properly, and you would do the same in your grep/awk usage. What if your keys contain equal signs and curly braces, how will json help you out? > Meanwhile, the JQ tool could do your greppings and awkings but preserving and perusing the structure of the document notwithstanding its idiosyncrasies. The structure of the data is preserved (I don't know what you mean with "notwithstanding its idiosyncrasies"). This is just a different, albeit redundant, but entirely equivalent representation of the same document. Despite their redundancy, denormalized representations do have advantages.

mbostock 8y ago

Your suggestion requires a fixed-depth hierarchy (such as three levels in your example), and then it requires a lot of redundancy. Here, I want to eliminate the redundancy not primarily for performance reasons, but to make it easier to edit the data by hand, for example to cut-and-paste some lines to move them around within the tree without having to change the leading values. To get a sense of what I mean, see this editor which lets you interactively construct a treemap: https://beta.observablehq.com/@mbostock/treemap-o-matic https://beta.observablehq.com/@mbostock/treemap-o-matic

mpolichette 8y ago
I'm not arguing against the CST format, but I don't think redundancy is necessarily bad or hard to work with. It can be good because each line is now an independent idea and doesn't require the indentation context, the items can be sorted or moved easily. As for hard to work with... a lot of editors have multi-cursor support these days, which makes editing these things in bulk pretty straight forward for smaller sizes and then there's always sed and awk. ;) Given all that, CST seems like it might be cool for building some quick trees like the given UIs you present. I'd probably keep to to rows for longer storage though.
lsh 8y ago
CSV is just a serialisation format like JSON or XML. CST appears to put the emphasis on readability (not a bad thing!). I've taken to using XSV to pretty-print CSV data in bug reports or feature requests for coworkers who give me CSV or spreadsheets to work with. The data doesn't need to go any further, it's already been processed, it just needs to be clearer to understand than raw CSV/JSON/XML: https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv (link has examples of pretty-printed CSV)
siddboots 8y ago
Something that I've used in the past: store the hierarchical data as a breadcrumb-like string in an ordinary column. This means the parsing is just ordinary csv parsing, followed by a relatively trivial extra step to parse the breadcrumb columns. You also retain the ability to describe arbitrary depth hierarchies: country,value World/Asia/China, 1409517397 World/Asia/India, 1339180127 World/Asia/Indonesia, 263991379 An added benefit of this approach is that you can have hierarchies in multiple columns. For example: datetime, source_account, destination_account, amount 2016-07-01, Assets/ProjectFunding, Assets/Capital/Delivery, 134.2 2017-07-01, Assets/ProjectFunding, Assets/Capital/Delivery, 72.2 2016-07-01, Assets/ProjectFunding, Assets/Capital/Other, 5.0 2016-07-01, Assets/ProjectFunding, Expenses/Development, 4.7 2017-07-01, Assets/ProjectFunding, Expenses/Development, 1.6 2017-07-01, Assets/Cash, Expenses/OPEX, 0.96 2018-07-01, Assets/Cash, Expenses/OPEX, 1.62
- kqr 8y ago
  That has the meta-problem of only allowing two different levels of hierarchies. For one more, you'd have to introduce a new separator. I feel like the fixpoint of this function is going back to only one separator, or something, but I am not seeing the reasoning clearly.

breck 8y ago

So far I've only used this format for tiny datasets smaller than 10k root entries (and around ~1M nodes). And even then, before I use Python/R/etc to analyze things, I transform it into a normalized table form exactly like you've shown above. I think the tree form is great for the smaller datasets that need to be edited or reviewed manually, not bulk machine generated data. For one project I have thousands of files, each with data stored in a tree, all stored in a git repo with many editors, sort of like a data wiki. It also works for the "presentation thing", as you say. You can use a table-backed storage system but present to the user a tree, and when they edit a node you propagate the change to the appropriate row. Finally, some editing plugins can give you good highlighting and more context if you create a grammar file. Source: have been tinkering with this stuff for a couple years.

philsnow 8y ago

I like dealing with data that's line-denormalized like this since it's a lot easier to manipulate with simple things like grep and awk. If you do too, and you find yourself dealing with JSON a lot, take a look at gron [0], it takes JSON and denormalizes all the structure into a line-oriented equivalent. % echo '{"root": {"left": [1, 1.4, 2, 2.88], "right": ["a", "b"]}}' | jq . { "root": { "left": [ 1, 1.4, 2, 2.88 ], "right": [ "a", "b" ] } } % echo '{"root": {"left": [1, 1.4, 2, 2.88], "right": ["a", "b"]}}' | gron json = {}; json.root = {}; json.root.left = []; json.root.left[0] = 1; json.root.left[1] = 1.4; json.root.left[2] = 2; json.root.left[3] = 2.88; json.root.right = []; json.root.right[0] = "a"; json.root.right[1] = "b"; [0] https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron

thechao 8y ago
AHHA! My brother in arms! I also have a gron-like ML called 'KVIN'[0]. It'd be like this: root.left[0] = 1 .left[1] = 1.4 .left[2] = 2 root.left[3] = 2.88 ..right[0] = "a" .right[1] = "b" [0] https://github.com/jaroslov/kvin https://github.com/jaroslov/kvin
aasasd 8y ago
Eurgh, you've just put another blemish on this world's karma with that sad affair of manipulating structured data as indiscriminate mess of letters. If keys in your input contain dots, what will grep and awk do to help you out? Meanwhile, the JQ tool could do your greppings and awkings but preserving and perusing the structure of the document notwithstanding its idiosyncrasies. (A possible problem with JQ, though, is that it may be difficult to query a structure which varies, i.e. use entries on a nesting level that isn't known in advance.)
- anyfoo 8y ago
  > If keys in your input contain dots, what will grep and awk do to help you out? Without knowing gron, I presume the dots are escaped/quoted properly, and you would do the same in your grep/awk usage. What if your keys contain equal signs and curly braces, how will json help you out? > Meanwhile, the JQ tool could do your greppings and awkings but preserving and perusing the structure of the document notwithstanding its idiosyncrasies. The structure of the data is preserved (I don't know what you mean with "notwithstanding its idiosyncrasies"). This is just a different, albeit redundant, but entirely equivalent representation of the same document. Despite their redundancy, denormalized representations do have advantages.