7 ms·
Ask HN: How do you view large JSON files?
I use JSON Viewer [1] to view data in JSON format but it freezes on files larger than 1 MB. I want to be able to easily collapse and expand elements to understand the structure of the data.
[1] https://jsonviewer.codeplex.com/
- rileytg 10y agoIntelliJ scratch files
- YuriNiyazov 10y agojq
- mtmail 10y agoThere's addons/extension for Chrome and Firefox. Both also called JSON Viewer (different author). For really large files I use a command-line tool plus grep and awk though https://stedolan.github.io/jq/ https://stedolan.github.io/jq/
- rickyc091 10y agoYep, all that above + `sed`.
- chuchuva 10y agojq looks promising but I still need to view the JSON file first. I want to be able to somehow get the general understanding of the structure very quickly and interactively. The tree control works very well for that. JSON Viewer is almost perfect but it doesn't work with local files.
- bsg75 10y agojq . file.json | less
- bsg75 10y agojq . file.json | less or curl ... | jq . | less
- dwaltrip 10y agoJSON Formatter for Chrome works with local files (you have to check the box in the settings to enable it). It's super handy. Drag and drop json files into a new tab. Link: https://chrome.google.com/webstore/detail/json-formatter/bcjindcccaagfpapjjmafapmmgkkhgoa?hl=en https://chrome.google.com/webstore/detail/json-formatter/bcj...
- slapresta 10y agoI usually use jq to understand the structure of the file. `keys` will get you all the key names in an object; `with_entries(.value |= type)` will replace each element on an object with its type. You can quickly delve into a JSON and gain some understanding of it with jq.
- mtsmithhn 10y agoYou could try the JSONView chrome extension.
- patrickmn 10y agoLink: https://chrome.google.com/webstore/detail/jsonview/chklaanhfefbnpoihckbnefhakgolnmc https://chrome.google.com/webstore/detail/jsonview/chklaanhf...
- chuchuva 10y agoI'm already using JSONView Chrome extension and it's great but it doesn't work with local files. Nor can I copy and paste JSON.
- ZoF 10y agoWhy not just throw the file(s) up on a box somewhere so you can use it, or host it locally, can you not use locally hosted links with it or something?
- jijji 10y agoprint_r(json_decode(file_get_contents("http://url.com/" http://url.com/"), TRUE));
- chuchuva 10y agoWhat about collapsing/expanding elements interactively?
- poooogles 10y agoThe Python REPL and the JSON lib. Probably not the most efficient if you're starting out but if you're familiar with Python it can be quite effective.
- ezstar 10y agoThis. I use the Ruby REPL, but same principle. You have to be a little careful about not accidentally printing out the whole thing to stdout, but after its loaded into memory you can check keys at a given level, drill into the substructure, etc.
- satori99 10y agoOn Windows I use Notepad++, which handles large local JSON files just fine, and has a collapsible tree structure.
- jsonninja 10y agoFor very large files, like 1M rows and 100's of MB's, of JSON arrays try http://www.jsondata.ninja http://www.jsondata.ninja
- JesseAldridge 10y agoI think Sublime Text should be able to handle fairly large json files without a problem. If not, I think I would just split data into multiple smaller files. Something like the python code below should work, assuming the file can fit in memory. If not, I assume you can find some json lib that can work in streaming mode and then do the same thing. import json, os json_input = '''{ "foo": 1, "bar": 2, "baz": 3, "bug": 4, "thing": 5 }''' entries_per_group = 2 if not os.path.exists('sub_files'): os.mkdir('sub_files') main_d = json.loads(json_input) iter = main_d.iteritems() for group_count in range(10 ** 6): sub_d = {} try: for _ in range(entries_per_group): k, v = iter.next() sub_d[k] = v except StopIteration: break finally: json_output = json.dumps(sub_d, indent=2) with open(os.path.join('sub_files', '{}.json'.format(group_count)), 'w') as f: f.write(json_output)
- i0exception 10y agoIf you're familiar with vim, you can do this cat unformatted.json | python -mjson.tool | vim - And then use vim's folding methods to navigate the file (http://vim.wikia.com/wiki/Folding http://vim.wikia.com/wiki/Folding)
- burkemw3 10y agoI use json.tool a lot too, but usually pipe it to `less`, using the paging and searching.
- ferrari8608 10y agoIt's not nice to abuse innocent cats. python -mjson.tool < unformatted.json | vim - If you're not using cat to concatenate things, the shell can probably handle the job.
- HappyTypist 10y agoPeople like cat because it allows notating the command closer to how they think. How can I pipe while putting the file first?
- yoha 10y agoWell, just do it: < unformatted.json python -mjson.tool | vim -
- thiht 10y agoWhat's the immediate advantage for me in using a less intuitive syntax? I understand cat is meant to concatenate files, so what?
- niccaluim 10y agoI view them unfavorably.
- gravypod 10y agoI personally think that's one of the more sane replies. It is really important to chop up your JSON files into smaller sub-files. This will not only make it easier to backup and read manually but will usually give you a speed boost (can read to and write to more then 1 part of the "db" at a time).
- TazeTSchnitzel 10y agoThis is probably sensible. JSON is a text-based format that requires you to parse the entire thing just to get an outline, unlike well-designed binary formats.
- misframer 10y agoWell-designed text formats too?
- mockery 10y agoNot necessarily. (I'd actually argue "not at all.") Presumably you have a text format in the first place because you want your representation to be human-readable [with common tools like a text editor], and very likely human-writable as well. Those are the real constraints of a (useful) text format, and they tend to be in direct conflict with high-performance parsing, or partial parsing. For an arbitrary example, a binary format could have an index table of objects at the start of the file, and then you could perform partial reads to access only the subset of objects you care about. That's something you could do in a text format too, but if the file is edited in a text editor you can't guarantee that the user remembered or bothered to update the index when they added a new object. The parser would effectively not be able to trust the index, and have to parse the entire file. (I suppose you could use CRCs or something to enforce this, but then you'd end up with a very brittle format that people get frustrated when trying to edit.) Really, the true advantage of a binary format is you generally assume that nobody messed with the data behind your back, so you can have duplicate data (like an index) if you want without worrying that it's out of sync. This pretty much goes hand in hand with the fact that you can't just open it in a text editor and fiddle with stuff. TLDR: Human-writability and high-performance are arguably mutually exclusive features.
- baseh 10y agoFor extremely large CSVs I use Emeditor (windows / paid). It also works with JSON, but never had to handle large JSONs
- beeper87 10y agoNot Atom. ;) [1] https://github.com/atom/atom/issues/8864 https://github.com/atom/atom/issues/8864 [2] https://github.com/atom/atom/issues/979 https://github.com/atom/atom/issues/979
- deleted 10y ago[deleted]
- yagop 10y agovim
- EN1 10y agoThis is something I use quite frequently when dealing with JSON. https://www.getpostman.com https://www.getpostman.com
- meeper16 10y agovi (5 gigs)
- tonyle 10y agoChrome console allows you to inspect an object and expand on the properties visually. For the most common cases, I just copy and paste the string into the chrome console if I need to inspect some random JSON file really quick. If it is too big to copy and paste, pipe it into a html file as a global variable and view the object in the console. You can also just open a node repl and load the JSON into memory. node lets you auto complete property names by pressing tab, this is great for inspecting some unknown object.
- mani04 10y agoI used to be a fan of JSONView or one of those chrome plugins / extensions. Then I realized that I can simply use the network tab to view the JSON data as it gets loaded. Here is how it works - keep the network tab ready. When you see the JSON data request, click on the request and hit the "Preview" tab. It gives you data in a collapse / expand format. Advantages: 1. There is one less plugin that scans all your browsing activity, 2. Slightly extra battery life when just browsing and not developing stuff. Disadvantage: You need to keep the network tab ready, otherwise you will have to reload the entire page with the network tab open. EDIT: My apologies if it wasn't clear. I was talking about the "Developer Tools" option in Chrome, in which there is a "Network" tab. It is available in "Chrome Menu" > "More Tools" > "Developer Tools". Alternatively you can hit Command + Option + I in mac, or some equivalent in Linux / Windows to get there.
- cyphax 10y agoUsually F12 opens these tools in most browsers (IE, Chrome, Firefox at least) on Linux and Windows, but I do not have a Mac to test Os X. Your tip also applies to Firefox, but I don't know about IE or Edge. :) I've never really had to handle really large JSON files, as I'm not a big fan of those but for smaller files I tend to be lazy and paste it in jsonlint.com. It's usually just for reference or debugging purposes (like finding the name of a property or some strange value).
- ferrari8608 10y ago>but it freezes on files larger than 1 MB That's a pretty large bug. Considering what JSON is supposed to be used for, passing data between two programs/processes, files larger than 1 MB should have been accounted for. Does it freeze then crash, freeze until the OS takes care of it, or freeze temporarily then resume?
- chuchuva 10y agoIt freezes temporarily then resumes after few minutes.
- twblalock 10y agoI use IntelliJ for that. It parses the JSON and adds code folding automatically. I've used it with fairly large files.
- hendzen 10y agojq with less/more as pagers...
- canada_dry 10y agonotebook++ has a great json viewer plugin and handles huge files
- uglycoyote 10y agoI have a question related to this which nobody has really touched on yet. Imagine you have a json structure like this: {"records" : [ {"name":"joe", ...lots of other fields in each record}, {"name":"fred", ...lots of other fields}, ... thousands of records ]} Now a lot of people in this thread have mentioned tools for collapsing the json, but the trouble I have with this is that you can't browse the record names without opening each one and looking at all the fields. It would be nice if there was a tool that took a few key identifiers (name, id) and "bubbled them up" so that in a collapsed view you would see something like: {"records":[ {...click to expand...}, // name=joe {...click to expand...}, // name=fred ]} I have the same problem when working with XML as well (though in XML sometimes the ID's are attributes of the parent which mitigates the problem, but other times the ID would be nested in a child element of the structure more like you would have in json). I even found it to be enough of a problem that I wrote my own XML editor to solve this issue. Of course one issue with this is that there's so clear standard as to which fields of an object represent "ID" information which would be important/useful to "bubble up" to the next level when collapsing. It would have to be something user-configurable (though having some sensible defaults like looking for "name" and "id" keys would work in a lot of cases). In the XML world, there's probably something to do with Schemas that would help with this problem, and fancy editors which understand your data using a schema, though some of the editors I looked at which went into that level of detail seemed like way overkill for what i wanted to do. So essentially my question here is whether this concept of "collapse child, but keep important identifying information of the collapsed child visible" exists in any json tools? is this a thing that has a name/buzzword associated with it that I don't know about? or is it purely an issue that's my own personal quirk which nobody else cares about?
- tonyle 10y agoThat data structure is basically a CSV file. If I were you, I would convert the JSON to a CSV file, load it into excel and use a pivot table dive into your data and convert it back if needed. A pivot table will do exactly what you want and more. If you hate excel, You could load it into a database and do a groupby query. I suspect most JSON in that data structure were auto generated from CSV files or database anyways. Another option is to just convert that JSON to another JSON keyed by name with Lodash then inspect it. var anotherJSON=_.keyBy(oldJSON, 'name');
- alexatkeplar 10y agoA human can't easily understand the structure of a large JSON file - JSON is internally self-describing, so there is no guarantee that the sample of records currently visible in your editor's viewport are representative of the whole. Instead, use a tool like Schema Guru (https://github.com/snowplow/schema-guru https://github.com/snowplow/schema-guru ; disclaimer: we wrote this at Snowplow) to programmatically extract the JSON Schema (http://json-schema.org/ http://json-schema.org/) which represents all JSON instances in the file.
- rhubarbquid 10y agoI'd suggest either use your browser with a good JSON viewing extension or a code editor. > I want to be able to easily collapse and expand elements That feature is often called "code folding", many programming oriented text editors do it.
- ilyash 10y agoI wrote a tool some time ago. It's not a viewer. It just shows the structure. https://github.com/ilyash/show-struct https://github.com/ilyash/show-struct
- nielsbot 10y agoOn the macOS I've had some success with "Cocoa JSON Editor.app". It takes a while to load files, but they are presented in an outline view you can search and browse.
- radoslawc 10y agohttp://kmkeen.com/jshon/ http://kmkeen.com/jshon/ or python -mjson piped to $editor
- TMSZ 10y agoThere are few viewers/editors listed at http://softwarerecs.stackexchange.com/questions/18839/json-viewer-for-windows http://softwarerecs.stackexchange.com/questions/18839/json-v... JSONedit should work well with files up to few MBs, with larger files loading time may be not acceptable. Side note: for editors that generate structured view file size may be not precise metric - number of elements might work better.
- 77897555 10y agoWhether you are looking at JSON with a bit of expected structure or otherwise, parsing something that big is not humanely possible in a timely fashion. Throw it into a db to analyse it.
- r24y 10y agoGenerally I'll copy it to the clipboard (usually `pbcopy < file.json`), fire up Chrome, Cmd+Alt+J, then paste into the console. Instant, attractive, collapsible rendering of the content. Usually when I do this I'm interested in performing some sort of transformation on the data, and this way I can prototype it without any further steps.
- jamesmalvi 10y agoI uses http://jsonformatter.org http://jsonformatter.org.