6 ms·

Nice, some initial remarks: - Sad to see it's an Electron app, but I don't want to start this flamewar all over again - Mid-sized CSV file opens fairly quick

Nice, some initial remarks: - Sad to see it's an Electron app, but I don't want to start this flamewar all over again - Mid-sized CSV file opens fairly quick - If you want to build this into a full-fledged app, be prepared to handle a lot of CSV edge cases, also think about supporting AVRO, Parquet files - Regarding filtering: I can filter on CONTAINS and '=' but not on NOT-CONTAINS or '<>' I think? This is annoying to filter out empty strings (didn't check in much detail so I might be wrong here) - Regarding filtering: I have a lot of columns, would be nice if the dropdown would allow me typing some characters to prune the list and quickly find what I'm looking for - You might think about including some statistics per column, e.g. variance or entropy to allow for exploration in case many columns are present and you quickly want to highlight the more "interesting" ones Congrats on putting something out! Some competition in this space: https://exploratory.io/ https://exploratory.io/, https://github.com/saulpw/visidata https://github.com/saulpw/visidata and http://www.delimitware.com/ http://www.delimitware.com/ (as others have mentioned).

mrmondo 9y ago
Good summary, thanks for the tldr; +1 for the comment about Electronic/Javascript, also don't want to start a flame war heh. exploratory.io looks really neat, it's a shame about the pricing and I'm not quite sure if it's a native app once you purchase or if it's an in-browser online-only app, because if it's online only 1) I'd be worried about uploading data to a 3rd party and 2) The internet in Australia is _terrible_ so uploading / downloading larger data sets might be painful.
- Pamar 9y ago
  From their FAQ (https://exploratory.io/faq https://exploratory.io/faq) - I have very sensitive data. Is my data safe? Any data you import into Exploratory Desktop always stays on your PC and never leave your PC unless you explicitly publish (share) it to Exploratory Cloud (exploratory.io). If you decided to publish the data to Exploratory Cloud for sharing or scheduling, you can share it in a private way so that so that only you and others you have invited can view it. The data is also stored in encrypted. Please take a look at our Privacy Policy for more details. - Where exactly my data is stored after importing? All the data you import into Exploratory Desktop is saved as a binary form (R's Rdata format) inside your repository, which is located under '/.exploratory' on your PC.
tconley 9y ago
exploratory.io looking promising. Thanks for the links and since it's an electron app then there's hope for a web version :)
baldfat 9y ago
> Sad to see it's an Electron app, but I don't want to start this flamewar all over again Then stop mentioning it. Just don't use anything electron and don't mention it. I will use my RStudio and VS Code.
- mhuffman 9y ago
  I think it is ok to express displeasure at the state of something without just silently doing something else and hoping that the entire ecosystem catches up to you. For example, I take every opportunity (like right now, for example) to shit on node.js and npm for that very reason. There is so much evangelism on one side, it is nice to hear the opposing view points.
  - sillysaurus3 9y ago
    Ok, so what are some legit criticisms of node.js and npm? I'm no fanboy, but every time I try to ask this, the answers are along the lines of "if you install packages in a different order, you get different results." Which is fair, but everybody installs using `npm i`, so that's mostly a moot point. Is there any substance here? I think it's fair to talk about this, as long as we go out of our way not to reduce it to flamewar territory. The app is built on Electron, after all. Maybe avoid mentioning memory usage, since that point has been done to death.
    - Jach 9y ago
      What counts as a legit criticism? So much of our field is taste-driven. And so much criticism was aimed at old iterations of something, gets carried on in flamewars, but no longer applies to the latest version (e.g. "Java is slow"). All the JS-the-language criticisms apply, those are probably the biggest at this point. The classic "node.js is cancer" is still somewhat relevant though I never thought it made its point well enough on how forced-asynchronous is at least as bad as the forced OOP and checked exceptions we put up with in e.g. Java again. Callback hell was long a common complaint, until people found Promises / the waterfall structure, but then the complaint is debugging hell. The new await stuff in node 8 sounds nice at least. For a long time npm didn't get stuff over https by default, I think it does for a year now? The only other criticisms I can think of are just shoddy engineering that turns into drama (I remember seeing something about their colorized output would slow everything down quite a lot) or more taste-level things like how seemingly trivial concepts like leftpad are core libraries (that go through multiple versions because it couldn't have just been done right the first time) and the drama involved when something everyone depends on has an issue. Disclaimer, I haven't done production node since ~2013. It was actually mostly enjoyable, and I wouldn't mind doing it again, despite on the taste-level I think the whole node ecosystem is just worse than many other options (when you have a choice) for many specific problems. But life is a continuous lesson in Worse is Better.
dgudkov 9y ago
I believe my http://easymorph.com http://easymorph.com would also count as competition. It loads CSV files and queries databases, filters, pivots/unpivots, cleanses, aggregates, merges. Free edition. Windows only (yeah, Mac owners, I know).
- feiss 9y ago
  Looks fantastic. I've been complaining for years that there wasn't any program like this, and even started many times working on it. Thumbs up!
  - dgudkov 9y ago
    Thank you!
_acme 9y ago
Why is it sad that it's an Electron app? Can you articulate a reason why that's an automatic negative? Would you rather software, such as this, not exist than be Electron-based?
- jpttsn 9y ago
  Not that we'd want to start a flame war or anything.
antonycourtney 9y ago
All great feedback, thanks! Some quick replies: - The omission of '<>' from the operator list was just an embarrassing mistake on my part. #facepalm #WILLFIX - Also agree that a drop-down list of checkboxes (ideally a searchable one!) with an 'in' operator for low cardinality columns would be great. But the UI for that is a bit involved, and probably also need to provide some user control over when to do this since gathering distinct values can be an expensive operation for large data sets. - Totally agree that it would be useful to provide more interesting column stats. Thanks for the suggestions and refs to other tools -- very helpful.
- leandot 9y ago
  Good job, curious about the performance when doing multiple AND, OR clauses on 1m+ rows? You'd need compound indices or?
  - antonycourtney 9y ago
    Absolutely all the analysis (including filtering) works by generating SQL queries and that SQLite evaluates. Tad doesn't generate any table indices, and I'd be reticent about adding that. However you should be able to Tad as a sqlite viewer from command line (sqlite://foo/bar.sqlite/tablename) and it should make use of indices on your existing SQLite table if you have them.
- dracodoc 9y ago
  You can check RStudio's data viewer for reference https://support.rstudio.com/hc/en-us/articles/205175388-Using-the-Data-Viewer https://support.rstudio.com/hc/en-us/articles/205175388-Usin... It's a modified dataTable(so search is builtin. I didn't see search in Tad yet), with selectize for categorical variable filters(which is a searchable dropdown list) Another good to have feature is to show column index. We often need to manipulate the columns in code, a column index is helpful. Based on column index, you can also select a subset of columns faster with numeric input -- over 100 columns are normal, using checkbox to select is too cumbersome.
leandot 9y ago
Genuinely curious, what can be used instead of Electron for cross-platform desktop app?
- wereHamster 9y ago
  Qt is an option. Supported on Windows, Mac, and Linux.
- stuffedBelly 9y ago
  A bit old-school, but many use JavaFX, especially for enterprise apps, even if people on HN may bash on Java for whatever reason.
  - invalidname 9y ago
    "Many" is untrue. I'm a huge Java fan but FX never picked: https://www.codenameone.com/blog/should-oracle-spring-clean-javafx.html https://www.codenameone.com/blog/should-oracle-spring-clean-... Enterprises still use Swing and some have touched FX as Swing is in maintenance mode and they don't have a choice.
- ForHackernews 9y ago
  Tk is crossplatform, free and open source: https://en.wikipedia.org/wiki/Tk_(software) https://en.wikipedia.org/wiki/Tk_(software) It might look a bit dated on some platforms, but it will be lean and fast, unlike Electron.
- shakna 9y ago
  Tk - It can look a bit dated, but it's easy to use, comes bundled with Python or Tcl, and bindings in most popular languages. Can run on Windows, Linux, macOS. Qt - The IDE is great. It doesn't look native. Either dynamically linked, or commercially licensed. Runs on Windows, Mac OS X, Linux, Android, iOS, and a range of embedded hardware. wxWidgets - Native backends, so it always looks like it belongs. Bindings in a ton of languages. Can be both simple or complex, depending on what you need it to do. Runs on Windows, Mac OS X, Linux, and in-progress for Android and iOS. JavaFX - Java's replacement for Spring. Runs anywhere Java does. Fairly flexible, and easy to use. Kivy - A Python framework. Mainly aimed at touch-compatibility. Runs on Windows, Linux, macOS, iOS, and Android. LCL - A Lazarus framework. (Think Pascal). Really easy to use, with great drag'n'drop and the like in the IDE. Runs on Windows, macOS and Linux. nuklear - Fairly easy to use, with bindings in a lot of languages. Sometimes requires a bit more work getting it to run on some platforms. This is not all, but the ones I find easy to use (as easy or easier than Electron), and easy to set up and deploy.

mrmondo 9y ago

Good summary, thanks for the tldr; +1 for the comment about Electronic/Javascript, also don't want to start a flame war heh. exploratory.io looks really neat, it's a shame about the pricing and I'm not quite sure if it's a native app once you purchase or if it's an in-browser online-only app, because if it's online only 1) I'd be worried about uploading data to a 3rd party and 2) The internet in Australia is _terrible_ so uploading / downloading larger data sets might be painful.

Pamar 9y ago
From their FAQ (https://exploratory.io/faq https://exploratory.io/faq) - I have very sensitive data. Is my data safe? Any data you import into Exploratory Desktop always stays on your PC and never leave your PC unless you explicitly publish (share) it to Exploratory Cloud (exploratory.io). If you decided to publish the data to Exploratory Cloud for sharing or scheduling, you can share it in a private way so that so that only you and others you have invited can view it. The data is also stored in encrypted. Please take a look at our Privacy Policy for more details. - Where exactly my data is stored after importing? All the data you import into Exploratory Desktop is saved as a binary form (R's Rdata format) inside your repository, which is located under '/.exploratory' on your PC.

tconley 9y ago

exploratory.io looking promising. Thanks for the links and since it's an electron app then there's hope for a web version :)

baldfat 9y ago

> Sad to see it's an Electron app, but I don't want to start this flamewar all over again Then stop mentioning it. Just don't use anything electron and don't mention it. I will use my RStudio and VS Code.

mhuffman 9y ago
I think it is ok to express displeasure at the state of something without just silently doing something else and hoping that the entire ecosystem catches up to you. For example, I take every opportunity (like right now, for example) to shit on node.js and npm for that very reason. There is so much evangelism on one side, it is nice to hear the opposing view points.
- sillysaurus3 9y ago
  Ok, so what are some legit criticisms of node.js and npm? I'm no fanboy, but every time I try to ask this, the answers are along the lines of "if you install packages in a different order, you get different results." Which is fair, but everybody installs using `npm i`, so that's mostly a moot point. Is there any substance here? I think it's fair to talk about this, as long as we go out of our way not to reduce it to flamewar territory. The app is built on Electron, after all. Maybe avoid mentioning memory usage, since that point has been done to death.
  - Jach 9y ago
    What counts as a legit criticism? So much of our field is taste-driven. And so much criticism was aimed at old iterations of something, gets carried on in flamewars, but no longer applies to the latest version (e.g. "Java is slow"). All the JS-the-language criticisms apply, those are probably the biggest at this point. The classic "node.js is cancer" is still somewhat relevant though I never thought it made its point well enough on how forced-asynchronous is at least as bad as the forced OOP and checked exceptions we put up with in e.g. Java again. Callback hell was long a common complaint, until people found Promises / the waterfall structure, but then the complaint is debugging hell. The new await stuff in node 8 sounds nice at least. For a long time npm didn't get stuff over https by default, I think it does for a year now? The only other criticisms I can think of are just shoddy engineering that turns into drama (I remember seeing something about their colorized output would slow everything down quite a lot) or more taste-level things like how seemingly trivial concepts like leftpad are core libraries (that go through multiple versions because it couldn't have just been done right the first time) and the drama involved when something everyone depends on has an issue. Disclaimer, I haven't done production node since ~2013. It was actually mostly enjoyable, and I wouldn't mind doing it again, despite on the taste-level I think the whole node ecosystem is just worse than many other options (when you have a choice) for many specific problems. But life is a continuous lesson in Worse is Better.

dgudkov 9y ago

I believe my http://easymorph.com http://easymorph.com would also count as competition. It loads CSV files and queries databases, filters, pivots/unpivots, cleanses, aggregates, merges. Free edition. Windows only (yeah, Mac owners, I know).

feiss 9y ago
Looks fantastic. I've been complaining for years that there wasn't any program like this, and even started many times working on it. Thumbs up!
- dgudkov 9y ago
  Thank you!

_acme 9y ago

Why is it sad that it's an Electron app? Can you articulate a reason why that's an automatic negative? Would you rather software, such as this, not exist than be Electron-based?

jpttsn 9y ago
Not that we'd want to start a flame war or anything.

antonycourtney 9y ago

All great feedback, thanks! Some quick replies: - The omission of '<>' from the operator list was just an embarrassing mistake on my part. #facepalm #WILLFIX - Also agree that a drop-down list of checkboxes (ideally a searchable one!) with an 'in' operator for low cardinality columns would be great. But the UI for that is a bit involved, and probably also need to provide some user control over when to do this since gathering distinct values can be an expensive operation for large data sets. - Totally agree that it would be useful to provide more interesting column stats. Thanks for the suggestions and refs to other tools -- very helpful.

leandot 9y ago
Good job, curious about the performance when doing multiple AND, OR clauses on 1m+ rows? You'd need compound indices or?
- antonycourtney 9y ago
  Absolutely all the analysis (including filtering) works by generating SQL queries and that SQLite evaluates. Tad doesn't generate any table indices, and I'd be reticent about adding that. However you should be able to Tad as a sqlite viewer from command line (sqlite://foo/bar.sqlite/tablename) and it should make use of indices on your existing SQLite table if you have them.
dracodoc 9y ago
You can check RStudio's data viewer for reference https://support.rstudio.com/hc/en-us/articles/205175388-Using-the-Data-Viewer https://support.rstudio.com/hc/en-us/articles/205175388-Usin... It's a modified dataTable(so search is builtin. I didn't see search in Tad yet), with selectize for categorical variable filters(which is a searchable dropdown list) Another good to have feature is to show column index. We often need to manipulate the columns in code, a column index is helpful. Based on column index, you can also select a subset of columns faster with numeric input -- over 100 columns are normal, using checkbox to select is too cumbersome.

leandot 9y ago

Genuinely curious, what can be used instead of Electron for cross-platform desktop app?

wereHamster 9y ago
Qt is an option. Supported on Windows, Mac, and Linux.
stuffedBelly 9y ago
A bit old-school, but many use JavaFX, especially for enterprise apps, even if people on HN may bash on Java for whatever reason.
- invalidname 9y ago
  "Many" is untrue. I'm a huge Java fan but FX never picked: https://www.codenameone.com/blog/should-oracle-spring-clean-javafx.html https://www.codenameone.com/blog/should-oracle-spring-clean-... Enterprises still use Swing and some have touched FX as Swing is in maintenance mode and they don't have a choice.
ForHackernews 9y ago
Tk is crossplatform, free and open source: https://en.wikipedia.org/wiki/Tk_(software) https://en.wikipedia.org/wiki/Tk_(software) It might look a bit dated on some platforms, but it will be lean and fast, unlike Electron.
shakna 9y ago
Tk - It can look a bit dated, but it's easy to use, comes bundled with Python or Tcl, and bindings in most popular languages. Can run on Windows, Linux, macOS. Qt - The IDE is great. It doesn't look native. Either dynamically linked, or commercially licensed. Runs on Windows, Mac OS X, Linux, Android, iOS, and a range of embedded hardware. wxWidgets - Native backends, so it always looks like it belongs. Bindings in a ton of languages. Can be both simple or complex, depending on what you need it to do. Runs on Windows, Mac OS X, Linux, and in-progress for Android and iOS. JavaFX - Java's replacement for Spring. Runs anywhere Java does. Fairly flexible, and easy to use. Kivy - A Python framework. Mainly aimed at touch-compatibility. Runs on Windows, Linux, macOS, iOS, and Android. LCL - A Lazarus framework. (Think Pascal). Really easy to use, with great drag'n'drop and the like in the IDE. Runs on Windows, macOS and Linux. nuklear - Fairly easy to use, with bindings in a lot of languages. Sometimes requires a bit more work getting it to run on some platforms. This is not all, but the ones I find easy to use (as easy or easier than Electron), and easy to set up and deploy.