5 ms·

Do you not think that using spreadsheets for ML experiment tracking is a symptom of broken tooling? I'm asking because one of the reasons we're building our pla

Do you not think that using spreadsheets for ML experiment tracking is a symptom of broken tooling? I'm asking because one of the reasons we're building our platform[0] with automatic experiment tracking, collaborative notebooks, and a bunch of things, is because experiment tracking was inconsistent between team members. Differences in tools used: (spreadsheets, flat files, logs, pen and paper, human memory). Forgetting to do it. Snippets to do it flying around. Different locations (laptop, group workstation, git repository, cloud sheet). Dissociated from the notebook that produced the model. Tighter tracking should answer questions like: what notebook ran on which data and produced which model with which parameters and which scores? Then questions like: give me all notebooks that ran on this dataset which produced a model with scores that are [condition]. Once you do that, the "spreadsheet" can just be a "view" of the underlying data. Something you can export as, but not the thing itself. I think it's good there are tools with this granularity that can be composed. - [0]: https://iko.ai https://iko.ai

breck 6y ago
> Once you do that, the "spreadsheet" can just be a "view" of the underlying data. Something you can export as, but not the thing itself. Yes, and sorry because rereading my comment I wasn't clear, but this is what I meant. I view Spreadsheets as the view (but also an editor of the view), but I don't mean people should be working with XLS files (TSVs seem to work the best— I personally hate tabs but may have lost that battle). I do think though of Spreadsheets as the primary view, and so always design my data structures with the understanding of "how will this interoperate with spreadsheets". JSON is the pits. IMO 2-D DSLs with Spreadsheets as the primary view/editor paradigm are the future. I think Spreadsheets are 1, if not 2 OOM better than notebooks for doing actual work (notebooks can be good for presenting results in a narrative): - non-linear for both humans and machines (allows for really creative and fast out-of-order parsing techniques on the machine side) - concise signal with high information density - unlimited cursors - fantastically easier for version control and multi-player experiences I maintain a list of all data science tools to try and stay on top of best practices, and when I see a tool is notebook first, I think "good, less work for me to track this one because they aren't getting the core things right yet." (Though often times notebooks will have really innovative orthogonal features).
- Jugurtha 6y ago
  I think we may be talking about different things. My first reply was based on my understanding you saved results in spreadsheets. breck wrote: >I would always save my experiment results so they were ready to analyze in spreadsheets (and other vis tools). I should have elicited further before making assumptions. My assumption was that you were referring to results of training machine learning models. Is this correct? If my assumption is correct, how do you go about training machine learning models? Also, one question concerning your "scare" of "Throw away your spreadsheets"... Do you mean that you'd like a tool that exports results to spreadsheets or something in that direction? I think we are addressing different problems, and a large part of that is due to assumptions I have made earlier.
  - breck 6y ago
    > My assumption was that you were referring to results of training machine learning models. Is this correct? I would set my hyper params in a spreadsheet, which would kick off training runs on a cluster, and report the results back in the spreadsheet hours/days later(really TSVs, but my UI was a spreadsheet). And repeat. This was primitive stuff, and haven't done DL in a while, so not even sure if hyperparam tuning is still a thing or if that is all automated now.
    - Jugurtha 6y ago
      How did you do that? Were you using some kind of hooks or polling for the spreadsheet state and triggering training jobs whenever it had changed? Also, how did you deal with changing which hyperparameters you used or which algorithms you used? Did you make a spreadsheet per project per model?
      - breck 6y ago
        I forget the exact details, but my guess is I made my own DSLs and then "executed" them. Something like: https://jtree.treenotation.org/designer/#grammar%0A%20inferredLanguageNode%0A%20%20root%0A%20%20inScope%20jobNode%20resultsNode%0A%20keywordCell%0A%20anyCell%0A%20intCell%0A%20jobNode%0A%20%20inScope%20nicknameNode%20someIdeaNode%20otherIdeaNode%20_46_46_46Node%0A%20%20crux%20job%0A%20%20cells%20keywordCell%0A%20nicknameNode%0A%20%20crux%20nickname%0A%20%20cells%20keywordCell%20anyCell%20anyCell%20anyCell%20anyCell%0A%20someIdeaNode%0A%20%20crux%20someIdea%0A%20%20cells%20keywordCell%20intCell%20anyCell%20intCell%0A%20otherIdeaNode%0A%20%20crux%20otherIdea%0A%20%20cells%20keywordCell%20intCell%20anyCell%20intCell%0A%20_46_46_46Node%0A%20%20crux%20...%0A%20%20cells%20keywordCell%0A%20resultsNode%0A%20%20inScope%20scoreNode%20_4681Node%20_4682Node%20_46_46_46Node%0A%20%20crux%20results%0A%20%20cells%20keywordCell%0A%20scoreNode%0A%20%20crux%20score%0A%20%20cells%20keywordCell%20anyCell%20anyCell%20anyCell%20anyCell%20anyCell%20anyCell%20anyCell%20anyCell%0A%20_4681Node%0A%20%20crux%20.81%0A%20%20cells%20keywordCell%20anyCell%20anyCell%20anyCell%20anyCell%20intCell%20anyCell%20intCell%0A%20_4682Node%0A%20%20crux%20.82%0A%20%20cells%20keywordCell%20anyCell%20anyCell%20anyCell%20anyCell%20intCell%20anyCell%20intCell%0Asample%0A%20job%0A%20%20nickname%20batchSize%20cluster%20seed%20...%0A%20%20someIdea%2030%20rc1%202%0A%20%20otherIdea%2030%20rc1%202%0A%20%20...%0A%20results%0A%20%20score%20started%20finished%20elapsed%20nickname%20batchSize%20cluster%20seed%20...%0A%20%20.81%202-22-2018.17%3A32%202-22-2018.20%3A32%203hr%20someIdea%2030%20rc1%202%0A%20%20.82%202-22-2018.17%3A33%202-22-2018.20%3A33%203hr%20otherIdea%2030%20rc1%202%0A%20%20 https://jtree.treenotation.org/designer/#grammar%0A%20inferr......

breck 6y ago

> Once you do that, the "spreadsheet" can just be a "view" of the underlying data. Something you can export as, but not the thing itself. Yes, and sorry because rereading my comment I wasn't clear, but this is what I meant. I view Spreadsheets as the view (but also an editor of the view), but I don't mean people should be working with XLS files (TSVs seem to work the best— I personally hate tabs but may have lost that battle). I do think though of Spreadsheets as the primary view, and so always design my data structures with the understanding of "how will this interoperate with spreadsheets". JSON is the pits. IMO 2-D DSLs with Spreadsheets as the primary view/editor paradigm are the future. I think Spreadsheets are 1, if not 2 OOM better than notebooks for doing actual work (notebooks can be good for presenting results in a narrative): - non-linear for both humans and machines (allows for really creative and fast out-of-order parsing techniques on the machine side) - concise signal with high information density - unlimited cursors - fantastically easier for version control and multi-player experiences I maintain a list of all data science tools to try and stay on top of best practices, and when I see a tool is notebook first, I think "good, less work for me to track this one because they aren't getting the core things right yet." (Though often times notebooks will have really innovative orthogonal features).

Jugurtha 6y ago
I think we may be talking about different things. My first reply was based on my understanding you saved results in spreadsheets. breck wrote: >I would always save my experiment results so they were ready to analyze in spreadsheets (and other vis tools). I should have elicited further before making assumptions. My assumption was that you were referring to results of training machine learning models. Is this correct? If my assumption is correct, how do you go about training machine learning models? Also, one question concerning your "scare" of "Throw away your spreadsheets"... Do you mean that you'd like a tool that exports results to spreadsheets or something in that direction? I think we are addressing different problems, and a large part of that is due to assumptions I have made earlier.
- breck 6y ago
  > My assumption was that you were referring to results of training machine learning models. Is this correct? I would set my hyper params in a spreadsheet, which would kick off training runs on a cluster, and report the results back in the spreadsheet hours/days later(really TSVs, but my UI was a spreadsheet). And repeat. This was primitive stuff, and haven't done DL in a while, so not even sure if hyperparam tuning is still a thing or if that is all automated now.
  - Jugurtha 6y ago
    How did you do that? Were you using some kind of hooks or polling for the spreadsheet state and triggering training jobs whenever it had changed? Also, how did you deal with changing which hyperparameters you used or which algorithms you used? Did you make a spreadsheet per project per model?
    - breck 6y ago
      I forget the exact details, but my guess is I made my own DSLs and then "executed" them. Something like: https://jtree.treenotation.org/designer/#grammar%0A%20inferredLanguageNode%0A%20%20root%0A%20%20inScope%20jobNode%20resultsNode%0A%20keywordCell%0A%20anyCell%0A%20intCell%0A%20jobNode%0A%20%20inScope%20nicknameNode%20someIdeaNode%20otherIdeaNode%20_46_46_46Node%0A%20%20crux%20job%0A%20%20cells%20keywordCell%0A%20nicknameNode%0A%20%20crux%20nickname%0A%20%20cells%20keywordCell%20anyCell%20anyCell%20anyCell%20anyCell%0A%20someIdeaNode%0A%20%20crux%20someIdea%0A%20%20cells%20keywordCell%20intCell%20anyCell%20intCell%0A%20otherIdeaNode%0A%20%20crux%20otherIdea%0A%20%20cells%20keywordCell%20intCell%20anyCell%20intCell%0A%20_46_46_46Node%0A%20%20crux%20...%0A%20%20cells%20keywordCell%0A%20resultsNode%0A%20%20inScope%20scoreNode%20_4681Node%20_4682Node%20_46_46_46Node%0A%20%20crux%20results%0A%20%20cells%20keywordCell%0A%20scoreNode%0A%20%20crux%20score%0A%20%20cells%20keywordCell%20anyCell%20anyCell%20anyCell%20anyCell%20anyCell%20anyCell%20anyCell%20anyCell%0A%20_4681Node%0A%20%20crux%20.81%0A%20%20cells%20keywordCell%20anyCell%20anyCell%20anyCell%20anyCell%20intCell%20anyCell%20intCell%0A%20_4682Node%0A%20%20crux%20.82%0A%20%20cells%20keywordCell%20anyCell%20anyCell%20anyCell%20anyCell%20intCell%20anyCell%20intCell%0Asample%0A%20job%0A%20%20nickname%20batchSize%20cluster%20seed%20...%0A%20%20someIdea%2030%20rc1%202%0A%20%20otherIdea%2030%20rc1%202%0A%20%20...%0A%20results%0A%20%20score%20started%20finished%20elapsed%20nickname%20batchSize%20cluster%20seed%20...%0A%20%20.81%202-22-2018.17%3A32%202-22-2018.20%3A32%203hr%20someIdea%2030%20rc1%202%0A%20%20.82%202-22-2018.17%3A33%202-22-2018.20%3A33%203hr%20otherIdea%2030%20rc1%202%0A%20%20 https://jtree.treenotation.org/designer/#grammar%0A%20inferr......