4 ms·

TensorFlow is the machine learning codebase, but typically how do machine learning research teams manage their training sets, dataset metadata and collaboration

TensorFlow is the machine learning codebase, but typically how do machine learning research teams manage their training sets, dataset metadata and collaboration on these large datasets?

barneso 10y ago
Most teams I have seen have either template scripts or boilerplate that generates datasets, and share both the generated data and the scripts via normal ways that people share data and code: disk, S3, github, emailing of notebooks, etc. It requires a fair amount of set-up, but works surprisingly well once there is a core team and problems established. We are building mldb.ai to help bring the data and the algorithms for ML together in a less ad-hoc manner and to help move things out of research and into prod once they are ready. Many of the hosted ML solutions (Azure ML, Amazon ML, Google Data Lab, etc) and other toolkits (eg Graphlab) are working on similar ML workflow and organizational structure problems.
- sysreader2016 10y ago
  Which projects you know use "disk, S3, github,..." to share their datasets? I'm curious what you think because I haven't read about any ML projects actually using hosted ML solutions like Amazon ML+S3. I've only seen Amazon recommend Amazon ML.
  - HappyTypist 10y ago
    S3 is a good way to share files
cosmoharrigan 10y ago
Here are a couple of examples from the Python deep learning ecosystem: IDSIA, affiliated with Juergen Schmidhuber and many other leading ML researchers, has released Sacred, "a tool to help you configure, organize, log and reproduce experiments." https://github.com/IDSIA/sacred https://github.com/IDSIA/sacred MILA, affiliated with Yoshua Bengio and the Theano project, offers fuel, "a data pipeline framework for machine learning": https://github.com/mila-udem/fuel https://github.com/mila-udem/fuel
sysreader2016 10y ago
I'm working toward moving my deep learning project DeepSentry into e-commerce. DeepSentry was developed in Python3 and C. After any significant amount of time 60+ days the dataset per host is TB-PB. So, I've been digging into implementing a Big Data solution. I've found that Apache Spark on Hadoop-MapR with Postgres is popular.
flatfilefan 10y ago
http://www.openml.org/ http://www.openml.org/

TensorFlow is the machine learning codebase, but typically how do machine learning research teams manage their training sets, dataset metadata and collaboration on these large datasets?

barneso 10y ago

Most teams I have seen have either template scripts or boilerplate that generates datasets, and share both the generated data and the scripts via normal ways that people share data and code: disk, S3, github, emailing of notebooks, etc. It requires a fair amount of set-up, but works surprisingly well once there is a core team and problems established. We are building mldb.ai to help bring the data and the algorithms for ML together in a less ad-hoc manner and to help move things out of research and into prod once they are ready. Many of the hosted ML solutions (Azure ML, Amazon ML, Google Data Lab, etc) and other toolkits (eg Graphlab) are working on similar ML workflow and organizational structure problems.

sysreader2016 10y ago
Which projects you know use "disk, S3, github,..." to share their datasets? I'm curious what you think because I haven't read about any ML projects actually using hosted ML solutions like Amazon ML+S3. I've only seen Amazon recommend Amazon ML.
- HappyTypist 10y ago
  S3 is a good way to share files

cosmoharrigan 10y ago

Here are a couple of examples from the Python deep learning ecosystem: IDSIA, affiliated with Juergen Schmidhuber and many other leading ML researchers, has released Sacred, "a tool to help you configure, organize, log and reproduce experiments." https://github.com/IDSIA/sacred https://github.com/IDSIA/sacred MILA, affiliated with Yoshua Bengio and the Theano project, offers fuel, "a data pipeline framework for machine learning": https://github.com/mila-udem/fuel https://github.com/mila-udem/fuel

sysreader2016 10y ago

I'm working toward moving my deep learning project DeepSentry into e-commerce. DeepSentry was developed in Python3 and C. After any significant amount of time 60+ days the dataset per host is TB-PB. So, I've been digging into implementing a Big Data solution. I've found that Apache Spark on Hadoop-MapR with Postgres is popular.

flatfilefan 10y ago

http://www.openml.org/ http://www.openml.org/