3 ms·

I’m an ML researcher, but did my grad work in systems. Here are several crucially important technologies invented since 1996 not mentioned in the article, which

I’m an ML researcher, but did my grad work in systems. Here are several crucially important technologies invented since 1996 not mentioned in the article, which are fundamental underpinnings of the research community: - Jupyter notebooks, for teaching machine learning. I use these for teaching. - OpenCL and other libraries for running scientific simulations on GPUs, gpflow for training on GPUs - Keras and PyTorch (libraries for simple training of deep learning models). More than half of machine learning research exists on top of these libraries Let’s not even get into the myriad recent discoveries in ML and libraries for them. Parquet file format and similar formats for mass data storage. Spark and Hadoop for massive parallel computation. Hive and Athena further build upon these innovations. A good portion of distributed computing literature is built on these. Eventually consistent databases and NOSQL. There’s so much here, hard to list everything. ElasticSearch and Lucene and other such tools for text search. Then there is all the low level systems research: new file systems like BTRFS and ZFS. wire guard is something that’s just been built that seems foundational. I am running out of words but let’s conclude by saying the premise of this article is laughable

Jugurtha 6y ago
>I’m an ML researcher, but did my grad work in systems. >- Jupyter notebooks, for teaching machine learning. I use these for teaching. - OpenCL and other libraries for running scientific simulations on GPUs, gpflow for training on GPUs - Keras and PyTorch (libraries for simple training of deep learning models). More than half of machine learning research exists on top of these libraries Would you share your usual workflow and the problems you face in that space with these tools?
- SoylentOrange 6y ago
  I wish pip install automatically updated the requirements file like in some other languages. I look up some boilerplate in stack overflow every week for matplotlib figures to look better. CSVs generated in excel cause errors when reading in python without a special incantation. It’s not super easy to set up multi machine training. AWS GPUs are extremely expensive. sklearn models, once saved, are not easy to rework into a different format to be used with other languages. I wish python could do compile-time checking instead of running for 30 minutes to then report a type mismatch. mypy is still not very good and in beta
  - Jugurtha 6y ago
    OK. Asking because we've started building https://iko.ai https://iko.ai as an internal machine learning platform to solve actual problems we have faced delivering machine learning projects for our clients these past few years. So far, it has the following: - No-setup Jupyter environments with the most popular libraries pre-installed - Real-time collaboration on notebooks: see cursors and changes, etc. - Multiple versions of your notebooks - Long-running notebook scheduling with output that survives closed tabs and network disruptions. You can watch the notebook running live even on your phone without opening the JupyterLab interface. - Automatic experiment tracking: automatically detects your models, parameters, and metrics and saves them without you remembering to do so or polluting your notebook with tracking code - Easily deploy your model and get a "REST endpoint" so data scientists don't tap on anyone's shoulder to deploy their model, and developers don't need to worry about ML dependencies to use the models - Build Docker images for your model and push it to a registry to user it wherever you want - Monitor your models' performance on a live dashboard - Publish notebooks as AppBooks: automatically parametrize a notebook to enable clients to interact with it without exporting PDFs or having to build an application or mutate the notebook. This is very useful when you want to expose some parameters that are very domain specific to a domain expert. Much more on our roadmap. We're only focusing on actual problems we have faced serving our clients, and problems we are facing now. We always are on the lookout for problems people who work in the field have. What do you think?
zepto 6y ago
I’m not agreeing with the author, however he accepts ML is an exception, and Jupyter notebooks are the same as mathematics notebooks except that they are open to languages other than wolf. He does cover this. Again, I don’t think you are necessarily wrong, but the author does cover Jupyter and ML.
- SoylentOrange 6y ago
  The author specifically calls ML non-human computing and I believe is talking about the advances in ML from a theoretical standpoint. I don’t believe it is possible to lump the human written software used to train models, generate adversarial examples, and perform other tasks on GPUs, as purely non human computing. In fact, a lot of the advances in computing recently are specifically to enable this type of computing to be more efficient: easily parallelize training, analyze bigger datasets, graph databases, and other “big data” technologies. In any case, software notebooks definitely existed before Jupyter like you say (I’m not familiar with Wolf but Mathematica has a similar idea), but I don’t believe they date back to 1996. I might be mistaken. I don’t see the author mention them in the text.
  - zepto 6y ago
    Mathematica is mentioned and was released in 1988. Also, I don’t see anything particularly new in how ML computing is organized. We’ve had vectorization, and massively parallel processors since the 80s. All that’s new is that those techniques are being used on desktop machines now because of Moore’s law. I’m not saying there are no new discoveries in ML, just that the engineering isn’t new.

Jugurtha 6y ago

>I’m an ML researcher, but did my grad work in systems. >- Jupyter notebooks, for teaching machine learning. I use these for teaching. - OpenCL and other libraries for running scientific simulations on GPUs, gpflow for training on GPUs - Keras and PyTorch (libraries for simple training of deep learning models). More than half of machine learning research exists on top of these libraries Would you share your usual workflow and the problems you face in that space with these tools?

SoylentOrange 6y ago
I wish pip install automatically updated the requirements file like in some other languages. I look up some boilerplate in stack overflow every week for matplotlib figures to look better. CSVs generated in excel cause errors when reading in python without a special incantation. It’s not super easy to set up multi machine training. AWS GPUs are extremely expensive. sklearn models, once saved, are not easy to rework into a different format to be used with other languages. I wish python could do compile-time checking instead of running for 30 minutes to then report a type mismatch. mypy is still not very good and in beta
- Jugurtha 6y ago
  OK. Asking because we've started building https://iko.ai https://iko.ai as an internal machine learning platform to solve actual problems we have faced delivering machine learning projects for our clients these past few years. So far, it has the following: - No-setup Jupyter environments with the most popular libraries pre-installed - Real-time collaboration on notebooks: see cursors and changes, etc. - Multiple versions of your notebooks - Long-running notebook scheduling with output that survives closed tabs and network disruptions. You can watch the notebook running live even on your phone without opening the JupyterLab interface. - Automatic experiment tracking: automatically detects your models, parameters, and metrics and saves them without you remembering to do so or polluting your notebook with tracking code - Easily deploy your model and get a "REST endpoint" so data scientists don't tap on anyone's shoulder to deploy their model, and developers don't need to worry about ML dependencies to use the models - Build Docker images for your model and push it to a registry to user it wherever you want - Monitor your models' performance on a live dashboard - Publish notebooks as AppBooks: automatically parametrize a notebook to enable clients to interact with it without exporting PDFs or having to build an application or mutate the notebook. This is very useful when you want to expose some parameters that are very domain specific to a domain expert. Much more on our roadmap. We're only focusing on actual problems we have faced serving our clients, and problems we are facing now. We always are on the lookout for problems people who work in the field have. What do you think?

zepto 6y ago

I’m not agreeing with the author, however he accepts ML is an exception, and Jupyter notebooks are the same as mathematics notebooks except that they are open to languages other than wolf. He does cover this. Again, I don’t think you are necessarily wrong, but the author does cover Jupyter and ML.

SoylentOrange 6y ago
The author specifically calls ML non-human computing and I believe is talking about the advances in ML from a theoretical standpoint. I don’t believe it is possible to lump the human written software used to train models, generate adversarial examples, and perform other tasks on GPUs, as purely non human computing. In fact, a lot of the advances in computing recently are specifically to enable this type of computing to be more efficient: easily parallelize training, analyze bigger datasets, graph databases, and other “big data” technologies. In any case, software notebooks definitely existed before Jupyter like you say (I’m not familiar with Wolf but Mathematica has a similar idea), but I don’t believe they date back to 1996. I might be mistaken. I don’t see the author mention them in the text.
- zepto 6y ago
  Mathematica is mentioned and was released in 1988. Also, I don’t see anything particularly new in how ML computing is organized. We’ve had vectorization, and massively parallel processors since the 80s. All that’s new is that those techniques are being used on desktop machines now because of Moore’s law. I’m not saying there are no new discoveries in ML, just that the engineering isn’t new.