3 ms·

This is missing a course on software testing, particularly one that has a focus on testing software-implemented models. I'm almost done a PhD in mechanical eng

This is missing a course on software testing, particularly one that has a focus on testing software-implemented models. I'm almost done a PhD in mechanical engineering. Some CS folks have lamented gaps in my programming-related education, but I actually did take an elective course on the reliability of science which included a lot on scientific software testing. I think such a class should be required for anyone working in theory or simulations. In engineering, concerns about the testing and accuracy of a model are typically called "verification and validation". There unfortunately doesn't seem to be a standard curriculum for the subject yet but Wikipedia can give you an idea of what this covers and how it's different from general software testing: https://en.wikipedia.org/wiki/Verification_and_validation_of_computer_simulation_models https://en.wikipedia.org/wiki/Verification_and_validation_of... AI/ML seems to have their own culture around this and I'm not sure its actually rigorous. Seems to me that they could have their own version of the reproducibility crisis. This course should be taken after a statistics course. Parts of it rely fairly heavily on statistics.

chriskanan 6y ago
AI is having some reproducibility issues, especially reinforcement learning: https://www.wired.com/story/artificial-intelligence-confronts-reproducibility-crisis/ https://www.wired.com/story/artificial-intelligence-confront... In other areas of machine learning, things have gotten a lot better. A decade ago people rarely released their code or trained models. I did this for an early feature learning paper in 2010, which led to it getting a lot of citations. What I'm more worried about is the lack of science in machine learning papers. The code reproduces the result, but the reasons for the efficacy given in the paper are spurious. I like this recent paper that pokes this issue in metric learning: https://arxiv.org/abs/2003.08505 https://arxiv.org/abs/2003.08505
- btrettel 6y ago
  Thanks for the links. I would consider the science issue you mention to be related. A lot of what was covered in the "verification, validation, and uncertainty quantification" course I took was basic science. The class wasn't only about software testing. Software testing was just a major tool covered in the class for the larger goal of making reliable scientific claims. What I wrote previously wasn't clear on this, so I clarified my previous comment on this point.
NalNezumi 6y ago
Baking in a software testing in an already tight 4 year AI/ML focused undergrad seems misplaced. Maybe in a masters, but most masters have Scientific methodology baked in to them so I'd still say it is redundant. CS undergrad should definitely do some level of testing though.
- btrettel 6y ago
  I think not making the 4th year primarily projects would help. In my mechanical engineering BS my 4th year was a combination of projects and electives (ignoring the senior design class). At the very least a class on software testing should be offered at the level of an elective. The curriculum doesn't mention other required courses like general education, so perhaps the author thought electives weren't worth mentioning either.
  - Junk_Collector 6y ago
    This recommendation only specifies 9 credit hours (at most) a semester which free study in the 4th year. I know that you have to build in room for your cross discipline and electives, but it's not a very tight schedule. Which is good because it's also missing Differential Equations, Set Theory, Fourier Analysis and Stochastic Calculus all of which are mathematical underpinnings of AI. I also agree that they need a class on Software testing or validation or something along those lines. A crash course in Filter theory, so that people have some historical background of where these methods derived from would be useful as well.

chriskanan 6y ago

AI is having some reproducibility issues, especially reinforcement learning: https://www.wired.com/story/artificial-intelligence-confronts-reproducibility-crisis/ https://www.wired.com/story/artificial-intelligence-confront... In other areas of machine learning, things have gotten a lot better. A decade ago people rarely released their code or trained models. I did this for an early feature learning paper in 2010, which led to it getting a lot of citations. What I'm more worried about is the lack of science in machine learning papers. The code reproduces the result, but the reasons for the efficacy given in the paper are spurious. I like this recent paper that pokes this issue in metric learning: https://arxiv.org/abs/2003.08505 https://arxiv.org/abs/2003.08505

btrettel 6y ago
Thanks for the links. I would consider the science issue you mention to be related. A lot of what was covered in the "verification, validation, and uncertainty quantification" course I took was basic science. The class wasn't only about software testing. Software testing was just a major tool covered in the class for the larger goal of making reliable scientific claims. What I wrote previously wasn't clear on this, so I clarified my previous comment on this point.

NalNezumi 6y ago

Baking in a software testing in an already tight 4 year AI/ML focused undergrad seems misplaced. Maybe in a masters, but most masters have Scientific methodology baked in to them so I'd still say it is redundant. CS undergrad should definitely do some level of testing though.

btrettel 6y ago
I think not making the 4th year primarily projects would help. In my mechanical engineering BS my 4th year was a combination of projects and electives (ignoring the senior design class). At the very least a class on software testing should be offered at the level of an elective. The curriculum doesn't mention other required courses like general education, so perhaps the author thought electives weren't worth mentioning either.
- Junk_Collector 6y ago
  This recommendation only specifies 9 credit hours (at most) a semester which free study in the 4th year. I know that you have to build in room for your cross discipline and electives, but it's not a very tight schedule. Which is good because it's also missing Differential Equations, Set Theory, Fourier Analysis and Stochastic Calculus all of which are mathematical underpinnings of AI. I also agree that they need a class on Software testing or validation or something along those lines. A crash course in Filter theory, so that people have some historical background of where these methods derived from would be useful as well.