3 ms·

Hyperparameter optimization for neural networks can be insanely expensive though[1], to the point where there are even startups (e.g: SigOpt) that do this. [1]

by eachro 9y ago

Hyperparameter optimization for neural networks can be insanely expensive though[1], to the point where there are even startups (e.g: SigOpt) that do this. [1] https://twitter.com/beenwrekt/status/961263599674150912 https://twitter.com/beenwrekt/status/961263599674150912

kirillseva 9y ago
the tweet you link to is architectural search, or figuring out what the neural network should look like (along with its parameters). If self-driving car ML is "data scientists replacing a driver with computers", architecture search is computers replacing data scientists. $1.2M is roughly 3-4 data scientists' annual compensations at a company like Google or Uber. Finding hyperparameters for an existing architecture is much easier because you would typically deal with a much lower cardinality in hyperparameter space. SigOpt is helping with that. Google has an internal solution that is able to take advantage of parameters within their job scheduling infrastructure (they actually have a bunch of them, the one I'm talking about is called hyperband [0]). CERN, as far as I know, was using hyperopt [1]. Avant uses an open-source version of SigOpt that we wrote in 2016 called loop [2]. [0] - https://arxiv.org/pdf/1603.06560.pdf https://arxiv.org/pdf/1603.06560.pdf [1] - http://hyperopt.github.io/hyperopt http://hyperopt.github.io/hyperopt [2] - https://github.com/avantoss/loop https://github.com/avantoss/loop
- pc86 9y ago
  Before this gets inundated with people making comments about Googler data scientists making $400k, these are folks with PhDs in math or statistics, very accomplished academic literature, leaders in their fields. These are not people messing around with Haskell on the weekends and leaving an insurance firm's IT department to go to Google.
  - snotrockets 9y ago
    They also don't make $400k – an engineer that costs $400k doesn't get that in take home pay.
    - jsjohnst 9y ago
      Entirely not true.
    - pc86 9y ago
      There are enough non-executive engineers at Google making half a million or more base/cash comp that you could fill a room with them. That's not the point.
- eachro 9y ago
  Yea, and my point was that this can be expensive regardless of the semantics of whether or not architectural degrees of freedom are hyperparameters.

by eachro 9y ago

kirillseva 9y ago

the tweet you link to is architectural search, or figuring out what the neural network should look like (along with its parameters). If self-driving car ML is "data scientists replacing a driver with computers", architecture search is computers replacing data scientists. $1.2M is roughly 3-4 data scientists' annual compensations at a company like Google or Uber. Finding hyperparameters for an existing architecture is much easier because you would typically deal with a much lower cardinality in hyperparameter space. SigOpt is helping with that. Google has an internal solution that is able to take advantage of parameters within their job scheduling infrastructure (they actually have a bunch of them, the one I'm talking about is called hyperband [0]). CERN, as far as I know, was using hyperopt [1]. Avant uses an open-source version of SigOpt that we wrote in 2016 called loop [2]. [0] - https://arxiv.org/pdf/1603.06560.pdf https://arxiv.org/pdf/1603.06560.pdf [1] - http://hyperopt.github.io/hyperopt http://hyperopt.github.io/hyperopt [2] - https://github.com/avantoss/loop https://github.com/avantoss/loop

pc86 9y ago
Before this gets inundated with people making comments about Googler data scientists making $400k, these are folks with PhDs in math or statistics, very accomplished academic literature, leaders in their fields. These are not people messing around with Haskell on the weekends and leaving an insurance firm's IT department to go to Google.
- snotrockets 9y ago
  They also don't make $400k – an engineer that costs $400k doesn't get that in take home pay.
  - jsjohnst 9y ago
    Entirely not true.
  - pc86 9y ago
    There are enough non-executive engineers at Google making half a million or more base/cash comp that you could fill a room with them. That's not the point.
eachro 9y ago
Yea, and my point was that this can be expensive regardless of the semantics of whether or not architectural degrees of freedom are hyperparameters.