3 ms·

Parameters are probably the lowest hanging fruit in any machine learning optimisation, being a trade secret is a big stretch. Also use LBFG so you never worry

• deepnotderp 9y ago
  "Also use LBFG so you never worry about learning rate." Literally never happens in practice.
• eachro 9y ago
  Hyperparameter optimization for neural networks can be insanely expensive though[1], to the point where there are even startups (e.g: SigOpt) that do this. [1] https://twitter.com/beenwrekt/status/961263599674150912 https://twitter.com/beenwrekt/status/961263599674150912
  • kirillseva 9y ago
    the tweet you link to is architectural search, or figuring out what the neural network should look like (along with its parameters). If self-driving car ML is "data scientists replacing a driver with computers", architecture search is computers replacing data scientists. $1.2M is roughly 3-4 data scientists' annual compensations at a company like Google or Uber. Finding hyperparameters for an existing architecture is much easier because you would typically deal with a much lower cardinality in hyperparameter space. SigOpt is helping with that. Google has an internal solution that is able to take advantage of parameters within their job scheduling infrastructure (they actually have a bunch of them, the one I'm talking about is called hyperband [0]). CERN, as far as I know, was using hyperopt [1]. Avant uses an open-source version of SigOpt that we wrote in 2016 called loop [2]. [0] - https://arxiv.org/pdf/1603.06560.pdf https://arxiv.org/pdf/1603.06560.pdf [1] - http://hyperopt.github.io/hyperopt http://hyperopt.github.io/hyperopt [2] - https://github.com/avantoss/loop https://github.com/avantoss/loop
    • pc86 9y ago
      Before this gets inundated with people making comments about Googler data scientists making $400k, these are folks with PhDs in math or statistics, very accomplished academic literature, leaders in their fields. These are not people messing around with Haskell on the weekends and leaving an insurance firm's IT department to go to Google.
      • snotrockets 9y ago
        They also don't make $400k – an engineer that costs $400k doesn't get that in take home pay.
        • jsjohnst 9y ago
          Entirely not true.
        • pc86 9y ago
          There are enough non-executive engineers at Google making half a million or more base/cash comp that you could fill a room with them. That's not the point.
    • eachro 9y ago
      Yea, and my point was that this can be expensive regardless of the semantics of whether or not architectural degrees of freedom are hyperparameters.
• make3 9y ago
  no one uses LBFG in deep learning or deep reinforcement learning, which is what self driving cars use