5 ms·

i wonder if the average professional coder, still has. chance to beat the big teams ? Reminded me of years ago(It might have been on HN) where some company/gro

i wonder if the average professional coder, still has. chance to beat the big teams ? Reminded me of years ago(It might have been on HN) where some company/group trained a state of the art nlp model to classify if a financial statment or press release was possitive or negative. They had some good results until someone did: Lets look how far down the press-release-statement the 'numbers' are :) If it was positive-sentiment it usually was the case that the numbers were high up the page. If it was bad thr bulk of the numbers were much lower in the page. Almost make sense,if you had goof numbers you want to screM it out. If you had bad numbers you want to preface/explain why first ??

OhHiMarkos 5y ago
> i wonder if the average professional coder, still has. chance to beat the big teams ? I shared the sentiment but I have began to think otherwise. First of all, we need solutions to big problems. I wouldn't want to tackle Brain tumor problems using DL but someone with the brainpower and resources such as NVIDIA, why not? But besides that, the whole scientific field is pretty young. An average coder or DL researcher isn't useless because Google has all the machines. There are fields where DL isn't used much. Even in the DL field there are problems to solve. Algorithms to optimize.
PartiallyTyped 5y ago
> i wonder if the average professional coder, still has. chance to beat the big teams ? The common sentiment in r/ML is that a) big labs need some accountability because they keep getting credit for work they published but never released source code and therefore third party audit is impossible *. Furthermore, Resnet-50 is back to sota because of up-to-date training protocols, and every paper that claimed superiority was using outdated training techniques (or even none at all) for the baseline. Furthermore, there are many suspicions that the current generation of DL/ML has reached a local minimum because we have collectively pushed for models, architectures, functions, and optimizers that did well on existing hardware. For example, one hypothesis is that the Adam and AdamW optimizers are doing so well on average because we optimized model architectures for them. * Before somebody claims that this is part of the research process and we should be able to validate and implement models on our own, I would like to point out that there have been multiple instances where the math appeared sound, but the implemented models were bugged, and outperformed the existing models *because* they were bugged (NB biased) and the test sets didn't capture that. When corrected, the models didn't perform any better than existing unbiased methods.
- joconde 5y ago
  > Furthermore, Resnet-50 is back to sota because of up-to-date training protocols, and every paper that claimed superiority was using outdated training techniques (or even none at all) for the baseline. That's interesting, do you have a source? I've been thinking about switching to some fancy new architecture in prod, but wasn't sure it would be worth it.
  - PartiallyTyped 5y ago
    Current SOTA afaik are: - Patches is all you need [1] - ResNet-50 with new protocols [2] - MLP Mixers [3] [1] https://old.reddit.com/r/MachineLearning/comments/q35lex/r_patches_are_all_you_need/ https://old.reddit.com/r/MachineLearning/comments/q35lex/r_p... [2] https://old.reddit.com/r/MachineLearning/comments/q0vt2b/r_resnet_strikes_back_an_improved_training/ https://old.reddit.com/r/MachineLearning/comments/q0vt2b/r_r... [3] https://old.reddit.com/r/MachineLearning/comments/n59kjo/r_mlpmixer_an_allmlp_architecture_for_vision/ https://old.reddit.com/r/MachineLearning/comments/n59kjo/r_m...
    - ShamelessC 5y ago
      I think it's somewhat confusing to compare an MLP Mixer architecture to a vanilla ResNet-50.
      - PartiallyTyped 5y ago
        What do you mean confusing? ResNet-50 with updated training protocol seems to be doing pretty well!
bigbillheck 5y ago
> i wonder if the average professional coder, still has. chance to beat the big teams ? I attended the MICCAI workshops this year, and while I didn't go to the brain segmentation challenge, in the ones I did go to the winners were generally academic individuals or groups. My MICCAI login is on my work computer, so I'm not going to go and check right now, but a quick google shows that the winners of FeTS2021 were all academics: https://twitter.com/FeTS_Challenge/status/1444010949371994114/photo/1 https://twitter.com/FeTS_Challenge/status/144401094937199411... and HEKTOR2021: https://www.aicrowd.com/challenges/miccai-2021-hecktor/leaderboards?challenge_leaderboard_extra_id=667&challenge_round_id=879 https://www.aicrowd.com/challenges/miccai-2021-hecktor/leade... looks similar. EDIT: Academic groups are of course not the same as 'the average professional coder', but this is a fast-moving research field and if an individual is competing with the state-of-the-art that doesn't sound particularly 'average' to me. EDIT 2: The DiSCO challenge appears to have been dominated by academics, both individuals and small teams : https://twitter.com/GabrielPGirard/status/1443972878714773560 https://twitter.com/GabrielPGirard/status/144397287871477356...

OhHiMarkos 5y ago

> i wonder if the average professional coder, still has. chance to beat the big teams ? I shared the sentiment but I have began to think otherwise. First of all, we need solutions to big problems. I wouldn't want to tackle Brain tumor problems using DL but someone with the brainpower and resources such as NVIDIA, why not? But besides that, the whole scientific field is pretty young. An average coder or DL researcher isn't useless because Google has all the machines. There are fields where DL isn't used much. Even in the DL field there are problems to solve. Algorithms to optimize.

PartiallyTyped 5y ago

> i wonder if the average professional coder, still has. chance to beat the big teams ? The common sentiment in r/ML is that a) big labs need some accountability because they keep getting credit for work they published but never released source code and therefore third party audit is impossible *. Furthermore, Resnet-50 is back to sota because of up-to-date training protocols, and every paper that claimed superiority was using outdated training techniques (or even none at all) for the baseline. Furthermore, there are many suspicions that the current generation of DL/ML has reached a local minimum because we have collectively pushed for models, architectures, functions, and optimizers that did well on existing hardware. For example, one hypothesis is that the Adam and AdamW optimizers are doing so well on average because we optimized model architectures for them. * Before somebody claims that this is part of the research process and we should be able to validate and implement models on our own, I would like to point out that there have been multiple instances where the math appeared sound, but the implemented models were bugged, and outperformed the existing models *because* they were bugged (NB biased) and the test sets didn't capture that. When corrected, the models didn't perform any better than existing unbiased methods.

joconde 5y ago
> Furthermore, Resnet-50 is back to sota because of up-to-date training protocols, and every paper that claimed superiority was using outdated training techniques (or even none at all) for the baseline. That's interesting, do you have a source? I've been thinking about switching to some fancy new architecture in prod, but wasn't sure it would be worth it.
- PartiallyTyped 5y ago
  Current SOTA afaik are: - Patches is all you need [1] - ResNet-50 with new protocols [2] - MLP Mixers [3] [1] https://old.reddit.com/r/MachineLearning/comments/q35lex/r_patches_are_all_you_need/ https://old.reddit.com/r/MachineLearning/comments/q35lex/r_p... [2] https://old.reddit.com/r/MachineLearning/comments/q0vt2b/r_resnet_strikes_back_an_improved_training/ https://old.reddit.com/r/MachineLearning/comments/q0vt2b/r_r... [3] https://old.reddit.com/r/MachineLearning/comments/n59kjo/r_mlpmixer_an_allmlp_architecture_for_vision/ https://old.reddit.com/r/MachineLearning/comments/n59kjo/r_m...
  - ShamelessC 5y ago
    I think it's somewhat confusing to compare an MLP Mixer architecture to a vanilla ResNet-50.
    - PartiallyTyped 5y ago
      What do you mean confusing? ResNet-50 with updated training protocol seems to be doing pretty well!

bigbillheck 5y ago

> i wonder if the average professional coder, still has. chance to beat the big teams ? I attended the MICCAI workshops this year, and while I didn't go to the brain segmentation challenge, in the ones I did go to the winners were generally academic individuals or groups. My MICCAI login is on my work computer, so I'm not going to go and check right now, but a quick google shows that the winners of FeTS2021 were all academics: https://twitter.com/FeTS_Challenge/status/1444010949371994114/photo/1 https://twitter.com/FeTS_Challenge/status/144401094937199411... and HEKTOR2021: https://www.aicrowd.com/challenges/miccai-2021-hecktor/leaderboards?challenge_leaderboard_extra_id=667&challenge_round_id=879 https://www.aicrowd.com/challenges/miccai-2021-hecktor/leade... looks similar. EDIT: Academic groups are of course not the same as 'the average professional coder', but this is a fast-moving research field and if an individual is competing with the state-of-the-art that doesn't sound particularly 'average' to me. EDIT 2: The DiSCO challenge appears to have been dominated by academics, both individuals and small teams : https://twitter.com/GabrielPGirard/status/1443972878714773560 https://twitter.com/GabrielPGirard/status/144397287871477356...