4 ms·

The thing is that progress is so fast in some areas of ML, you dont really know when something could emerge. Who knows, maybe with enough training data and a de

The thing is that progress is so fast in some areas of ML, you dont really know when something could emerge. Who knows, maybe with enough training data and a deep enough network with the right supervision, something extraordinary pops out. Google google bert nlp. Google recently released an NLP algorithm that basically beats all predecessors, but the algorithm itself is extremely simple. Just lots of data and compute, no telling what will happen in ten years when the amount of data has exploded and GPUs are cheap.

throwawaymath 8y ago
This mentality is exactly what hprotagonist is talking about. All signs currently point to AGI being far away - at least several breakthroughs in currently unknown unknowns away, in fact. If the most you can say about the likelihood of something happening is that, "there's no telling what can happen in the next ten years!", then the only informed forecast is that it won't happen. We can likewise say we don't know if an extinction level event will happen due to a meteor (that we couldn't detect for some reason) hitting us in the next decade. Every time someone makes a sobering statement grounded in reality about artificial intelligence, another comment pops up which reduces to, "but who knows!" It doesn't mean anything.
glup 8y ago
bert provides incremental improvements on tasks that don't really stress-test contextualized world knowledge in linguistic tasks — contrast with Winograd schemas (SOTA: https://arxiv.org/pdf/1806.02847.pdf https://arxiv.org/pdf/1806.02847.pdf). Corpus data and compute don't magically solve everything.
- p1esk 8y ago
  The paper you linked to shows how a simple DL brute force approach (large model trained on lots of data) "successfully discovers important features of the context that decide the correct answer, indicating a good grasp of commonsense knowledge." Which kinda goes contrary to your last sentence. Go figure.
  - glup 8y ago
    My point is that while the DL + large corpus approach is the best available, it's still way, way lower than human performance (Tables 4 and 5).
    - p1esk 8y ago
      Sure, but we are talking about how fast it's progressing. Given that NNs for NLP field pretty much started with Bengio's paper in 2003, I'd say it's accelerating at an amazing rate.
      - glup 8y ago
        But NNs for language processing have been around much longer than that, e.g., Elman, 1990: https://crl.ucsd.edu/~elman/Papers/fsit.pdf https://crl.ucsd.edu/~elman/Papers/fsit.pdf.
freeone3000 8y ago
Yeah, Bert's great, but it has the Google Paper problem in that nobody else can reproduce the results because nobody else has the compute to train the dang thing. It also doesn't solve the fundamental issues - it scores a few percent higher on some standardized sets, but it gets wrong answers that are just as explainable as any other system - shrug-emoji. ‍️‍️

throwawaymath 8y ago

This mentality is exactly what hprotagonist is talking about. All signs currently point to AGI being far away - at least several breakthroughs in currently unknown unknowns away, in fact. If the most you can say about the likelihood of something happening is that, "there's no telling what can happen in the next ten years!", then the only informed forecast is that it won't happen. We can likewise say we don't know if an extinction level event will happen due to a meteor (that we couldn't detect for some reason) hitting us in the next decade. Every time someone makes a sobering statement grounded in reality about artificial intelligence, another comment pops up which reduces to, "but who knows!" It doesn't mean anything.

glup 8y ago

bert provides incremental improvements on tasks that don't really stress-test contextualized world knowledge in linguistic tasks — contrast with Winograd schemas (SOTA: https://arxiv.org/pdf/1806.02847.pdf https://arxiv.org/pdf/1806.02847.pdf). Corpus data and compute don't magically solve everything.

p1esk 8y ago
The paper you linked to shows how a simple DL brute force approach (large model trained on lots of data) "successfully discovers important features of the context that decide the correct answer, indicating a good grasp of commonsense knowledge." Which kinda goes contrary to your last sentence. Go figure.
- glup 8y ago
  My point is that while the DL + large corpus approach is the best available, it's still way, way lower than human performance (Tables 4 and 5).
  - p1esk 8y ago
    Sure, but we are talking about how fast it's progressing. Given that NNs for NLP field pretty much started with Bengio's paper in 2003, I'd say it's accelerating at an amazing rate.
    - glup 8y ago
      But NNs for language processing have been around much longer than that, e.g., Elman, 1990: https://crl.ucsd.edu/~elman/Papers/fsit.pdf https://crl.ucsd.edu/~elman/Papers/fsit.pdf.

freeone3000 8y ago

Yeah, Bert's great, but it has the Google Paper problem in that nobody else can reproduce the results because nobody else has the compute to train the dang thing. It also doesn't solve the fundamental issues - it scores a few percent higher on some standardized sets, but it gets wrong answers that are just as explainable as any other system - shrug-emoji. ‍️‍️