6 ms·

Author jumps straight into how to implement this in code before giving any clues as to what the strategy or purpose is. The idea is funny, but it would be nice

• Tenoke 8y ago
  >These posts (along with the last) are my first two in a series where I will attempt to increase the size of OpenAI's GPT-2 model while taking advantage of the training the model has already gotten. Their model is a great candidate for this experiment, as OpenAI have already demonstrated great results with what is basically a bigger version of it. Not that I disagree that a bigger intro would be a bad thing - you are right to point out that there is little reasoning given.
• phowon 8y ago
  The Hug of Death seems to have killed the page so I can't really tell what it's about, but here's an example of inserting new trainable layers in BERT: https://arxiv.org/abs/1902.00751 https://arxiv.org/abs/1902.00751
  • Tenoke 8y ago
    I don't think it's Hug of Death even since it's not actually that much traffic and it's not down for me. Cloudflare + Github Pages just seems to be a much less reliable combination than I believed before I moved to it a year ago (it sometimes goes down even with almost no traffic, and then is up a second later). This paper is definitely relevant though!
• bmcooley 8y ago
  I don't have a strong background in AI/ML, but aren't quite a few advances in this tech driven by a "throw at the wall and see what sticks" attitude? I was under the impression that the "why" in some mechanics in advanced neural nets were poorly understood and just happened to work well. Please enlighten me if that isn't the case.