8 ms·
I'm curious how long this work took. Did you encountered many negative results before you got good results? What inspired you for the main insight?
by sytelus 6y ago
I'm curious how long this work took. Did you encountered many negative results before you got good results? What inspired you for the main insight?
- phillypham 6y agoI think the insight is not incredibly original and follows naturally from OpenAI's Sparse Transformer. The idea is similar to Longformer. Two teams at Google had a similar insight, hence the high number of authors. The original implementation only took a couple of months and was primarily motivated by internal Google applications. Natural Questions was the first external benchmark we tried to validate on, which took a few months to find the right setup. All the other datasets, took a few weeks but the effort was done in parallel given the large team. There was quite a bit of frustration dealing with Tensorflow, TPUs, and the XLA compiler that maybe set us back a few months, too.
- phillypham 6y agoEdit: This is not meant to criticize the Tensorflow, TPU, or XLA team. They responded quickly to our bugs and made the work possible. I just meant there was some extra organizational overhead.