3 ms·
Scaling if context window size has been a problem but now good potential of solutions using mamba. HyenaDNA is one to look at wrt DNA. And here are some other
by robwwilliams 3y ago
Scaling if context window size has been a problem but now good potential of solutions using mamba.
HyenaDNA is one to look at wrt DNA.
And here are some other interesting links from Erik Garrison—a leader in pangenomics.
https://hazyresearch.stanford.edu/blog/2023-06-29-hyena-dna https://hazyresearch.stanford.edu/blog/2023-06-29-hyena-dna
https://github.com/instadeepai/nucleotide-transformer https://github.com/instadeepai/nucleotide-transformer
https://dl.acm.org/doi/pdf/10.1145/3535508.3545512 https://dl.acm.org/doi/pdf/10.1145/3535508.3545512
https://github.com/dnbaker/bioseq https://github.com/dnbaker/bioseq
https://huggingface.co/AIRI-Institute/gena-lm-bert-base https://huggingface.co/AIRI-Institute/gena-lm-bert-base
https://discuss.huggingface.co/t/dna-long-sequence-tokenization/16154 https://discuss.huggingface.co/t/dna-long-sequence-tokenizat...