2 ms·
ProtTrans: Understanding the Language of Life Through Self-Supervised Learning
- criticaltinker 5y agoFantastic paper all around, and truly amazing results. The supercomputers they used at Oak Ridge National Laboratory have a mind-blowing amount of computational power, enough to make anyone training machine learning models a bit envious. Here are quotes from the abstract: > we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids. > The protein LMs (pLMs) were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores. > Dimensionality reduction revealed that the raw pLM-embeddings from unlabeled data captured some biophysical features of protein sequences. > We validated the advantage of using the embeddings as exclusive input for several subsequent tasks: (1) a per-residue (per-token) prediction of protein secondary structure (3-state accuracy Q3=81%-87%); (2) per-protein (pooling) predictions of protein sub-cellular location (ten-state accuracy: Q10=81%) and membrane vs. water-soluble (2-state accuracy Q2=91%). > For secondary structure, the most informative embeddings (ProtT5) for the first time outperformed the state-of-the-art without multiple sequence alignments (MSAs) or evolutionary information thereby bypassing expensive database searches. > Taken together, the results implied that pLMs learned some of the grammar of the language of life.