That's the beauty of it - you don't have to choose, the network chooses for you. If you set up your layers such that the first input layer has one input for eac

That's the beauty of it - you don't have to choose, the network chooses for you. If you set up your layers such that the first input layer has one input for each possible token (lets say total of 80K weights), and the next layer connected to that layer only 1024 values, you will get "word embeddings" as a consequence of training the network (i.e. each node in the 80K layer will have 1024 values pointing to individual nodes of the next layer) The simplest way to get word embeddings (without necessarily building a complex GPT like model) is word2vec: https://towardsdatascience.com/creating-word-embeddings-coding-the-word2vec-algorithm-in-python-using-deep-learning-b337d0ba17a8 https://towardsdatascience.com/creating-word-embeddings-codi... - the principle is similar but the network is smaller.