I am curious what additional attacks knowing the last layer of an LLM enables. Eg you go from a black box attack to some sort of white box [1] Does it help wi

I am curious what additional attacks knowing the last layer of an LLM enables. Eg you go from a black box attack to some sort of white box [1] Does it help with adversarial prompt injection? What % of the network do you need to know to identify whether an item was included in the pretraining data with k% confidence? I assume we will see more of these and possibly complex zero days. Interesting if you can steal any non trivial % of model weights from a production model for relatively little money (compared to pretraining cost) [1] https://lilianweng.github.io/posts/2023-10-25-adv-attack-llm/ https://lilianweng.github.io/posts/2023-10-25-adv-attack-llm...