There are multiple choices. For example BERT authors use the pooled output from the last four layers after experimenting with a few other choices like concatena

There are multiple choices. For example BERT authors use the pooled output from the last four layers after experimenting with a few other choices like concatenation and only using the last or penultimate layers.