This isn't the same with lookup tables, but in Explaining Attention with Program Synthesis they were able to replace some attention heads with Python programs:

This isn't the same with lookup tables, but in Explaining Attention with Program Synthesis they were able to replace some attention heads with Python programs: https://arxiv.org/abs/2606.19317 https://arxiv.org/abs/2606.19317