Common Lisp allows one to use arrays or other structures, and can even be used to inline assembly. So despite lists being a major part of the ecosystem and code

Common Lisp allows one to use arrays or other structures, and can even be used to inline assembly. So despite lists being a major part of the ecosystem and code representation itself, they are not mandatory for implementing an algorithm, or necessarily a performance drawback to using lisp. By the same token, it's easy for people to accidentally use lists and pointers to implement algorithms just as inefficiently in other languages - Python comes to mind. A standard approach in Python is to lean on external libraries for high performance computation, and this can be done just the same way in Lisp - but Lisp can also be used directly to write efficient low level algorithms by making a conscious effort not to use the list- and pointer-based functions to do so.