The whole point of using seL4 is that you can use it to provide guaranteed isolation. If you were to use seL4 it would be so that you could write those drivers

The whole point of using seL4 is that you can use it to provide guaranteed isolation. If you were to use seL4 it would be so that you could write those drivers in user mode and so that a bug in one would not let you extract crypto keys in another.