The problem is the ecosystem. You can get response times for the main page down pretty low (especially with caching plugins) and the time to serve static assets

The problem is the ecosystem. You can get response times for the main page down pretty low (especially with caching plugins) and the time to serve static assets is the same for Wordpress or a static site. Most likely if your site feels slow, the culprit is your theme and one or more plugins on your site. You can usually track down problem plugin(s) using PageSpeed Insights since their frontend assets are served from the plugin folders.