I have an ARM board with 4 A53 cores with just 1GB of RAM. Despite these limitations I can successfully cross compile from my host. So yes, under circumstances

I have an ARM board with 4 A53 cores with just 1GB of RAM. Despite these limitations I can successfully cross compile from my host. So yes, under circumstances memory might not be such a big issue.