x64 has all kinds of vendor exclusive instructions sets (AVX-512 and SEV-SNP for example). The solution for most programs and compilers is to either be conservative (only including the most common instruction sets) or to compile code to multiple different paths depending on what extensions are available (resulting in larger binaries that can use more modern features).

For a while, I rented a dedicated server for cheap that was running a first generation i7. Performance was fine for web server stuff, but the lack of AES acceleration was definitely annoying and several times I’ve run into attempts to run SSE instructions that the processor simply couldn’t run. I’ve also run into instruction set annoyances when trying to run pre-compiled machine learning tools that assumed functional AVX instructions that my desktop doesn’t understand.

Most ARM devices run Armv8-A or these days Armv9-A, with possibly some extensions. There’s also the special ARM Thumb (which is a requirement for some platform code) but that’s supported by most ARM chips as well. Like on x64, there are a few optional extensions (like Neon for SIMD, or Apple’s additions to make Rosetta2 work as well as it does).

The instruction sets match enough that distros like Debian and Alpine have most of their packages available for ARM (aarch64). Once you get a kernel loaded and the system up and running, you can grab generic binaries from standard distros and run a normal Linux system without ever having to recompile your userland code.

The biggest problem on ARM remains getting a working kernel and getting it to boot.