The calculator has an RSA public key in it (exponent e, modulus n). TI keeps secret an RSA private key (exponent d, same modulus n). The private key is used to

The calculator has an RSA public key in it (exponent e, modulus n). TI keeps secret an RSA private key (exponent d, same modulus n). The private key is used to sign the OS and the calculator verifies the signature before running the OS. The modulus is composed of two primes, p and q. Factoring n into p and q is sufficient to regenerate the private exponent d. Thus, the hackers can now sign their own OS and have it loaded on stock calculators. I suspect the 512-bit RSA key was used in order for the Z80 to verify a signature in a short period of time. It's possible they even use a small public exponent (e=3), which could enable simpler/quicker attacks than factoring. TI could have solved this in a number of ways. Partitioning the key space so there isn't one global key (like broadcast encryption) is one solution. Another would be to use ECDSA with a fast curve.