Your private key files are the equivalent of a password.

You should protect them under any and all circumstances. If someone acquires your private key, they can log in to any SSH server as an identity that authorizes the corresponding public key to log in.