fwTPM: RSA-OAEP with SHA-1 uses MGF1-SHA-256, and TPM_CAP_ALGS doesn't list SHA-1 #650

SHA-1 is implemented, but OAEP with SHA-1 doesn't work and the algorithm list leaves SHA-1 out. Two related problems:

1. FwGetMgfType has no TPM_ALG_SHA1 case, so it falls back to WC_MGF1SHA256. TPM2_RSA_Decrypt with an OAEP SHA-1 scheme therefore uses SHA-1 for the label hash but MGF1-SHA-256 for the mask, and can't decrypt a standard RSA-OAEP(SHA-1) ciphertext: it returns TPM_RC_FAILURE. TPM2_RSA_Encrypt produces non-standard ciphertexts the same way. SHA-512 was added to FwGetMgfType on master; SHA-1 wasn't.
2. TPM2_GetCapability(TPM_CAP_ALGS) doesn't list TPM_ALG_SHA1 (0x0004), though fwTPM computes SHA-1 everywhere else (TPM2_GetHashType, PCR banks, HMAC, OAEP). SHA-512 was added to the list on master; SHA-1 wasn't. Clients that check the list before choosing a hash (tpm2-pkcs11 does, for OAEP and RSA-PSS) take SHA-1 for missing.

Reproduction. Create an RSA-2048 decrypt key with a NULL scheme. Encrypt a message with OpenSSL using the key's public half and RSA_PKCS1_OAEP_PADDING with EVP_sha1() for both the OAEP hash and MGF1. Then call Esys_RSA_Decrypt with {TPM2_ALG_OAEP, SHA1} and an empty label:

                                      fwTPM master          swtpm 0.10.2
RSA_Decrypt(OAEP-SHA1) of OpenSSL ct  TPM_RC_FAILURE        success, same plaintext
TPM_CAP_ALGS                          0001 0005 0006 0008   0001 0004 0005 ...
                                      000b 000c 000d ...

Suggested fix. Add the SHA-1 cases, under NO_SHA like the hash mapping. With this change the decryption returns the plaintext and the list includes 0x0004.

diff --git a/src/fwtpm/fwtpm_command.c b/src/fwtpm/fwtpm_command.c
index c4091f3..0bba845 100644
--- a/src/fwtpm/fwtpm_command.c
+++ b/src/fwtpm/fwtpm_command.c
@@ -1835,6 +1835,9 @@ static TPM_RC FwCmd_GetCapability(FWTPM_CTX* ctx, TPM2_Packet* cmd,
             #ifndef NO_RSA
                 { TPM_ALG_RSA, TPMA_ALGORITHM_asymmetric |
                     TPMA_ALGORITHM_object },
+            #endif
+            #ifndef NO_SHA
+                { TPM_ALG_SHA1, TPMA_ALGORITHM_hash },
             #endif
                 { TPM_ALG_SHA256, TPMA_ALGORITHM_hash },
             #ifdef WOLFSSL_SHA384
diff --git a/src/fwtpm/fwtpm_crypto.c b/src/fwtpm/fwtpm_crypto.c
index c0de62e..0b52268 100644
--- a/src/fwtpm/fwtpm_crypto.c
+++ b/src/fwtpm/fwtpm_crypto.c
@@ -87,6 +87,9 @@ int FwGetMgfType(UINT16 hashAlg)
     #endif
     #ifdef WOLFSSL_SHA384
         case TPM_ALG_SHA384: return WC_MGF1SHA384;
+    #endif
+    #ifndef NO_SHA
+        case TPM_ALG_SHA1:   return WC_MGF1SHA1;
     #endif
         case TPM_ALG_SHA256: /* fallthrough */
         default:             return WC_MGF1SHA256;

Environment

• wolfTPM master at d7f73b5 (also seen in v4.2.0), fwTPM built with ./configure --enable-fwtpm-only --enable-swtpm on wolfSSL v5.9.4-stable (--enable-wolftpm --enable-pkcallbacks --enable-keygen).
• fwtpm_server --port 2341 --platform-port 2342, reached through the tpm2-tss 4.2.0 swtpm TCTI (swtpm:host=127.0.0.1,port=2341).
• For comparison, the same program against swtpm 0.10.2 (libtpms).
• Found while running tpm2-pkcs11 on fwTPM.