crypto: rsassa-pkcs1 - reject undersized keys when signing

rsassa_pkcs1_sign() subtracts 11 from the unsigned key size before
checking that the key is large enough for PKCS#1 v1.5 padding:

	if (slen + hash_prefix->size > ctx->key_size - 11)
		return -EOVERFLOW;

If the RSA modulus is shorter than 11 bytes, the subtraction wraps.
With a one-byte key and hash=none, the padding memset() writes past
the output buffer.

KASAN reports:

    BUG: KASAN: slab-out-of-bounds in rsassa_pkcs1_sign+0x1ad/0x3b0
    Write of size 4294967294 at addr ...
    The buggy address is located 0 bytes to the right of
     allocated 1-byte region [...]

Reject keys shorter than the minimum encoded message size.

Fixes: 3d5b1ecdea ("crypto: rsa - RSA padding algorithm")
Assisted-by: Codex:gpt-5
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Reviewed-by: Sudhakar Kuppusamy <sudhakar@linux.ibm.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This commit is contained in:
Jérémy Jean 2026-08-26 10:37:43 +00:00 committed by Herbert Xu
parent 307e27d3e0
commit 4cf50332b5

View File

@ -169,7 +169,7 @@ static int rsassa_pkcs1_sign(struct crypto_sig *tfm,
u8 *in_buf;
int err;
if (!ctx->key_size)
if (ctx->key_size < 11)
return -EINVAL;
if (dlen < ctx->key_size)