您错误地使用了密钥。在公用密钥加密中，加密使用公用密钥：

You are using keys wrongly. In public-key cryptography, encryption uses a public key:

openssl rsautl -in txt.txt -out txt2.txt -inkey public.pem -pubin -encrypt

并且对于解密，私钥

openssl rsautl -in txt2.txt inkey private.pem -decrypt

私钥（无 -pubin ）可用于加密，因为它实际上包含公用指数。请注意，通常不应该将RSA直接用于加密数据，而只能封装（RSA-KEM）或包装用于对称加密的密钥。

The private key (without -pubin) can be used for encryption since it actually contains the public exponent. Note that RSA should not normally be used to encrypt data directly, but only to 'encapsulate' (RSA-KEM) or 'wrap' the key(s) used for symmetric encryption.

但是您提到您实际上想学习签名。尽管从历史上看，RSA签名有时有时被描述为使用私钥加密，但这种描述具有误导性，实际上实施起来并不安全。签名和验证实际上是不同于加密和解密的不同操作， rsautl 仅执行其中的 part 个。例如，您可以执行以下操作：

But you mention you actually want to study signature. Although historically RSA signature was sometimes described as 'encrypting with the private key', that description is misleading and actually implementing that was found to be insecure. Sign and verify are actually different operations separate from encryption and decryption, and rsautl performs only part of them. For example, you can do:

# hash the data and encode the result in ASN.1 
openssl rsautl -sign -in hashenc.dat -out sig.dat -inkey private.pem
...
# on the recipient (with signature and purportedly correct data)
openssl rsautl -verify -in sig.dat -out hashenc.dat -inkey public.pem -pubin 
# or often more appropriate use a certificate for the public key
openssl rsautl -verify -in sig.dat -out hashenc.dat -inkey cert.pem -certin
# now either decode hashenc.dat and compare the hash
# to a new hash of the data (which should be the same)
# or compare all of hashenc.dat to an encoding of a new hash

相反，***使用 openssl dgst 执行PKCS1指定的完整签名和验证顺序 例如rfc8017 。例如，对于带有SHA256的RSASSA-PKCS1v1_5签名 ：

Instead it is better to use openssl dgst which performs the entire signature and verification sequence as specified by PKCS1 e.g. rfc8017. For example for RSASSA-PKCS1v1_5 signature with SHA256:

openssl dgst -sha256 -sign private.pem -in data.txt -out sig.dat
# or can be abbreviated
openssl sha256 -sign private.pem -in data.txt -out sig.dat
# hashes the data, encodes the hash, does type 1 padding and modexp d
...
openssl dgst -sha256 -verify public.pem -in data.txt -signature     sig.dat
# or abbreviated 
openssl sha256 -verify public.pem -in data.txt -signature sig.dat 
# does modexp e and type 1 unpadding, and compares the result to a hash of the data

# notice you don't specify which key is public or private
# because this command knows what to expect

# however it does not accept the public key from a certificate, 
# you must extract the public key from the cert first

这种形式（但不是 rsautl ）还支持更新的和技术上更好的PS，但使用的并不广泛S填充。这仅在 dgst 手册页上进行了引用，并且大部分记录在 pkeyutl 手册页上，但并不完全显而易见。

This form (but not rsautl) also supports the newer and technically better, but not as widely used, PSS padding. This is only referenced on the dgst man page, and mostly documented on the pkeyutl man page, which isn't totally obvious.

在其他堆栈上，这些都是较主题化的，例如：
https://security.stackexchange.com/questions/93603/understanding-digitial-certifications

https://security.stackexchange.com/questions/87325/if-the-public-key -不能用于解密

https://security.stackexchange.com/questions/11879/is-encrypting-data-with-a-private-key-dangerous

https：//security.stackexchang e.com/questions/68822/trying-to-understand-rsa-and-its-terminology

https://crypto.stackexchange.com/questions/2123/rsa-encryption-with-private-key-and-用公钥解密

https://crypto.stackexchange.com/questions/15997/is-rsa-encryption-the-same-as-signature-generation

https://crypto.stackexchange.com/questions/15295/why-签名前需要散列的小数据

On other Stacks where this is more on-topic, see e.g.: https://security.stackexchange.com/questions/93603/understanding-digitial-certifications
https://security.stackexchange.com/questions/87325/if-the-public-key-cant-be-used-for-decrypting
https://security.stackexchange.com/questions/11879/is-encrypting-data-with-a-private-key-dangerous
https://security.stackexchange.com/questions/68822/trying-to-understand-rsa-and-its-terminology
https://crypto.stackexchange.com/questions/2123/rsa-encryption-with-private-key-and-decryption-with-a-public-key
https://crypto.stackexchange.com/questions/15997/is-rsa-encryption-the-same-as-signature-generation
https://crypto.stackexchange.com/questions/15295/why-the-need-to-hash-before-signing-small-data