Why does SSL handshake give 'Could not generate DH keypair' exception?
The problem is the prime size. The maximum-acceptable size that Java accepts is 1024 bits. This is a known issue (see JDK-6521495).
The bug report that I linked to mentions a workaround using BouncyCastle's JCE implementation. Hopefully that should work for you.
UPDATE
This was reported as bug JDK-7044060 and fixed recently.
Note, however, that the limit was only raised to 2048 bit. For sizes > 2048 bit, there is JDK-8072452 - Remove the maximum prime size of DH Keys; the fix appears to be for 9.
Getting Could not generate DH keypair Exception
The keyexchange is DHE_RSA which means Diffie Hellman Ephemeral authenticated by RSA. The DHE key, and even the DHE parameters, are not in the certificate, which is longterm (FSVO long) not ephemeral. The key in the certificate is RSA, and Java has no problem dealing with RSA at secure sizes, which today is more than 1024 and conventionally 2048.
The direct fix is to use old-Java-compatible weak DH(E) parameters. There are probably a dozen or more SSL/TLS server programs that use OpenSSL, or otherwise use PEM format files for the cert and (RSA) privatekey. (Lots of software uses or can handle PEM for cert, but mostly OpenSSL does so for privatekey.) Many of these server programs allow configuring the DH(E) parameters, but all that I've seen do so differently, and you don't identify yours.
A workaround is to avoid DHE_anything and use plain-RSA keyexchange. All SSL/TLS implementations including OpenSSL implement this. However, most applications and/or middlewares no longer prefer it and some don't even allow it, because it doesn't provide Forward Secrecy. How you control the ciphersuites, and thus keyexchange, again depends on the unidentified server program.
A better workaround is to enable ECDHE (and specifically ECDHE_RSA, since your cert/key is RSA) if you can.
3A. Java6 JSSE implements ECDHE protocols, but enables them only if there is a JCE "provider" for ECC primitives in your JRE -- and Oracle/Sun JRE6 as delivered (and TTBOMK the OpenJDK one also) does not have an ECC provider. To add ECC provider to JRE6
download
bcprov-jdk15on-$version.jar
from http://www.bouncycastle.org/latest_releases.html and put it in yourJRE/lib/ext
edit
JRE/lib/security/java.security
to add to the list of providers a line likesecurity.provider.N=org.bouncycastle.jce.provider.BouncyCastleProvider
where N is the next available number.
JRE7 does include ECC provider, and supports ECDHE out-of-the-box, if that is an option.
3B. If your server uses OpenSSL 1.0.0 or higher (except for certain older RedHat builds) it implements ECDHE, but it can be used only if (1) the ciphersuites are enabled, which is true by default but could be disabled by the server program or its configuration, and (2) the server program sets tmp_ecdh parameters (or in 1.0.2 enables auto-setting). Both of these depend on the unidentified server program, and if the server program doesn't use OpenSSL the answer may be very different.
If you identify your server program and the SSL/TLS-related parts of its configuration, I (or perhaps someone else) can be more specific.
EDIT server is nginx:
(1) nginx should be able to use DHE params Java(6,7) can handle, see http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_dhparam and create the file with
openssl dhparam 1024 >mydhfile
(or if you really want, a smaller DSA-1 size = 512 to 1024 and a multiple of 64, but you don't need and I don't recommend smaller than 1024).
(2) alernatively, to disable DHE, don't add it but do add at least some kRSA (or just RSA) in the ssl_ciphers string. Change the string in your comment at least to
EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA256:EECDH:kRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS
or because unmodified JRE6 won't offer ECDHE (or TLSv1.2) and we haven't added any PSK or SRP or DSS or aNULL that need to be deleted, and you distrust 3DES (why?) and RC4 and Java doesn't implement SEED or Camellia, you could use much more simply just
AES128-SHA:AES256-SHA
(These are actually TLS_RSA_WITH_AES*_CBC_SHA but for hysterical raisins the OpenSSL name omits the RSA, and the CBC.)
Tomcat by default uses Java (JSSE) to handle HTTPS connections, but depending on the packaging/install often can use APR also called "Tomcat native" or just "native" which is actually OpenSSL. If Tomcat/JSSE is run on JRE6 or 7, it uses DHE size 768 which JRE6,7 client can handle; if run on JRE8 it defaults to size 1024 which JRE6,7 client can handle. I don't know what Tomcat/APR uses (and can't easily test) but it could well be 1024 or less. If you want to find out and have Tomcat/APR running and openssl 1.0.2 available, use openssl s_client -connect host:port -tls1 -cipher EDH+AES
; when it connects enter Q, return; and look about 20 lines up for "Server Temp Key".
Related Topics
How to Directly Initialize a Hashmap (In a Literal Way)
Scale the Imageicon Automatically to Label Size
Initial Bytes Incorrect After Java Aes/Cbc Decryption
Connecting to Remote Url Which Requires Authentication Using Java
What Is the Default Initialization of an Array in Java
Regex for Matching Something If It Is Not Preceded by Something Else
Executorservice That Interrupts Tasks After a Timeout
Concatenating Null Strings in Java
Retrieve Column Names from Java.Sql.Resultset
Slf4J: Failed to Load Class "Org.Slf4J.Impl.Staticloggerbinder"
Java Array, Finding Duplicates
Centering a Jlabel on a JPAnel
Compiling a Java Program into an Executable
How to Load a Jar File at Runtime