How to Deal with a Slow Securerandom Generator

How to deal with a slow SecureRandom generator?

If you want true random data, then unfortunately you have to wait for it. This includes the seed for a SecureRandom PRNG. Uncommon Maths can't gather true random data any faster than SecureRandom, although it can connect to the internet to download seed data from a particular website. My guess is that this is unlikely to be faster than /dev/random where that's available.

If you want a PRNG, do something like this:

SecureRandom.getInstance("SHA1PRNG");

What strings are supported depends on the SecureRandom SPI provider, but you can enumerate them using Security.getProviders() and Provider.getService().

Sun is fond of SHA1PRNG, so it's widely available. It isn't especially fast as PRNGs go, but PRNGs will just be crunching numbers, not blocking for physical measurement of entropy.

The exception is that if you don't call setSeed() before getting data, then the PRNG will seed itself once the first time you call next() or nextBytes(). It will usually do this using a fairly small amount of true random data from the system. This call may block, but will make your source of random numbers far more secure than any variant of "hash the current time together with the PID, add 27, and hope for the best". If all you need is random numbers for a game, though, or if you want the stream to be repeatable in future using the same seed for testing purposes, an insecure seed is still useful.

Slow SecureRandom initialization

(Note: see also latest updates at the end of this answer)

Explanation

Reason for this is default SecureRandom provider.

On Windows, there are 2 SecureRandom providers available:

- provider=SUN, type=SecureRandom, algorithm=SHA1PRNG
- provider=SunMSCAPI, type=SecureRandom, algorithm=Windows-PRNG

On Linux (tested in Alpine docker with Oracle JDK 8u162):

- provider=SUN, type=SecureRandom, algorithm=NativePRNG
- provider=SUN, type=SecureRandom, algorithm=SHA1PRNG
- provider=SUN, type=SecureRandom, algorithm=NativePRNGBlocking
- provider=SUN, type=SecureRandom, algorithm=NativePRNGNonBlocking

These are specified in jre/lib/security/java.security file.

security.provider.1=sun.security.provider.Sun
...
security.provider.10=sun.security.mscapi.SunMSCAPI

By default, first SecureRandom provider is used. On Windows, the default one is sun.security.provider.Sun, and this implementation reports following when JVM is run with -Djava.security.debug="provider,engine=SecureRandom":

Provider: SecureRandom.SHA1PRNG algorithm from: SUN
provider: Failed to use operating system seed generator: java.io.IOException: Required native CryptoAPI features not  available on this machine
provider: Using default threaded seed generator

And the default threaded seed generator is very slow.

You need to use SunMSCAPI provider.

Solution 1: Configuration

Reorder providers in configuration:

Edit jre/lib/security/java.security:

security.provider.1=sun.security.mscapi.SunMSCAPI
...
security.provider.10=sun.security.provider.Sun

I am not aware this can be done via system properties.

Or maybe yes, using-Djava.security.properties (untested, see this)

Solution 2: Programmatic

Reorder providers programmatically:

Optional.ofNullable(Security.getProvider("SunMSCAPI")).ifPresent(p->{
    Security.removeProvider(p.getName());
    Security.insertProviderAt(p, 1);
});

JVM now reports following (-Djava.security.debug="provider,engine=SecureRandom"):

Provider: SecureRandom.Windows-PRNG algorithm from: SunMSCAPI

Solution 3: Programmatic v2

Inspired by this idea, following piece of code inserts only a single SecureRandom service, configured dynamically from existing SunMSCAPI provider without the explicit reliance on sun.* classes. This also avoids the potential risks associated with indiscriminate prioritization of all services of SunMSCAPI provider.

public interface WindowsPRNG {

    static void init() {
        String provider = "SunMSCAPI"; // original provider
        String type = "SecureRandom"; // service type
        String alg = "Windows-PRNG"; // algorithm
        String name = String.format("%s.%s", provider, type); // our provider name
        if (Security.getProvider(name) != null) return; // already registered
        Optional.ofNullable(Security.getProvider(provider)) // only on Windows
                .ifPresent(p-> Optional.ofNullable(p.getService(type, alg)) // should exist but who knows?
                        .ifPresent(svc-> Security.insertProviderAt( // insert our provider with single SecureRandom service
                                new Provider(name, p.getVersion(), null) {{
                                    setProperty(String.format("%s.%s", type, alg), svc.getClassName());
                                }}, 1)));
    }

}

Performance

<140 msec (instead of 5000+ msec)

Details

There is a call to new SecureRandom() somewhere down the call stack when you use URL.openConnection("https://...")

It calls getPrngAlgorithm() (see SecureRandom:880)

And this returns first SecureRandom provider it finds.

For testing purposes, call to URL.openConnection() can be replaced with this:

new SecureRandom().generateSeed(20);

Disclaimer

I am not aware of any negative side effects caused by providers reordering. However, there may be some, especially considering default provider selection algorithm.

Anyway, at least in theory, from functional point of view this should be transparent to application.

Update 2019-01-08

Windows 10 (version 1803): Cannot reproduce this issue anymore on any of latest JDKs (tested all from old oracle 1.7.0_72 up to openjdk "12-ea" 2019-03-19).

It looks like it was Windows issue, fixed in latest OS updates. Related updates may or may not have taken place in recent JRE releases, too.
However, I cannot reproduce the original issue even with my oldest JDK 7 update 72 installation which was definitelly affected, and definitelly not patched in any way.

There are still minor performance gains when using this solution (cca 350 msec on average) but the default behavior no longer suffers the intolerable 5+ seconds penalty.

Slow service response Times : Java SecureRandom & /dev/random

I could not check your OpenJDK concrete version, but I could check jdk6-b33.

SecureRandom uses SeedGenerator to get the seed bytes

public byte[] engineGenerateSeed(int numBytes) {
    byte[] b = new byte[numBytes];
    SeedGenerator.generateSeed(b);
    return b;
}

SeedGenerator gets the seedSource (String) from SunEntries

String egdSource = SunEntries.getSeedSource();

SunEntries tries to get the source from the system property java.security.egd first, if is not found then tries to get the property securerandom.source from the java.security properties file, if the property is not found returns a blank string.

// name of the *System* property, takes precedence over PROP_RNDSOURCE
private final static String PROP_EGD = "java.security.egd";
// name of the *Security* property
private final static String PROP_RNDSOURCE = "securerandom.source";

final static String URL_DEV_RANDOM = "file:/dev/random";
final static String URL_DEV_URANDOM = "file:/dev/urandom";

private static final String seedSource;

static {
    seedSource = AccessController.doPrivileged(
            new PrivilegedAction<String>() {

        public String run() {
            String egdSource = System.getProperty(PROP_EGD, "");
            if (egdSource.length() != 0) {
                return egdSource;
            }
            egdSource = Security.getProperty(PROP_RNDSOURCE);
            if (egdSource == null) {
                return "";
            }
            return egdSource;
        }
    });
}

the SeedGenerator check this value to initialize the instance

// Static instance is created at link time
private static SeedGenerator instance;

private static final Debug debug = Debug.getInstance("provider");

final static String URL_DEV_RANDOM = SunEntries.URL_DEV_RANDOM;
final static String URL_DEV_URANDOM = SunEntries.URL_DEV_URANDOM;

// Static initializer to hook in selected or best performing generator
static {
    String egdSource = SunEntries.getSeedSource();

    // Try the URL specifying the source
    // e.g. file:/dev/random
    //
    // The URL file:/dev/random or file:/dev/urandom is used to indicate
    // the SeedGenerator using OS support, if available.
    // On Windows, the causes MS CryptoAPI to be used.
    // On Solaris and Linux, this is the identical to using
    // URLSeedGenerator to read from /dev/random

    if (egdSource.equals(URL_DEV_RANDOM) || egdSource.equals(URL_DEV_URANDOM)) {
        try {
            instance = new NativeSeedGenerator();
            if (debug != null) {
                debug.println("Using operating system seed generator");
            }
        } catch (IOException e) {
            if (debug != null) {
                debug.println("Failed to use operating system seed "
                              + "generator: " + e.toString());
            }
        }
    } else if (egdSource.length() != 0) {
        try {
            instance = new URLSeedGenerator(egdSource);
            if (debug != null) {
                debug.println("Using URL seed generator reading from "
                              + egdSource);
            }
        } catch (IOException e) {
            if (debug != null)
                debug.println("Failed to create seed generator with "
                              + egdSource + ": " + e.toString());
        }
    }

    // Fall back to ThreadedSeedGenerator
    if (instance == null) {
        if (debug != null) {
            debug.println("Using default threaded seed generator");
        }
        instance = new ThreadedSeedGenerator();
    }
}

if the source is

final static String URL_DEV_RANDOM = "file:/dev/random";

or

final static String URL_DEV_URANDOM = "file:/dev/urandom"

uses the NativeSeedGenerator, on Windows tries to use the native CryptoAPI on Linux the class simply extends the SeedGenerator.URLSeedGenerator

package sun.security.provider;

import java.io.IOException;

/**
 * Native seed generator for Unix systems. Inherit everything from
 * URLSeedGenerator.
 *
 */
class NativeSeedGenerator extends SeedGenerator.URLSeedGenerator {

    NativeSeedGenerator() throws IOException {
        super();
    }

}

and call to the superclass constructor who loads /dev/random by default

URLSeedGenerator() throws IOException {
    this(SeedGenerator.URL_DEV_RANDOM);
}

so, OpenJDK uses /dev/random by default until you do not set another value in the system property java.security.egd or in the property securerandom.source of security properties file.

If you want to see the read results using strace you can change the command line and add the trace=open,read expression

sudo strace -o a.strace -f -e trace=open,read java class

the you can see something like this (I did the test with Oracle JDK 6)

13225 open("/dev/random", O_RDONLY)     = 8
13225 read(8, "@", 1)                   = 1
13225 read(3, "PK\3\4\n\0\0\0\0\0RyzB\36\320\267\325u\4\0\0u\4\0\0 \0\0\0", 30) = 30
....
....

The Tomcat Wiki section for faster startup suggest using a non-blocking entropy source like /dev/urandom if you are experiencing delays during startup

More info: https://wiki.apache.org/tomcat/HowTo/FasterStartUp#Entropy_Source

Hope this helps.

Behaviour of SecureRandom

You are victim to a common misbelief about random numbers in general: a random sequence doesn't mean that a number cannot be repeated in that sequence. Quite on the contrary, it has to with a high probability. That misbelief is actually used to tell a "random" sequence generated by humans from a real one. A "random" sequence of 0's and 1's generated by a human will probably look like this:

0, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0, ....

while a real random sequence is not shy of repeating the same number more than twice :) A good example is that statistical tests also look for repetition.

Both kinds of generators have good "statistical properties"

It's also a common misbelief that cryptographically secure random numbers would somehow yield "much more random" values. Their statistical probabilities will probably be pretty much alike and both will perform really well in those standard statistical tests.

Where to use which

So it actually depends on what you want to do whether your choice should be a PRNG or a cryptographically secure PRNG (CSPRNG). "Normal" PRNGs are perfectly fine for simulation purposes such as Monte-Carlo methods etc. The additional benefit of a CSPRNG will give you is that of non-predictability. Because the CSPRNG can "do more" chances are high that its performance will also be worse than that of a vanilla PRNG.

It can be shown that the concept of a "secure" PRNG is tightly coupled with the ability to predict the next bit of its output. For a CSPRNG, predicting the next bit of its output at any time is computationally infeasible. This only holds if you treat its seed value as a secret, of course. Once anyone finds out the seed, the whole thing becomes easily predictable - just recompute the values already generated by the CSPRNG's algorithm and then compute the next value. It can further be shown that being immune to "next-bit prediction" actually implies that there's no statistical test whatsoever that could distinguish the distribution of the CSPRNG from that of a real random uniform distribution. So there's another difference between PRNG and CSPRNG: While a good PRNG will perform well in many statistical tests, a CSPRNG is guaranteed to perform well in all tests.

The rule of thumb where to use which is that

• You use the CSPRNG in a "hostile" environment where you don't want externals to be able to guess sensitive information (session IDs, online poker where real money is won/lost, ....)
• And the PRNG in a benevolent environment where you just require good statistical properties but don't care about predictability (Monte-Carlo simulations, single player poker vs. computer, computer games in general) - i.e. no money can be won or lives will be lost should somebody be able to predict those random numbers successfully.

SecureRandom: init once or every time it is needed?

Unlike the java.util.Random class, the java.security.SecureRandom class must produce non-deterministic output on each call.

What that means is, in case of java.util.Random, if you were to recreate an instance with the same seed each time you needed a new random number, you would essentially get the same result every time. However, SecureRandom is guaranteed to NOT do that - so, creating a single instance or creating a new one each time does not affect the randomness of the random bytes it generates.

So, from just normal good coding practices view point, why create too many instances when one will do?

Slow startup on Tomcat 7.0.57 because of SecureRandom

The secure random calls may be blocking as there is not enough entropy to feed them in /dev/random.

If you have the line

securerandom.source=file:/dev/random

in /jre/lib/security/java.security, changing this to urandom may improve things (although this is probably already the default).

Alternatively there are some suggestions on how to feed the pool here

https://security.stackexchange.com/questions/89/feeding-dev-random-entropy-pool

Tomcat takes too much time to start - Java SecureRandom

After some extensive Googling with the right search terms, I came across this nice article on DigitalOcean. I am merely quoting the relevant part here.

There are two general random devices on Linux: /dev/random and
/dev/urandom. The best randomness comes from /dev/random, since it's a
blocking device, and will wait until sufficient entropy is available
to continue providing output. Assuming your entropy is sufficient, you
should see the same quality of randomness from /dev/urandom; however,
since it's a non-blocking device, it will continue producing “random”
data, even when the entropy pool runs out. This can result in lower
quality random data, as repeats of previous data are much more likely.
Lots of bad things can happen when the available entropy runs low on a
production server, especially when this server performs cryptographic
functions.

So, its a potential security risk.

Solution for Populating Entropy Pools

Linux already gets very good quality random data from the
different hardware sources, but since a headless machine usually
has no keyboard or mouse, there is much less entropy generated. Disk
and network I/O represent the majority of entropy generation sources
for these machines, and these produce very sparse amounts of entropy.
Since very few headless machines like servers or cloud servers/virtual
machines have any sort of dedicated hardware RNG solution available,
there exist several userland solutions to generate additional entropy
using hardware interrupts from devices that are “noisier” than hard
disks, like video cards, sound cards, etc. This once again proves to
be an issue for servers unfortunately, as they do not commonly contain
either one

Solution: haveged

Based on the HAVEGE principle, and previously based on its associated
library, haveged allows generating randomness based on variations in
code execution time on a processor. Since it's nearly impossible for
one piece of code to take the same exact time to execute, even in the
same environment on the same hardware, the timing of running a single
or multiple programs should be suitable to seed a random source. The
haveged implementation seeds your system's random source (usually
/dev/random) using differences in your processor's time stamp counter
(TSC) after executing a loop repeatedly

How to install haveged

Follow the steps in this article. https://www.digitalocean.com/community/tutorials/how-to-setup-additional-entropy-for-cloud-servers-using-haveged

---

Related Topics