Lambda Expression VS Method Reference

Lambda expression vs method reference implementation details

This is an extract from the Brian Goetz's doc linked by Brett Oken:

When the compiler encounters a lambda expression, it first lowers
(desugars) the lambda body into a method whose argument list and
return type match that of the lambda expression, possibly with some
additional arguments (for values captured from the lexical scope, if
any.) At the point at which the lambda expression would be captured,
it generates an invokedynamic call site, which, when invoked, returns
an instance of the functional interface to which the lambda is being
converted. This call site is called the lambda factory for a given
lambda. The dynamic arguments to the lambda factory are the values
captured from the lexical scope. The bootstrap method of the lambda
factory is a standardized method in the Java language runtime library,
called the lambda metafactory. The static bootstrap arguments capture
information known about the lambda at compile time (the functional
interface to which it will be converted, a method handle for the
desugared lambda body, information about whether the SAM type is
serializable, etc.)

Method references are treated the same way as lambda expressions,
except that most method references do not need to be desugared into a
new method; we can simply load a constant method handle for the
referenced method and pass that to the metafactory.

Examples extracted from same doc:

As an example, consider a lambda that captures a field minSize:

list.filter(e -> e.getSize() < minSize )

We desugar this as an instance method, and pass the receiver as the first captured argument:

list.forEach(INDY((MH(metaFactory), MH(invokeVirtual Predicate.apply),
                    MH(invokeVirtual B.lambda$1))( this ))));

private boolean lambda$1(Element e) {
    return e.getSize() < minSize; }

While

list.filter(String::isEmpty)

is translated as:

list.filter(indy(MH(metaFactory), MH(invokeVirtual Predicate.apply),
             MH(invokeVirtual String.isEmpty))()))

Java Lambda Expressions and Method References to Generic Methods

The issue there is that "a lambda expression can be used for a functional interface only if the method in the functional interface has NO type parameters". (JLS11, 15.27.3 Type of a Lambda Expression) with one exception - this is not the case for congruent method references.

That is why it works in your first example and doesn't in the second:

SubmitterCompletable submitterCompletable = CompletableFutureUtils::runAsync; (OK)
SubmitterCompletable submitterCompletable = c -> <anything> (NOT OK)

---

There aren't many options out there I could think of to achieve what you want:

1. Implement the interface (either in-place using an anonymous class as you've mentioned or as a standalone class).

2. Use an intermediate helper class inside your CompletableFutureUtils that would keep a ref to the executor and expose a method congruent with your Submitter's functional method which will delegate the call to the underlying runAsync(Callable<U> callable, Executor executor) util's method.

Example code:

public final static class CompletableFutureUtils {
    public static <U> CompletableFuture<U> runAsync(Callable<U> callable) {
        ...
    }

    public static <U> CompletableFuture<U> runAsync(Callable<U> callable, Executor executor) {
        ...
    }
    
    public static ExecutorRunnerProxy using(Executor executor) {
        return new ExecutorRunnerProxy(executor);
    }

    public static final class ExecutorRunnerProxy {
        private final Executor executor;

        private ExecutorRunnerProxy(Executor executor) {
            this.executor = executor;
        }

        public <T> CompletableFuture<T> runAsync(Callable<T> task) {
            return CompletableFutureUtils.runAsync(task, executor);
        }
    }
}

Example usage:

SubmitterCompletable submitterCompletable = CompletableFutureUtils::runAsync; 
SubmitterCompletable submitterWithExecutor = CompletableFutureUtils.using(executor)::runAsync;

What is the difference between a lambda and a method reference at a runtime level?

Getting Started

To investigate this we start with the following class:

import java.io.Serializable;
import java.util.Comparator;

public final class Generic {

    // Bad implementation, only used as an example.
    public static final Comparator<Integer> COMPARATOR = (a, b) -> (a > b) ? 1 : -1;

    public static Comparator<Integer> reference() {
        return (Comparator<Integer> & Serializable) COMPARATOR::compare;
    }

    public static Comparator<Integer> explicit() {
        return (Comparator<Integer> & Serializable) (a, b) -> COMPARATOR.compare(a, b);
    }

}

After compilation, we can disassemble it using:

javap -c -p -s -v Generic.class

Removing the irrelevant parts (and some other clutter, such as fully-qualified types and the initialisation of COMPARATOR) we are left with

  public static final Comparator<Integer> COMPARATOR;    

  public static Comparator<Integer> reference();
      0: getstatic     #2  // Field COMPARATOR:LComparator;    
      3: dup    
      4: invokevirtual #3   // Method Object.getClass:()LClass;    
      7: pop    
      8: invokedynamic #4,  0  // InvokeDynamic #0:compare:(LComparator;)LComparator;    
      13: checkcast     #5  // class Serializable    
      16: checkcast     #6  // class Comparator    
      19: areturn

  public static Comparator<Integer> explicit();
      0: invokedynamic #7,  0  // InvokeDynamic #1:compare:()LComparator;    
      5: checkcast     #5  // class Serializable    
      8: checkcast     #6  // class Comparator    
      11: areturn

  private static int lambda$explicit$d34e1a25$1(Integer, Integer);
     0: getstatic     #2  // Field COMPARATOR:LComparator;
     3: aload_0
     4: aload_1
     5: invokeinterface #44,  3  // InterfaceMethod Comparator.compare:(LObject;LObject;)I
    10: ireturn

BootstrapMethods:    
  0: #61 invokestatic invoke/LambdaMetafactory.altMetafactory:(Linvoke/MethodHandles$Lookup;LString;Linvoke/MethodType;[LObject;)Linvoke/CallSite;    
    Method arguments:    
      #62 (LObject;LObject;)I    
      #63 invokeinterface Comparator.compare:(LObject;LObject;)I    
      #64 (LInteger;LInteger;)I    
      #65 5    
      #66 0    

  1: #61 invokestatic invoke/LambdaMetafactory.altMetafactory:(Linvoke/MethodHandles$Lookup;LString;Linvoke/MethodType;[LObject;)Linvoke/CallSite;    
    Method arguments:    
      #62 (LObject;LObject;)I    
      #70 invokestatic Generic.lambda$explicit$df5d232f$1:(LInteger;LInteger;)I    
      #64 (LInteger;LInteger;)I    
      #65 5    
      #66 0

Immediately we see that the bytecode for the reference() method is different to the bytecode for explicit(). However, the notable difference isn't actually relevant, but the bootstrap methods are interesting.

An invokedynamic call site is linked to a method by means of a bootstrap method, which is a method specified by the compiler for the dynamically-typed language that is called once by the JVM to link the site.

(Java Virtual Machine Support for Non-Java Languages, emphasis theirs)

This is the code responsible for creating the CallSite used by the lambda. The Method arguments listed below each bootstrap method are the values passed as the variadic parameter (i.e. args) of LambdaMetaFactory#altMetaFactory.

Format of the Method arguments

1. samMethodType - Signature and return type of method to be implemented by the function object.
2. implMethod - A direct method handle describing the implementation method which should be called (with suitable adaptation of argument types, return types, and with captured arguments prepended to the invocation arguments) at invocation time.
3. instantiatedMethodType - The signature and return type that should be enforced dynamically at invocation time. This may be the same as samMethodType, or may be a specialization of it.
4. flags indicates additional options; this is a bitwise OR of desired flags. Defined flags are FLAG_BRIDGES, FLAG_MARKERS, and FLAG_SERIALIZABLE.
5. bridgeCount is the number of additional method signatures the function object should implement, and is present if and only if the FLAG_BRIDGES flag is set.

In both cases here bridgeCount is 0, and so there is no 6, which would otherwise be bridges - a variable-length list of additional methods signatures to implement (given that bridgeCount is 0, I'm not entirely sure why FLAG_BRIDGES is set).

Matching the above up with our arguments, we get:

1. The function signature and return type (Ljava/lang/Object;Ljava/lang/Object;)I, which is the return type of Comparator#compare, because of generic type erasure.
2. The method being called when this lambda is invoked (which is different).
3. The signature and return type of the lambda, which will be checked when the lambda is invoked: (LInteger;LInteger;)I (note that these aren't erased, because this is part of the lambda specification).
4. The flags, which in both cases is the composition of FLAG_BRIDGES and FLAG_SERIALIZABLE (i.e. 5).
5. The amount of bridge method signatures, 0.

We can see that FLAG_SERIALIZABLE is set for both lambdas, so it's not that.

Implementation methods

The implementation method for the method reference lambda is Comparator.compare:(LObject;LObject;)I, but for the explicit lambda it's Generic.lambda$explicit$df5d232f$1:(LInteger;LInteger;)I. Looking at the disassembly, we can see that the former is essentially an inlined version of the latter. The only other notable difference is the method parameter types (which, as mentioned earlier, is because of generic type erasure).

When is a lambda actually serializable?

You can serialize a lambda expression if its target type and its captured arguments are serializable.

Lambda Expressions (The Java™ Tutorials)

The important part of that is "captured arguments". Looking back at the disassembled bytecode, the invokedynamic instruction for the method reference certainly looks like it's capturing a Comparator (#0:compare:(LComparator;)LComparator;, in contrast to the explicit lambda, #1:compare:()LComparator;).

Confirming capturing is the issue

ObjectOutputStream contains an extendedDebugInfo field, which we can set using the -Dsun.io.serialization.extendedDebugInfo=true VM argument:

$ java -Dsun.io.serialization.extendedDebugInfo=true Generic

When we try to serialize the lambdas again, this gives a very satisfactory

Exception in thread "main" java.io.NotSerializableException: Generic$$Lambda$1/321001045
        - element of array (index: 0)
        - array (class "[LObject;", size: 1)
/* ! */ - field (class "invoke.SerializedLambda", name: "capturedArgs", type: "class [LObject;") // <--- !!
        - root object (class "invoke.SerializedLambda", SerializedLambda[capturingClass=class Generic, functionalInterfaceMethod=Comparator.compare:(LObject;LObject;)I, implementation=invokeInterface Comparator.compare:(LObject;LObject;)I, instantiatedMethodType=(LInteger;LInteger;)I, numCaptured=1])
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1182)
    /* removed */
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
    at Generic.main(Generic.java:27)

What's actually going on

From the above, we can see that the explicit lambda is not capturing anything, whereas the method reference lambda is. Looking over the bytecode again makes this clear:

  public static Comparator<Integer> explicit();
      0: invokedynamic #7,  0  // InvokeDynamic #1:compare:()LComparator;    
      5: checkcast     #5  // class java/io/Serializable    
      8: checkcast     #6  // class Comparator    
      11: areturn

Which, as seen above, has an implementation method of:

  private static int lambda$explicit$d34e1a25$1(java.lang.Integer, java.lang.Integer);
     0: getstatic     #2  // Field COMPARATOR:Ljava/util/Comparator;
     3: aload_0
     4: aload_1
     5: invokeinterface #44,  3  // InterfaceMethod java/util/Comparator.compare:(Ljava/lang/Object;Ljava/lang/Object;)I
    10: ireturn

The explicit lambda is actually calling lambda$explicit$d34e1a25$1, which in turn calls the COMPARATOR#compare. This layer of indirection means it's not capturing anything that isn't Serializable (or anything at all, to be precise), and so is safe to serialize. The method reference expression directly uses COMPARATOR (the value of which is then passed to the bootstrap method):

  public static Comparator<Integer> reference();
      0: getstatic     #2  // Field COMPARATOR:LComparator;    
      3: dup    
      4: invokevirtual #3   // Method Object.getClass:()LClass;    
      7: pop    
      8: invokedynamic #4,  0  // InvokeDynamic #0:compare:(LComparator;)LComparator;    
      13: checkcast     #5  // class java/io/Serializable    
      16: checkcast     #6  // class Comparator    
      19: areturn

The lack of indirection means that COMPARATOR must be serialized along with the lambda. As COMPARATOR does not refer to a Serializable value, this fails.

The fix

I hesitate to call this a compiler bug (I expect the lack of indirection serves as an optimisation), although it is very strange. The fix is trivial, but ugly; adding the explicit cast for COMPARATOR at declaration:

public static final Comparator<Integer> COMPARATOR = (Serializable & Comparator<Integer>) (a, b) -> a > b ? 1 : -1;

This makes everything perform correctly on Java 1.8.0_45. It's also worth noting that the eclipse compiler produces that layer of indirection in the method reference case as well, and so the original code in this post does not require modification to execute correctly.

Different behavior between lambda expression and method reference by instantiation

The timing of method reference expression evaluation differs from
which of lambda expressions.

With a method reference that has an expression (rather than a type) preceding the :: the subexpression is evaluated immediately and the result of evaluation is stored and reused then.

So here :

new Instance()::set

new Instance() is evaluated a single time.

From 15.12.4. Run-Time Evaluation of Method Invocation (emphasis is mine) :

The timing of method reference expression evaluation is more complex
than that of lambda expressions (§15.27.4). When a method reference
expression has an expression (rather than a type) preceding the ::
separator, that subexpression is evaluated immediately. The result of
evaluation is stored until the method of the corresponding functional
interface type is invoked; at that point, the result is used as the
target reference for the invocation. This means the expression
preceding the :: separator is evaluated only when the program
encounters the method reference expression, and is not re-evaluated on
subsequent invocations on the functional interface type.

Method reference notation vs standard Lambda notation

Here is a quote from the Java language specification:

A method reference expression (§15.13) is potentially compatible with
a functional interface type T if, where the arity of the function type
of T is n, there exists at least one potentially applicable method
when the method reference expression targets the function type with
arity n (§15.13.1), and one of the following is true:

• The method reference expression has the form ReferenceType :: [TypeArguments] Identifier and at least one potentially applicable
  method is either (i) static and supports arity n, or (ii) not static
  and supports arity n-1.

• The method reference expression has some other form and at least one potentially applicable method is not static.

...

The definition of potential applicability goes beyond a basic arity
check to also take into account the presence and "shape" of functional
interface target types.

Every method reference should conform to a function interface (which is an interface that declares a single abstract method, non-overriding methods from Object class). The compiler needs to verify whether the provided reference resolves to a single existing method that has required arity (number of parameters) and their types match the types of the method declared by the target functional interface.

Let's have a look at the prior Java 8 code (code-sample from JavaFX tutorial by created Oracle):

button2.setOnAction(new EventHandler<ActionEvent>() {
    @Override public void handle(ActionEvent e) {
        label.setText("Accepted");
    }
});

That is the "shape" that needs to be filled with a code describing an action. Method handle() of the EventHandler interface expects an event as an argument. Whether it would be used or not, that's up to you, the key point is that the abstract method of the target interface expects this argument to be provided.

By using a lambda expression e -> track.play() you're explicitly telling to ignore it.

And when you're passing a method reference track::play, which should be classified as a reference to an instance method of a particular object (see), the compiler will try to resolve it to a method play(Event) and you're getting a compilation error because it fails to find one.

In this case, reference track::play is not an equivalent of lambda e -> track.play(), but () -> track.play(), which doesn't conform to the target functional interface EventHandler.

In case if you wonder, how a method reference which can be applicable to a non-static method of arity n-1 mentioned in the specification (see case ii) can look like, here is an example:

BiPredicate<String, String> startsWith = String::startsWith; // the same as (str1, str2) -> str1.strarsWith(str2);

System.out.println(startsWith.test("abc", "a"));    // => true
System.out.println(startsWith.test("fooBar", "a")); // => false

And you can construct a similar reference which conforms to EventHandler interface and applicable an instance method of arity n-1 using one of the parameterless methods of the Event type. It's not likely to be useful in practice, but it would be valid from the compiler perspective of view, so feel free to try it as an exercise.

---

Related Topics