Java8: Ambiguity with Lambdas and Overloaded Methods

Java8: ambiguity with lambdas and overloaded methods

There is a lot of complexity at the intersection of overload resolution and type inference. The current draft of the lambda specification has all the gory details. Sections F and G cover overload resolution and type inference, respectively. I don't pretend to understand it all. The summary sections in the introduction are fairly understandable, though, and I recommend that people read them, particularly the summaries of sections F and G, to get an idea of what's going on in this area.

To recap the issues briefly, consider a method call with some arguments in the presence of overloaded methods. Overload resolution has to choose the right method to call. The "shape" of the method (arity, or number of arguments) is most significant; obviously a method call with one argument can't resolve to a method that takes two parameters. But overloaded methods often have the same number of parameters of different types. In this case, the types start to matter.

Suppose there are two overloaded methods:

    void foo(int i);
    void foo(String s);

and some code has the following method call:

    foo("hello");

Obviously this resolves to the second method, based on the type of the argument being passed. But what if we are doing overload resolution, and the argument is a lambda? (Especially one whose types are implicit, that relies on type inference to establish the types.) Recall that a lambda expression's type is inferred from the target type, that is, the type expected in this context. Unfortunately, if we have overloaded methods, we don't have a target type until we've resolved which overloaded method we're going to call. But since we don't yet have a type for the lambda expression, we can't use its type to help us during overload resolution.

Let's look at the example here. Consider interface A and abstract class B as defined in the example. We have class C that contains two overloads, and then some code calls the apply method and passes it a lambda:

    public void apply(A a)    
    public B apply(B b)

    c.apply(x -> System.out.println(x));

Both apply overloads have the same number of parameters. The argument is a lambda, which must match a functional interface. A and B are actual types, so it's manifest that A is a functional interface whereas B is not, therefore the result of overload resolution is apply(A). At this point we now have a target type A for the lambda, and type inference for x proceeds.

Now the variation:

    public void apply(A a)    
    public <T extends B> T apply(T t)

    c.apply(x -> System.out.println(x));

Instead of an actual type, the second overload of apply is a generic type variable T. We haven't done type inference, so we don't take T into account, at least not until after overload resolution has completed. Thus both overloads are still applicable, neither is most specific, and the compiler emits an error that the call is ambiguous.

You might argue that, since we know that T has a type bound of B, which is a class, not a functional interface, the lambda can't possibly apply to this overload, thus it should be ruled out during overload resolution, removing the ambiguity. I'm not the one to have that argument with. :-) This might indeed be a bug in either the compiler or perhaps even in the specification.

I do know that this area went through a bunch of changes during the design of Java 8. Earlier variations did attempt to bring more type checking and inference information into the overload resolution phase, but they were harder to implement, specify, and understand. (Yes, even harder to understand than it is now.) Unfortunately problems kept arising. It was decided to simplify things by reducing the range of things that can be overloaded.

Type inference and overloading are ever in opposition; many languages with type inference from day 1 prohibit overloading (except maybe on arity.) So for constructs like implicit lambdas, which require inference, it seems reasonable to give up something in overloading power to increase the range of cases where implicit lambdas can be used.

-- Brian Goetz, Lambda Expert Group, 9 Aug 2013

(This was quite a controversial decision. Note that there were 116 messages in this thread, and there are several other threads that discuss this issue.)

One of the consequences of this decision was that certain APIs had to be changed to avoid overloading, for example, the Comparator API. Previously, the Comparator.comparing method had four overloads:

    comparing(Function)
    comparing(ToDoubleFunction)
    comparing(ToIntFunction)
    comparing(ToLongFunction)

The problem was that these overloads are differentiated only by the lambda return type, and we actually never quite got the type inference to work here with implicitly-typed lambdas. In order to use these one would always have to cast or supply an explicit type argument for the lambda. These APIs were later changed to:

    comparing(Function)
    comparingDouble(ToDoubleFunction)
    comparingInt(ToIntFunction)
    comparingLong(ToLongFunction)

which is somewhat clumsy, but it's entirely unambiguous. A similar situation occurs with Stream.map, mapToDouble, mapToInt, and mapToLong, and in a few other places around the API.

The bottom line is that getting overload resolution right in the presence of type inference is very difficult in general, and that the language and compiler designers traded away power from overload resolution in order to make type inference work better. For this reason, the Java 8 APIs avoid overloaded methods where implicitly typed lambdas are expected to be used.

Java 8 Lambda Ambiguous with Overloaded Functions

You can just indicate the correct type using a cast like syntax;

// bg.Play(Bounceable)
bg.play((Bounceable) x -> System.out.println("lambda: Ball bouncing "+ x));

// kg.Play(Fly)
kg.play((Fly) x -> System.out.println("lambda: Ball bouncing "+ x));

You can find more info on how this works in this answer.

Method overload ambiguity with Java 8 ternary conditional and unboxed primitives

Let's first consider a simplified version that doesn't have a ternary conditional and doesn't compile on Java HotSpot VM (build 1.8.0_25-b17):

public class Test {

    void f(Object o1, int i) {}
    void f(Object o1, Object o2) {}

    void test() {
        double d = 1.0;

        int i = 1;
        f(d, i); // ERROR!  Ambiguous
    }
}

The compiler error is:

Error:(12, 9) java: reference to f is ambiguous
both method f(java.lang.Object,int) in test.Test and method f(java.lang.Object,java.lang.Object) in test.Test match

According to JLS 15.12.2. Compile-Time Step 2: Determine Method Signature

A method is applicable if it is applicable by one of strict invocation (§15.12.2.2), loose invocation (§15.12.2.3), or variable arity invocation (§15.12.2.4).

Invocation has to do with invocation context which is explained here JLS 5.3. Invocation Contexts

When no boxing or unboxing is involved for a method invocation then strict invocation applies. When boxing or unboxing is involved for a method invocation then loose invocation applies.

Identifying applicable methods is divided into 3 phases.

The first phase (§15.12.2.2) performs overload resolution without permitting boxing or unboxing conversion, or the use of variable arity method invocation. If no applicable method is found during this phase then processing continues to the second phase.

The second phase (§15.12.2.3) performs overload resolution while allowing boxing and unboxing, but still precludes the use of variable arity method invocation. If no applicable method is found during this phase then processing continues to the third phase.

The third phase (§15.12.2.4) allows overloading to be combined with variable arity methods, boxing, and unboxing.

For our case there are no methods applicable by strict invocation. Both methods are applicable by loose invocation since the double value has to be boxed.

According to JLS 15.12.2.5 Choosing the Most Specific Method:

If more than one member method is both accessible and applicable to a
method invocation, it is necessary to choose one to provide the
descriptor for the run-time method dispatch. The Java programming
language uses the rule that the most specific method is chosen.

Then:

One applicable method m1 is more specific than another applicable
method m2, for an invocation with argument expressions e1, ..., ek, if
any of the following are true:

1. m2 is generic, and m1 is inferred to be more specific than m2 for
   argument expressions e1, ..., ek by §18.5.4.

2. m2 is not generic, and m1 and m2 are applicable by strict or loose
   invocation, and where m1 has formal parameter types S1, ..., Sn and m2
   has formal parameter types T1, ..., Tn, the type Si is more specific
   than Ti for argument ei for all i (1 ≤ i ≤ n, n = k).

3. m2 is not generic, and m1 and m2 are applicable by variable arity
   invocation, and where the first k variable arity parameter types of m1
   are S1, ..., Sk and the first k variable arity parameter types of m2
   are T1, ..., Tk, the type Si is more specific than Ti for argument ei
   for all i (1 ≤ i ≤ k). Additionally, if m2 has k+1 parameters, then
   the k+1'th variable arity parameter type of m1 is a subtype of the
   k+1'th variable arity parameter type of m2.

The above conditions are the only circumstances under which one method may be more specific than another.

A type S is more specific than a type T for any expression if S <: T (§4.10).

It may look that the 2nd condition matches for this case but in fact it doesn't because int is not a subtype of Object: it's not true that int <: Object. However if we replace int with Integer in the f method signature this condition would match. Note that the 1st parameter in methods matches this condition since Object <: Object is true.

According to $4.10 no subtype/supertype relation is defined between primitive types and Class/Interface types. So int is not a subtype of Object for example. Thus int is not more specific than Object.

Since among the 2 methods there are no more specific methods thus there can be no strictly more specific and can be no most specific method (the JLS gives definitions for those terms in the same paragraph JLS 15.12.2.5 Choosing the Most Specific Method). So both methods are maximally specific.

In this case the JLS gives 2 options:

If all the maximally specific methods have override-equivalent signatures (§8.4.2) ...

This is not our case, thus

Otherwise, the method invocation is ambiguous, and a compile-time error occurs.

The compile-time error for our case looks valid according to the JLS.

What happens if we change method parameter type from int to Integer?

In this case both methods are still applicable by loose invocation. However the method with Integer parameter is more specific than the method with 2 Object parameters since Integer <: Object. The method with Integer parameter is strictly more specific and most specific thus the compiler will choose it and not throw a compile error.

What happens if we change double to Double in this line: double d = 1.0;?

In this case there is exactly 1 method applicable by strict invocation: no boxing or unboxing is required for invocation of this method: f(Object o1, int i). For the other method you need to do boxing of int value so it's applicable by loose invocation. The compiler can choose the method applicable by strict invocation thus no compiler error is thrown.

As Marco13 pointed out in his comment there is a similar case discussed in this post Why is this method overloading ambiguous?

As explained in the answer there were some major changes related to the method invocation mechanisms between Java 7 and Java 8. This explains why the code compiles in Java 7 but not in Java 8.

---

Now comes the fun part!

Let's add a ternary conditional operator:

public class Test {

    void f(Object o1, int i) {
        System.out.println("1");
    }
    void f(Object o1, Object o2) {
        System.out.println("2");
    }

    void test(boolean b) {
        String s = "string";
        double d = 1.0;
        int i = 1;

        f(b ? s : d, i); // ERROR!  Ambiguous
    }

    public static void main(String[] args) {
        new Test().test(true);
    }
}

The compiler complains about ambiguous method invocation.
The JLS 15.12.2 doesn't dictate any special rules related to ternary conditional operators when performing method invocations.

However there are JLS 15.25 Conditional Operator ? : and JLS 15.25.3. Reference Conditional Expressions. The former one categorizes conditional expressions into 3 subcategories: boolean, numeric and reference conditional expression. The second and third operands of our conditional expression have types String and double respectively. According to the JLS our conditional expression is a reference conditional expression.

Then according to JLS 15.25.3. Reference Conditional Expressions our conditional expression is a poly reference conditional expression since it appears in an invocation context. The type of our poly conditional expression thus is Object (the target type in the invocation context). From here we could continue the steps as if the first parameter is Object in which case the compiler should choose the method with int as the second parameter (and not throw the compiler error).

The tricky part is this note from JLS:

its second and third operand expressions similarly appear in a context of the same kind with target type T.

From this we can assume (also the "poly" in the name implies this) that in the context of method invocation the 2 operands should be considered independently. What this means is that when the compiler has to decide whether a boxing operation is required for such argument it should look into each of the operands and see if a boxing may be required. For our specific case String doesn't require boxing and double will require boxing. Thus the compiler decides that for both overloaded methods it should be a loose method invocation. Further steps are the same as in the case when instead of a ternary conditional expression we use a double value.

From the explanation above it seems that the JLS itself is vague and ambiguous in the part related to conditional expressions when applied to overloaded methods so we had to make some assumptions.

What's interesting is that my IDE (IntelliJ IDEA) doesn't detect the last case (with the ternary conditional expression) as a compiler error. All other cases it detects according to the java compiler from JDK. This means that either JDK java compiler or the internal IDE parser has a bug.

Lambda expression and method overloading doubts

I think you found this bug in the compiler: JDK-8029718 (or this similar one in Eclipse: 434642).

Compare to JLS §15.12.2.1. Identify Potentially Applicable Methods:

…

• A lambda expression (§15.27) is potentially compatible with a functional interface type (§9.8) if all of the following are true:

  • The arity of the target type's function type is the same as the arity of the lambda expression.

  • If the target type's function type has a void return, then the lambda body is either a statement expression (§14.8) or a void-compatible block (§15.27.2).

  • If the target type's function type has a (non-void) return type, then the lambda body is either an expression or a value-compatible block (§15.27.2).

Note the clear distinction between “void compatible blocks” and “value-compatible blocks”. While a block might be both in certain cases, the section §15.27.2. Lambda Body clearly states that an expression like () -> {} is a “void compatible block”, as it completes normally without returning a value. And it should be obvious that i -> {} is a “void compatible block” too.

And according to the section cited above, the combination of a lambda with a block that is not value-compatible and target type with a (non-void) return type is not a potential candidate for the method overload resolution. So your intuition is right, there should be no ambiguity here.

Examples for ambiguous blocks are

() -> { throw new RuntimeException(); }
() -> { while (true); }

as they don’t complete normally, but this is not the case in your question.

Disambiguate overloaded methods that accept different functional interfaces

From the "Effective Java" by Joshua Bloch:

Do not provide a method with multiple overloadings that take different
functional interfaces in the same argument position if it could create
a possible ambiguity in the client.

The easiest way to avoid this problem is not to write overloadings
that take different functional interfaces in the same argument
position.

One more possible solution could be to use a different names for these two methods:

<U> U withTxFunction(Function<OrientGraph, U> fc);

void withTxConsumer(Consumer<OrientGraph> consumer);

A good example of this approach one can find in the Java API itself, for example in the  IntStream interface:

mapToDouble(IntToDoubleFunction mapper);

mapToLong(IntToLongFunction mapper);

Update:

I'd like to add a clarification of why does the compiler complain? That is because...

The lambda g -> anotherMethod(g) can be assigned to both Function<T, R> and Consumer<T>:

Function<T, R> func  = g -> anotherMethod(g);

Consumer<T> consumer = g -> anotherMethod(g); // here you just ignore the return value

<T> T anotherMethod(T t) { ... }

So, when you write withTx(g -> anotherMethod(g)), you get the "Ambiguous method call" error, the compiler fails to find out which of the overloaded method should be used:

withTx(function) OR withTx(consumer) ... ?

Lambda overloaded method

A lambda expression of the form

map -> {
        // whatever
    }

is not compatible with the functional signature. According to your inner class example code, it has to return a boolean value. Hence, it has to be either

map -> {
        // whatever
        return false;
    }

or

map -> /* simple expression, e.g.: */ false

In case of overloaded methods, the compiler uses the shape of a lambda expression, i.e. whether it’s value compatible or void compatible (see JLS §15.27.2), to determine which methods are applicable.

Keep in mind that if the lambda’s body is a block containing conditional code, all code flows which can complete normally must return a value.

Since all methods are inapplicable, the compiler picks one to report an error, according to an algorithm which is independent of the fact that one argument is a lambda expression. It seems that it reports against the method find(String, Object...) because that method can consume most kind of arguments. You could say, that method could consume anything, if there weren’t lambda expressions.

That might be a compiler specific behavior which has been implemented before lambda expressions were to consider. However, that’s only the error reporting behavior, the primary problem is the lambda expression which is not value compatible. Fixing it should solve all problems.

How does Java know which overloaded method to call with lambda expressions? (Supplier, Consumer, Callable, ...)

I believe I have found where this is described in official documentation, although a bit hard to read.

Here is mentioned:

15.27.3. Type of a Lambda Expression

Note that while boxing is not allowed in a strict invocation context,
boxing of lambda result expressions is always allowed - that is, the
result expression appears in an assignment context, regardless of the
context enclosing the lambda expression. However, if an explicitly
typed lambda expression is an argument to an overloaded method, a
method signature that avoids boxing or unboxing the lambda result is
preferred by the most specific check (§15.12.2.5).

and then here (15.12.2.5) is described analytically how the most specific method is chosen.

So according to this for example as described

One applicable method m1 is more specific than another applicable
method m2, for an invocation with argument expressions e1, ..., ek, if
any of the following are true:

m2 is generic, and m1 is inferred to be more specific than m2 for
argument expressions e1, ..., ek

So

// Callable -> why is it not a Supplier?
execute(() -> null);   <-- Callable shall be picked from 2 options as M2 is generic and M1 is inferred to be more specific

void execute(Callable<Void> callable) {  // <------ M1 
   try {
    callable.call();
  } catch (Exception e) {
      e.printStackTrace();
  }
}

 <T> T execute(Supplier<T> supplier) {  // <------ M2 is Generic
    return supplier.get();
 }

Why M1 is inferred to be more specific can be traced down from this process described here (18.5.4 More Specific Method Inference)

Java 8 Consumer/Function Lambda Ambiguity

This line is definitely ambiguous:

doStuff(getPattern(x -> String.valueOf(x)));

Reread this from the linked JLS chapter:

A lambda expression (§15.27) is potentially compatible with a functional interface type (§9.8) if all of the following are true:

• The arity of the target type's function type is the same as the arity of the lambda expression.

• If the target type's function type has a void return, then the lambda body is either a statement expression (§14.8) or a void-compatible block (§15.27.2).

• If the target type's function type has a (non-void) return type, then the lambda body is either an expression or a value-compatible block (§15.27.2).

In your case for Consumer you have a statement expression as any method invocation can be used as statement expression even if the method is non-void. For example, you can simply write this:

public void test(Object x) {
    String.valueOf(x);
}

It makes no sense, but compiles perfectly. Your method may have a side-effect, compiler doesn't know about it. For example, were it List.add which always returns true and nobody cares about its return value.

Of course this lambda also qualifies for Function as it's an expression. Thus it's ambigue. If you have something which is an expression, but not a statement expression, then the call will be mapped to Function without any problem:

doStuff(getPattern(x -> x == null ? "" : String.valueOf(x)));

When you change it to { return String.valueOf(x); }, you create a value-compatible block, so it matches the Function, but it does not qualify as a void-compatible block. However you may have problems with blocks as well:

doStuff(getPattern(x -> {throw new UnsupportedOperationException();}));

This block qualifies both as a value-compatible and a void-compatible, thus you have an ambiguity again. Another ambigue block example is an endless loop:

doStuff(getPattern(x -> {while(true) System.out.println(x);}));

As for System.out.println(x) case it's a little bit tricky. It surely qualifies as statement expression, so can be matched to Consumer, but seems that it matches to expression as well as spec says that method invocation is an expression. However it's an expression of limited use like 15.12.3 says:

If the compile-time declaration is void, then the method invocation must be a top level expression (that is, the Expression in an expression statement or in the ForInit or ForUpdate part of a for statement), or a compile-time error occurs. Such a method invocation produces no value and so must be used only in a situation where a value is not needed.

So compiler perfectly follows the specification. First it determines that your lambda body is qualified both as an expression (even though its return type is void: 15.12.2.1 makes no exception for this case) and a statement expression, so it's considered an ambiguity as well.

Thus for me both statements compile according to the specification. ECJ compiler produces the same error messages on this code.

In general I'd suggest you to avoid overloading your methods when your overloads has the same number of parameters and has the difference only in accepted functional interface. Even if these functional interfaces have different arity (for example, Consumer and BiConsumer): you will have no problems with lambda, but may have problems with method references. Just select different names for your methods in this case (for example, processStuff and consumeStuff).

---

Related Topics