Thread Safety of Mpi Send Using Threads Created with Std::Async

thread safety of MPI send using threads created with std::async

Thread-safety in MPI doesn't work out of the box. First, you have to ensure that your implementation actually supports multiple threads making MPI calls at once. With some MPI implementations, for example Open MPI, this requires the library to be configured with special options at build time. Then you have to tell MPI to initialise at the appropriate thread support level. Currently the MPI standard defines four levels of thread support:

• MPI_THREAD_SINGLE - means that the user code is single threaded. This is the default level at which MPI is initialised if MPI_Init() is used;
• MPI_THREAD_FUNNELED - means that the user code is multithreaded, but only the main thread makes MPI calls. The main thread is the one which initialises the MPI library;
• MPI_THREAD_SERIALIZED - means that the user code is multithreaded, but calls to the MPI library are serialised;
• MPI_THREAD_MULTIPLE - means that the user code is multithreaded and all threads can make MPI calls at any time with no synchronisation whatsoever.

In order to initialise MPI with thread support, one has to use MPI_Init_thread() instead of MPI_Init():

int provided;

MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided);
if (provided < MPI_THREAD_MULTIPLE)
{
    printf("ERROR: The MPI library does not have full thread support\n");
    MPI_Abort(MPI_COMM_WORLD, 1);
}

Equivalent code with the obsoleted (and removed from MPI-3) C++ bindings:

int provided = MPI::Init_thread(argc, argv, MPI::THREAD_MULTIPLE);
if (provided < MPI::THREAD_MULTIPLE)
{
    printf("ERROR: The MPI library does not have full thread support\n");
    MPI::COMM_WORLD.Abort(1);
}

Thread support levels are ordered like this: MPI_THREAD_SINGLE < MPI_THREAD_FUNNELED < MPI_THREAD_SERIALIZED < MPI_THREAD_MULTIPLE, so any other provided level, different from MPI_THREAD_MULTIPLE would have lower numerical value - that's why the if (...) code above is written so.

MPI_Init(&argc, &argv) is equivalent to MPI_Init_thread(&argc, &argv, MPI_THREAD_SINGLE, &provided). Implementations are not required to initialise exactly at the requested level - rather they could initialise at any other level (higher or lower), which is returned in the provided output argument.

For more information - see §12.4 of the MPI standard, freely available here.

With most MPI implementations, the thread support at level MPI_THREAD_SINGLE is actually equivalent to that provided at level MPI_THREAD_SERIALIZED - exactly what you observe in your case.

Since you've not specified which MPI implementation you use, here comes a handy list.

I've already said that Open MPI has to be compiled with the proper flags enabled in order to support MPI_THREAD_MULTIPLE. But there is another catch - its InfiniBand component is not thread-safe and hence Open MPI would not use native InfiniBand communication when initialised at full thread support level.

Intel MPI comes in two different flavours - one with and one without support for full multithreading. Multithreaded support is enabled by passing the -mt_mpi option to the MPI compiler wrapper which enables linking with the MT version. This option is also implied if OpenMP support or the autoparalleliser is enabled. I am not aware how the InfiniBand driver in IMPI works when full thread support is enabled.

MPICH(2) does not support InfiniBand, hence it is thread-safe and probably most recent versions provide MPI_THREAD_MULTIPLE support out of the box.

MVAPICH is the basis on which Intel MPI is built and it supports InfiniBand. I have no idea how it behaves at full thread support level when used on a machine with InfiniBand.

The note about multithreaded InfiniBand support is important since lot of compute clusters nowadays use InfiniBand fabrics. With the IB component (openib BTL in Open MPI) disabled, most MPI implementations switch to another protocol, for example TCP/IP (tcp BTL in Open MPI), which results in much slower and more latent communication.

Is it safe to call MPI_Init from different thread?

The MPI standard has a section about MPI and Threads. One important part is:

Initialization and Completion The call to MPI_FINALIZE should occur on the same thread that initialized MPI. We call this thread the main thread. The call should occur only after
all process threads have completed their MPI calls, and have no pending communications or I/O operations.

Your code does fulfill this requirement.

MPI knows several levels of thread support. Your code requires:

MPI_THREAD_FUNNELED The process may be multi-threaded, but the application must ensure that only the main thread makes MPI calls.

For good measure, you should call the following instead of MPI_Init:

int provided;
MPI_Init_thread(NULL, NULL, MPI_THREAD_FUNNELED, &provided);
if (provided < MPI_THREAD_FUNNELED)
    MPI_Abort(MPI_COMM_WORLD, -1);

A MPI library that is not thread compliant must return provided == MPI_THREAD_SINGLE indicating you won't be able to use this implementation correctly with your code.

In practice, you should be fine with the common implementations. For more information about the different levels of thread support, see 12.4.3 in the MPI standard. With the higher levels of thread support, your mileage may vary depending on the implementation you chose.

Calling MPI functions from multiple threads

(Since you haven't provided an example, the following is just speculation.)

You must initialize MPI using MPI_Init_thread() instead of MPI_Init(). If I understand your explanation correctly, the "required" argument must have the value MPI_THREAD_MULTIPLE. If MPI_Init_thread() then returns a lower level thread support in the "provided" argument, it means that your MPI implementation doesn't support MPI_THREAD_MULTIPLE; in that case you must do something else. See http://www.mpi-forum.org/docs/mpi-20-html/node165.htm .

Safety guarantee for asynchronous sends in MPI

Your code is perfectly safe. This is guaranteed by the semantics of the non-blocking operations as defined in the MPI standard §3.7.4 - Semantics of Nonblocking Communications:

Progress A call to MPI_WAIT that completes a receive will eventually terminate and return if a matching send has been started, unless the send is satised by another receive. In particular, if the matching send is nonblocking, then the receive should complete even if no call is executed by the sender to complete the send. Similarly, a call to MPI_WAIT that completes a send will eventually return if a matching receive has been started, unless the receive is satised by another send, and even if no call is executed to complete the receive.

A blocking operation in that context is equivalent to initiating a non-blocking one, immediately followed by a wait.

If the words of the standard are not reassuring enough, then this code section from the implementation of MPI_SENDRECV in Open MPI might help:

if (source != MPI_PROC_NULL) { /* post recv */
    rc = MCA_PML_CALL(irecv(recvbuf, recvcount, recvtype,
                            source, recvtag, comm, &req));
    OMPI_ERRHANDLER_CHECK(rc, comm, rc, FUNC_NAME);
}

if (dest != MPI_PROC_NULL) { /* send */
    rc = MCA_PML_CALL(send(sendbuf, sendcount, sendtype, dest,
                           sendtag, MCA_PML_BASE_SEND_STANDARD, comm));
    OMPI_ERRHANDLER_CHECK(rc, comm, rc, FUNC_NAME);
}

if (source != MPI_PROC_NULL) { /* wait for recv */
    rc = ompi_request_wait(&req, status);
} else {
    if (MPI_STATUS_IGNORE != status) {
        *status = ompi_request_empty.req_status;
    }
    rc = MPI_SUCCESS;
}

It doesn't matter if you use Irecv / Send / Wait(receive) or Isend / Recv / Wait(send) - both are equally safe when it comes to possible deadlocks. Of course deadlocks could (and will) occur if the interleaved operation is not properly matched.

The only thing that brings your code to non-conformance is the fact that it uses the C++ MPI bindings. Those were deprecated in MPI-2.2 and deleted in MPI-3.0. You should use the C API instead.

Safety guarantee for interleaved MPI Isend / Recv

Question: Can I safely assume that most implementations of MPI don't choke up on this piece of code?

In practice - yes, if you add synchronisation (which your code is lacking); in theory - no. While it is possible that some implementations allow for serialised calls from different threads at the MPI_THREAD_SINGLE level (with Open MPI being such one - see here), the MPI standard requires that the library must be initialised at the MPI_THREAD_SERIALIZED level. If you intent for your software to be portable and to be able to compile and run correctly with other MPI implementations, you should not rely on some particular Open MPI behaviour.

That said, Open MPI can be configured to support multithreading (MPI_THREAD_MULTIPLE) when the library is built. The default is that MT support is not enabled for performance reasons. You can check the state of your particular installation using ompi_info:

$ ompi_info | grep MPI_THREAD_MULTIPLE
     Thread support: poxis (MPI_THREAD_MULTIPLE: no, progress: no)
                            ^^^^^^^^^^^^^^^^^^^^^^^

That particular build does not support multithreading and will always return MPI_THREAD_SINGLE in the provided output argument of MPI_Init_thread.

Invisible runtime errors in threads

I found out that these runtime-errors are displayed in the syslog files, that completely fulfills my requirements.

---

Related Topics