Difference Between Posix Aio and Libaio on Linux

Difference between POSIX AIO and libaio on Linux?

On linux, the two AIO implementations are fundamentally different.

The POSIX AIO is a user-level implementation that performs normal blocking I/O in multiple threads, hence giving the illusion that the I/Os are asynchronous. The main reason to do this is that:

1. it works with any filesystem
2. it works (essentially) on any operating system (keep in mind that gnu's libc is portable)
3. it works on files with buffering enabled (i.e. no O_DIRECT flag set)

The main drawback is that your queue depth (i.e. the number of outstanding operations you can have in practice) is limited by the number of threads you choose to have, which also means that a slow operation on one disk may block an operation going to a different disk. It also affects which I/Os (or how many) is seen by the kernel and the disk scheduler as well.

The kernel AIO (i.e. io_submit() et.al.) is kernel support for asynchronous I/O operations, where the io requests are actually queued up in the kernel, sorted by whatever disk scheduler you have, presumably some of them are forwarded (in somewhat optimal order one would hope) to the actual disk as asynchronous operations (using TCQ or NCQ). The main restriction with this approach is that not all filesystems work that well or at all with async I/O (and may fall back to blocking semantics), files have to be opened with O_DIRECT which comes with a whole lot of other restrictions on the I/O requests. If you fail to open your files with O_DIRECT, it may still "work", as in you get the right data back, but it probably isn't done asynchronously, but is falling back to blocking semantics.

Also keep in mind that io_submit() can actually block on the disk under certain circumstances.

Linux aio and Linux native aio are the same thing?

Linux native AIO is kernel AIO. The interfaces are io_setup(2), io_submit(2), io_getevents(2), and a handful of other calls.

When people are talking about "linux AIO", they probably mean "Linux native AIO" - though who knows. This is the Internet, after all.

Posix AIO is something altogether different. It's implemented with threading in user-space. See the top answer to Difference between POSIX AIO and libaio on Linux? for a good, well vetted description of the differences between Posix AIO and Linux kernel AIO.

Linux Asynch IO - difference between aio.h and libaio.h

None of these are really intended for sockets.

The POSIX AIO interface creates threads that use normal blocking IO. They work with the buffer cache and should in principle even work with sockets (though I've admittedly not tried).

The Linux kernel AIO interface does not create threads to handle requests. It works exclusively in "no buffering" mode. Beware of non-obvious behaviour such as blocking when submitting requests in some situations, which you can neither foresee nor prevent (nor know about other than your program acting "weird").

What you want is nonblocking sockets (a nonblocking socket is "kind of asynchronous") and epoll to reduce the overhead of readiness notification to a minimum, and -- if you can figure out the almost non-existing documentation -- splice and vmsplice to reduce the IO overhead. Using splice/vmsplice you can directly DMA from disk to a kernel buffer and push to the network stack from there. Or, you can directly move pages from your application's address space to kernel, and push to the network.

The downside is that the documentation is sparse (to say the least) and especially with TCP, some questions remain unaddressed, e.g. when it is safe to reclaim memory.

What is the status of POSIX asynchronous I/O (AIO)?

Network I/O is not a priority for AIO because everyone writing POSIX network servers uses an event based, non-blocking approach. The old-style Java "billions of blocking threads" approach sucks horribly.

Disk write I/O is already buffered and disk read I/O can be prefetched into buffer using functions like posix_fadvise. That leaves direct, unbuffered disk I/O as the only useful purpose for AIO.

Direct, unbuffered I/O is only really useful for transactional databases, and those tend to write their own threads or processes to manage their disk I/O.

So, at the end that leaves POSIX AIO in the position of not serving any useful purpose. Don't use it.

linux kernel aio functionality

First of all, good job using libaio instead of POSIX aio.

Are there any restrictions on the usage of O_DIRECT ?

I'm not 100% sure this is the real problem, but O_DIRECT has some requirements (quoting mostly from TLPI):

• The data buffer being transferred must be aligned on a memory boundary that is a multiple of the block size (use posix_memalign)
• The file or device offset at which data transfer commences must be a multiple of the block size
• The length of the data to be transferred must be a multiple of the block size

At a glance, I can see you are not taking aby precautions to align memory in allocate_2D_matrix.

If I do not open the file using O_DIRECT, things work fine, but it
beats the purpose of having async writes.

This happens not to be the case. Asynchronous I/O works well without O_DIRECT (for instance think of the number of system calls slashed).

revisiting how do you use aio and epoll together

note you CAN use POSIX aio with epoll, there's signalfd(2) it creates a file descriptor that you can then use to be notified of signals in an epoll based loop.

Also the second aio API is supposed to eventually be what glibc bases it's implementation of POSIX aio on, it's just not quite there yet... (I don't know if anyone is working on it either)

---

Related Topics