/Tmp VS. /Dev/Shm for Temp File Storage on Linux

/tmp vs. /dev/shm for temp file storage on Linux?

/dev/shm is intended for a very special purpose, not for files to be put to by arbitrary programs.

In contrast, /tmp is exactly made for this. On my systems, /tmp is a tmpfs as well, in contrast to /var/tmp which is designed for putting larger files, potentially staying longer.

How and when to use /dev/shm for efficiency?

You don't use /dev/shm. It exists so that the POSIX C library can provide shared memory support via the POSIX API. Not so you can poke at stuff in there.

If you want an in-memory filesystem of your very own, you can mount one wherever you want it.

mount -t tmpfs tmpfs /mnt/tmp, for example.

A Linux tmpfs is a temporary filesystem that only exists in RAM. It is implemented by having a file cache without any disk storage behind it. It will write its contents into the swap file under memory pressure. If you didn't want the swapfile you can use a ramfs.

I don't know where you got the idea of using /dev/shm for efficiency in reading files, because that isn't what it does at all.

Maybe you were thinking of using memory mapping, via the mmap system call?

Read this answer here: https://superuser.com/a/1030777/4642 it covers a lot of tmpfs information.

What type of memory objects does `shm_open` use?

Usually, shared memory is implemented using portions of On-Disk files mapped to processes address spaces. Whenever a memory access occurs on the shared region, the filesystem is involved to write changes on the disk which is a great overhead.

That seems rather presumptuous, and not entirely correct. Substantially all machines that implement shared memory regions (in the IPC sense) have virtual memory units by which they support the feature. There may or may not be any persistent storage backing any particular shared memory segment, or any part of it. Only the part, if any, that is paged out needs to be backed by such storage.

shm_open, apparently, works in the same way. It returns a file descriptor which can even be used with regular file operations (e.g ftruncate, ftell, fseek ...etc).

That shm_open() has an interface modeled on that of open(), and that it returns a file descriptor that can meaningfully be used with certain general-purpose I/O function, do not imply that shm_open() "works in the same way" in any broader sense. Pretty much all system resources are represented to processes as files. This affords a simpler overall system interface, but it does not imply any commonality of the underlying resources beyond the fact that they can be manipulated via the same functions -- to the extent that indeed they can be.

So, what is the string parameter passed to shm_open & what does shm_open creates/opens ?

The parameter is a string identifying the shared memory segment. You already knew that, but you seem to think there's more to it than that. There isn't, at least not at the level (POSIX) at which the shm_open interface is specified. The identifier is meaningful primarily to the kernel. Different implementations handle the details differently.

Is it a file on some temporary filesystem (/tmp) which is eventually used by many processes to create the shared region

Could be, but probably isn't. Any filesystem interface provided for it is likely (but not certain) to be a virtual filesystem, not actual, accessible files on disk. Persistent storage, if used, is likely to be provided out of the system's swap space.

(Well, i think it has to be some kind of files since it returns a file descriptor) ?

Such a conclusion is unwarranted. Sockets and pipes are represented via file descriptors, too, but they don't have corresponding accessible files.

Or is it some kind of a mysterious and hidden filesystem backed by the kernel ?

That's probably a better conception, though again, there might not be any persistent storage at all. To the extent that there is any, however, it is likely to be part of the system's swap space, which is not all that mysterious.

Do all tmpfs instances on Linux share same memory pool?

Quotes from man tmpfs.

1. The tmpfs [...] contents reside in virtual memory. Virtual memory is a big topic, you can browse it on the net. The sources for physical memory may be RAM (I don't know if you count separate RAM modules as different source) or from swap or from any other source if you write your driver for it. Virtual memory is build on top of that. The memory can be compressed "on the fly" zswap and have more futures (like removing duplication KSM and so on), so it's not like 5MB in virtual memory is 5MB in physical memory. I guess each tmpfs mountpoint has it's own virtual memory address space, but I think it's up to the implementation really.

2. The [tmpfs] consumes only as much physical memory and swap space as is required to store the current contents of the filesystem..

3. I don't really get that. I don't know what "static allocation" on a tmpfs/virtual memory level means. I guess you can write your own kernel which doesn't add a specified RAM module into the memory pool or creates a memory pool just for that RAM module and write your own kernel driver which uses that specified RAM module for your purposes, which will then simulate tmpfs.

4. -

   4.1. Yes, the tmpfs driver allocates virtual memory from kernel.

   4.2. size=bytes Specify an upper limit on the size of the filesystem.. Only as much as big the resources in tmpfs are. You can optionally specify an upper limit. I guess tmpfs kernel driver also consumes some memory while being loaded, but that's negligible.

5. There is not just one mount point for tmpfs stuff because of the granulation of developers and the software on your computer. If I write an application which will store big chunks of files in memory, I will mount a tmpfs just for my application. If you write another application, you will mount another tmpfs just for your application. That way we won't be writing together to the same folder, we can have same filenames for different applications. But the standard way is to use /tmp directory as the temporary place for files (POSIX.1-2008) and use function like mktemp to create files and folders there, so in reality there are one or two tmpfs mounted on a PC.

6. If you are referring to proc, sysfs, cgroup or like devtmpfs (see udevd) and others, they are not tmpfs and are a separate topic.

better Java IPC@Linux tactic: (a) java.nio File API on /dev/shm or (b) JNI to shmctl(2)?

I vote "NIO and /dev/shm".

But before making any final decisions, you should also consider other options, including CLIP:

• http://ambientideas.com/blog/index.php/tag/java/page/2/

• http://ltsllc.com/talks/20090407_ipc.pdf

• inter jvm communication

Sockets, message queues and named pipes are other IPC methods I wouldn't necessarily dismiss out-of-hand. IMHO...

When I create a Temporary File/Directory, when will it be removed?

If the app didn't clean after itself, the OS will delete the files eventually. It depends on system settings when temp files are deleted. For example, it can happen on boot or nightly (via cron job) or some another way.

See this answer, for example: How is the /tmp directory cleaned up?

---

Related Topics