What Is File Hole and How Can It Be Used

what is file hole and how can it be used?

Files with holes are usually referred to as sparse files.

They are useful when a program needs to access a wide range of addresses (offset) but is unlikely to touch all of the potential blocks. This can be used by virtualization products to store virtual disks. Let's say you configure a virtual machine with a 20 GB disk but it won't be full of data quickly. It is much faster to create a 20 GB sparse file that will only use a couple of disk blocks at the beginning and then have the VM creating a file system and storing files at a low pace.

A large sparse file can also have its size reduced when some of its blocks are blanked (i.e. filled with null bytes). The sparse file aware program doing it can, instead of actually writing to the blocks, remove them from the file (i.e. punch holes in the file) with the very same effect because unallocated blocks are returning zeroes when read by a program.

Sparse files are the opposite of preallocation, they are what is called thin provisioning or might also be called disk overcommitment. This allows creating more "virtual disk space" than the actual hardware supports and add more disk to grow the file system only when necessary.

How to create a file with file holes?

Use the dd command with a seek parameter.

dd if=/dev/urandom bs=4096 count=2 of=file_with_holes
dd if=/dev/urandom bs=4096 seek=7 count=2 of=file_with_holes

That creates for you a file with a nice hole from byte 8192 to byte 28671.

Here's an example, demonstrating that indeed the file has holes in it (the ls -s command tells you how many disk blocks are being used by a file):

$ dd if=/dev/urandom bs=4096 count=2 of=fwh # fwh = file with holes
2+0 records in
2+0 records out
8192 bytes (8.2 kB) copied, 0.00195565 s, 4.2 MB/s

$ dd if=/dev/urandom seek=7 bs=4096 count=2 of=fwh
2+0 records in
2+0 records out
8192 bytes (8.2 kB) copied, 0.00152742 s, 5.4 MB/s

$ dd if=/dev/zero bs=4096 count=9 of=fwnh # fwnh = file with no holes
9+0 records in
9+0 records out
36864 bytes (37 kB) copied, 0.000510568 s, 72.2 MB/s

$ ls -ls fw*
16 -rw-rw-r-- 1 hopper hopper 36864 Mar 15 10:25 fwh
36 -rw-rw-r-- 1 hopper hopper 36864 Mar 15 10:29 fwnh

As you can see, the file with holes takes up fewer disk blocks, despite being the same size.

If you want a program that does it, here it is:

#include <unistd.h>
#include <sys/types.h>
#include <stdio.h>
#include <fcntl.h>

int main(int argc, const char *argv[])
{
    char random_garbage[8192]; /* Don't even bother to initialize */
    int fd = -1;
    if (argc < 2) {
        fprintf(stderr, "Usage: %s <filename>\n", argv[0]);
        return 1;
    }
    fd = open(argv[1], O_WRONLY | O_CREAT | O_TRUNC, 0666);
    if (fd < 0) {
        perror("Can't open file: ");
        return 2;
    }
    write(fd, random_garbage, 8192);
    lseek(fd, 5 * 4096, SEEK_CUR);
    write(fd, random_garbage, 8192);
    close(fd);
    return 0;
}

The above should work on any Unix. Someone else replied with a nice alternative method that is very Linux specific. I highlight it here because it's a method distinct from the two I gave, and can be used to put holes in existing files.

How can we know there is hole in a file in C?

"hole" is maybe a misnomer - there's really no literal hole in the file. Let's a assume a "hole" to be an area in a sparse file for which storage has not yet been allocated. There is no standard way (yet) to detect such un-allocated storage in files. For all intents and purposes, the "holes" are indeed part of the file, containing just a lot of zero bytes - These are just not written on disk.

On some systems, the lseek system call may support seek modes named SEEK_HOLE and SEEK_DATA. This is a nonstandard extension that is present in some versions of Linux, BSD, and Solaris.

These position the file pointer to the next hole when it is currently in allocated space (for SEEK_HOLE) or the next "real" data position when in a hole (for SEEK_DATA).

Other systems might support an ioctl named FIEMAP to map logical to physical file positions (which also allows you to detect non-allocated storage).

how file holes in linux actually work

Since "Are you sure that lseek is successful in all calls? You do not check its result code." helped identify a problem I would recomend just add after your filesystem calls:

  int res = lseek(fd, 10000, SEEK_SET);
  if (res == -1) {
    perror("lseek has failed");
    return 1;
  }

You problem is that you use parameters in a wrong order:

lseek(fd, SEEK_SET, 10000); /* WRONG order for second and third parametes ! */

The correct order:

lseek(fd, 10000, SEEK_SET);

Here is a man lseek:

off_t lseek(int fd, off_t offset, int whence);

The lseek() function repositions the file offset of the open file
description associated with the file descriptor fd to the argument
offset according to the directive whence as follows:

SEEK_SET
      The file offset is set to offset bytes.

SEEK_CUR
      The file offset is set to its current location plus offset bytes.

SEEK_END
      The file offset is set to the size of the file plus offset
              bytes.

mmap on file with holes

Holes in files have no effect on any normal file operations, they're just an optimization in the way the file is stored on disk. As far as any operations like reading, seeking, memory mapping, etc. are concerned, it's just a long series of zero bytes. The filesystem driver takes care of turning the hole into a block of zeroes when reading the relevant pages into memory.

For the difference between fallocate() and ftruncate(), see what's the difference between fallocate and ftruncate

File Holes and Shared Memory in Linux?

Can I erase the middle of a file forcing the system to deallocate those blocks on the drive?

You probably want the Linux specific fallocate(2); beware, it might not work on some filesystems (e.g. NFS, VFAT, ...), because some filesystems don't have holes. See also lseek(2) with SEEK_HOLE, posix_fadvise(2), madvise(2), memfd_create(2), etc...

Raw block devices (like a disk partition, or an USB key, or an SSD) don't have any holes (but you could mmap into them). Holes are a file system software artifact.

Would be nice to allocate 1GB shared memory but never utilize it

this is contradictory. If the memory is shared it is used (by the thing -generally another process- with which you share that memory). Read shm_overview(7) if you really want shared memory (and read carefully mmap(2)). Read more about virtual memory, address space, paging, MMUs, page faults, operating systems, kernels, mmap, demand paging, copy-on-write, memory map, ELF, sparse files ... Try also the cat /proc/$$/maps command in a terminal, and understand the output (see proc(5)...).

Perhaps you want to pre-allocate some address space range, and later really allocate the virtual memory. This is possible using Linux version of mmap(2).

To pre-allocate a gigabyte memory range, you'll first call mmap with MAP_NORESERVE

size_t onegiga = 1L<<30;
void* startad = mmap(NULL, onegiga, PROT_NONE, 
                MAP_ANONYMOUS|MAP_NORESERVE|MAP_SHARED,
                -1, 0);
if (startad==MAP_FAILED) { perror("mmap MAP_NORESERVE"); exit(EXIT_FAILURE); }
void* endad = (char*)startad + onegiga;

The MAP_NORESERVE does not consume a lot of resources (i.e. does not eat swap space, which is not reserved, hence the name of the flag). It is pre-allocating address space, in the sense that further mmap calls (without MAP_FIXED) won't give an address inside the returned range (unless you munmap some of it).

Later on, you can allocate some subsegment of that, in multiples of the page size (generally 4Kbytes), using MAP_FIXED inside the previous segment, e.g.

size_t segoff = 1024*1024; // or something else such that ....
assert (segoff >=0  && segoff < onegiga && segoff % sysconf(_SC_PAGESIZE)==0);
size_t segsize = 65536; // or something else such that ....
assert (segsize > 0 && segsize % sysconf(_SC_PAGESIZE)==0 
        && startad + segoffset + segsize < endad);
void* segmentad = mmap(startad + segoffset, segsize,
                       PROT_READ|PROT_WRITE, 
                       MAP_FIXED | MAP_PRIVATE, 
                       -1, 0);
if (segmentad == MAP_FAILED) { perror("mmap MAP_FIXED"); exit(EXIT_FAILURE); }

This re-allocation with MAP_FIXED will use some resources (e.g. consume some swap space).

IIRC, the SBCL runtime and garbage collection uses such tricks.

Read also Advanced Linux Programming and carefully syscalls(2) and the particular man pages of relevant system calls.

Read also about memory overcommitment. This is a Linux feature that I dislike and generally disable (e.g. thru proc(5)).

BTW, the Linux kernel is free software. You can download its source code from kernel.org and study the source code. And you can write some experimental code also. Then ask another more focused question showing your code and the results of your experiments.

What is a sparse file and why do we need it?

Say you have a file with many empty bytes \x00. These many empty bytes \x00 are called holes. Storing empty bytes is just not efficient, we know there are many of them in the file, so why store them on the storage device? We could instead store metadata describing those zeros. When a process reads the file those zero byte blocks get generated dynamically as opposed to being stored on physical storage (look at this schematic from Wikipedia):

This is why a sparse file is efficient, because it does not store the zeros on disk, instead it holds enough data describing the zeros that will be generated.

Note: the logical file size is greater than the physical file size for sparse files. This is because we have not stored the zeros physically on a storage device.

---

Edit:

When you run:

$ dd if=/dev/zero of=output bs=1G count=4

The command here copies 4G blocks of null bytes to output. To see that:

$ stat output
File: ouput
  Size: 4294967296      Blocks: 8388616    IO Block: 4096   regular file
--omitted--

You can see that this file has 8388616 blocks allocated to it, these blocks store nothing but empty bytes copied from /dev/zero and they do occupy physical disk space, they're holes stored on disk (sparse zeros). dd did what you asked for, copying blocks of data from one file to another.

Now, run this command to detect the holes and make the file sparse in-place:

$ fallocate -d output
$ stat output
File: swapfile
  Size: 4294967296      Blocks: 0          IO Block: 4096   regular file
--omitted--

Do you notice something? The the number of blocks now is 0 because the blocks that were storing only empty bytes were de-allocated. Remember, output's blocks store nothing, only a bunch of empty zeros, fallocate -d detected the blocks that contain only empty zeros and deallocated them, since all the blocks for this file contain zeros, they were all de-allocated.

Also notice how the size remained the same. This is the logical (virtual) size of the file, not its size on disk. It's crucial to know that output doesn't occupy physical storage space now, it has 0 blocks allocated to it and thus I doesn't really use disk space. The size preserved after running fallocate -d so when you later read from the file, you get the empty bytes generated to you by the filesystem at runtime. The physical size of output however, is zero, it uses no data blocks.

Remember, when you read output file the empty bytes are generated by the filesystem at runtime dynamically, they're not really physically stored on disk, and the file's size as reported by stat is the logical size, and the physical size is zero for output. In this case the filesystem has to generate 4G of empty bytes when a process reads the file.

To generate a sparse file using dd:

$ dd if=/dev/zero of=output2 bs=1G seek=0 count=0
$ stat 
stat output2
  File: output2
  Size: 4294967296      Blocks: 0          IO Block: 4096   regular file

GNU dd internally uses lseek and ftruncate, so check truncate(2) and lseek(2).

Difference between punch hole and zero range

It depends on what the application wants w.r.t the disk space consumed by a file as a result of using either.

The FALLOC_FL_PUNCH_HOLE flag deallocates blocks. Since it has to be ORed with FALLOC_FL_KEEP_SIZE, what this means is you end up in a sparse file.

FALLOC_FL_ZERO_RANGE on the other hand, allocates blocks for the (offset, length) if not already present and zeroes it out. So in effect you are losing some of its sparseness if the file had holes to begin with. Also, it is a method of zeroing out regions of a file without the application manually having to write(2) zeroes.

All these flags to fallocate(2) are typically used by virtualization software like qemu.

How to create files with holes using lseek?

if( (fd = open( "book.txt", O_RDWR | O_CREAT , S_IWRITE | S_IREAD ) < 0 )) {

This sets fd to 0 if the open succeeds and 1 if it fails. Because you set it to 0, which is your console, that's where it wrote "Harry Potter", rather than to the disk. And you can't lseek on a terminal. You want

if( (fd = open( "book.txt", O_RDWR | O_CREAT , S_IWRITE | S_IREAD )) < 0 ) {

Also

a) There's no need to check that errno != 0 after a system call fails.

b) You should exit upon error rather than falling through.

c) sizeof(char) is always 1 so there's no need to multiply by it.

d) main should have a prototype, e.g., int main(void)

---

Related Topics