How Come a File Doesn't Get Written Until I Stop the Program

How come a file doesn't get written until I stop the program?

Writing to disk is slow, so many programs store up writes into large chunks which they write all-at-once. This is called buffering, and Python does it automatically when you open a file.

When you write to the file, you're actually writing to a "buffer" in memory. When it fills up, Python will automatically write it to disk. You can tell it "write everything in the buffer to disk now" with

f.flush()

This isn't quite the whole story, because the operating system will probably buffer writes as well. You can tell it to write the buffer of the file with

os.fsync(f.fileno())

Finally, you can tell Python not to buffer a particular file with open(f, "w", 0) or only to keep a 1-line buffer with open(f,"w", 1). Naturally, this will slow down all operations on that file, because writes are slow.

File is not written on disk until program ends

If there's some time between the fputs and fclose, add

fflush(fp);

This will cause the contents of the disk file to be written.

CSV file not updating until script is terminated when continuously appending file

The answer is actually pretty simple and easily reproducible. Python does not write to files until you close the file that was opened.

Proof:

with open(file_name, 'w') as file:
    file.write('1')
    input('Press enter to continue')
    file.close()
input('Press enter to continue')
with open(file_name, 'a+') as file:
    file.write('2')
    file.close()    
    input('Press enter to continue')

If you run this, and look at the file at all of the breaks you will find that the file
after the first break is empty, then becomes 1, then becomes 12.

In order to fix this, you should occasionally save the file by closing the opened file with .close() then reopening it.

Edit:

There was an important correction to my answer by martineau. Adding here so that other viewers don't copy my mistake

Python does write to files before they're closed. Files generally have an associated buffer in memory where data goes initially, and when that fills up it's actually written out to the physical file. When a file is closed, this happens before the buffer is full.

In order to clear this buffer, the .flush() function can be called, which will cause the buffer to write to the physical file.

Finally, in my example, I did open the file in append mode on the second call, instead of write mode, which changes the behavior from overwriting a file to appending data to the end.

C# file not being written to until the application finishes running

It seems that the file is being written, the issue is that it wasn't showing up in Windows Explorer.
I stepped through again in debug mode, checked in Explorer, the file wasn't there. I copied the file path from the variable's value and opened that exact path in Notepad++ and the file was there. Following that the file now is showing in Explorer and updating after each test scenario finishes.

Edit:
It seems that the issue was

private string rootFolderPath = Directory.GetParent(@"..\..\..\..") +@"\";

was returning a different result after each scenario to what it was after the test run. One was in the root of my user folder (correct) and the other 3 directories above the Specflow project folder. In the first folder it's only outputting at the end, but in the second it's updating after every scenario.

I get an error that I don't know for what reason. a file disappears when I run the program

I'm guessing that your file exists, but the code that you wrote is reading the file before the changes are "flushed to disk".

Right here:

defer dst.Close()
 _, err = io.Copy(dst, src)

Should Close() or Sync() your writer as soon as possible, otherwise you may read before the write is finished. And since your readFile() function isn't re-using the file, you might as well just close (not Sync()) it immediately, not deferred

Try this:

 _, err = io.Copy(dst, src)
dst.Close()
if err != nil {
  
}

There could be an error while copying, but we still want to Close() the file (if there wasn't an error during the os.Create, os.Open, or os.OpenFile...

---

Related Topics