How to Read a Text File Without Locking It

How can I read a text file without locking it?

You need to make sure that both the service and the reader open the log file non-exclusively. Try this:

For the service - the writer in your example - use a FileStream instance created as follows:

var outStream = new FileStream(logfileName, FileMode.Open, 
                               FileAccess.Write, FileShare.ReadWrite);

For the reader use the same but change the file access:

var inStream = new FileStream(logfileName, FileMode.Open, 
                              FileAccess.Read, FileShare.ReadWrite);

Also, since FileStream implements IDisposable make sure that in both cases you consider using a using statement, for example for the writer:

using(var outStream = ...)
{
   // using outStream here
   ...
}

Good luck!

How to read text file without locking the file on disk?

First, if your application is multi-threaded, you shouldn't be using a bool guard. You should be using thread synchronization tools such as locks, mutexes, events and/or semaphores.

Also, your read is opening for share, but your writes aren't.

You are also not wrapping streams in using blocks. This is another problem. You should never do this:

StreamWriter sw = new StreamWriter(fs);

You should do this:

using(var sw = new StreamWriter(fs))
{
    // ...
}

The cardinal rule with objects that implement Dispose is you should always wrap them in a using block.

That aside, you probably don't want to read while writing or write while reading. That is going to give you massive problems with race conditions that will be hard to recreate when you need to debug what is going on.

Since you aren't using async/await, I would suggest using a lock. This will only allow one thread at a time to do file operations. No race conditions, no "sharing" files.

private static readonly object _fileLock = new object();

public static void RemoveCoinFromBuyOrderLogs(string symbol)
{
    lock(_fileLock)
    {
        var newlines = File.ReadAllLines(walletFilename)
            .Where(c =>
                !c.StartsWith(symbol + "USDT") &&
                !c.StartsWith(symbol + "BUSD") &&
                !c.StartsWith(symbol + "USDC") &&
                !c.StartsWith(symbol + "TUSD"));

        File.WriteAllLines(walletFilename, newlines);
    }
}

public static void AddCoinToOrderLogs(string newOrder, long orderId)
{
    lock (_fileLock)
    {
        var lines = File.ReadAllLines(walletFilename).ToList();
        lines = lines.Select(line => line.Replace("\r", "")).ToList();
        lines = lines.Where(line => line != "").Select(line => line).ToList();

        var fields = lines.Select(line => line.Split('\t')).ToList();

        bool duplicate = false;
        foreach (var field in fields)
        {
            if (field.Length >= 5)
            {
                long id = Convert.ToInt64(field[4]);
                if (id == orderId)
                    duplicate = true;
            }
        }

        if (!duplicate)
        {
            lines.Add(newOrder);
            lines.Sort();
            File.WriteAllLines(walletFilename, lines);
        }
    }
}

I can't test this code as I do not have the data to test, but try to get it to look close to that.

And, in all honesty, it is my opinion that you should be using something like an SQLite database to do this type of work. Manipulating a single flat file with multiple threads is a tough thing to do properly and efficiently.

ETA

Here is an example of an async/await pattern using SemaphoreSlim for synchronization

private static readonly SemaphoreSlim _smph = new SemaphoreSlim(1, 1);

private static async Task<IEnumerable<string>> ReadAllLinesAsync(
    string fileName, bool removeEmptyLines = true)
{
    using (var s = File.OpenText(fileName))
    {
        var data = await s.ReadToEndAsync().ConfigureAwait(false);
        return await Task.Run(() =>
            data.Split(new[] { Environment.NewLine },
                removeEmptyLines ? StringSplitOptions.RemoveEmptyEntries : StringSplitOptions.None));
    }
}

private static async Task WriteAllLinesAsync(string fileName, IEnumerable<string> lines)
{
    using (var s = File.OpenWrite(fileName))
    using (var sr = new StreamWriter(s))
    {
        var data = await Task.Run(() => 
            string.Join(Environment.NewLine, lines)).ConfigureAwait(false);
        await sr.WriteAsync(data);
    }
}

public static async Task RemoveCoinFromBuyOrderLogsAsync(string symbol)
{
    await _smph.WaitAsync().ConfigureAwait(false);
    try
    {
        var lines = await ReadAllLinesAsync(walletFilename);
        lines = lines.Where(c =>
                !c.StartsWith(symbol + "USDT") &&
                !c.StartsWith(symbol + "BUSD") &&
                !c.StartsWith(symbol + "USDC") &&
                !c.StartsWith(symbol + "TUSD"));
        await WriteAllLinesAsync(walletFilename, lines);
    }
    finally
    {
        _smph.Release();
    }
}

public static async Task AddCoinToOrderLogsAsync(string newOrder, long orderId)
{
    await _smph.WaitAsync().ConfigureAwait(false);
    try
    {
        var lines = await ReadAllLinesAsync(walletFilename);

        var duplicate = lines.Select(line => line.Split('\t'))
                .Any(x => (x.Length >= 5) && Convert.ToInt64(x[4]) == orderId);

        if (!duplicate)
        {
            var newLines = await Task.Run(() =>
            {
                var newList = lines.ToList();
                newList.Add(newOrder);
                newList.Sort();
                return newList;
            });

            await WriteAllLinesAsync(walletFilename, newLines);
        }
    }
    finally
    {
        _smph.Release();
    }
}

I added Task.Run on parts I thought could be CPU intensive operations.

Reading a file without locking it in Python

Opening a file does not put a lock on it. In fact, if you needed to ensure that separate processes did not access a file simultaneously, all these processes would have to to cooperatively take special steps to ensure that only a single process accessed the file at one time (see Locking a file in Python). This can also be demonstrated by the following small program that purposely takes its time in reading a file to give another process (namely me with a text editor) a chance to append some data to the end of the file while the program is running. This program reads and outputs the file one byte at a time pausing .1 seconds between each read. During the running of the program I added some additional text to the end of the file and the program printed the additional text:

import time

with open('test.txt', "rb") as infile:
    while True:
        data = infile.read(1)
        if data == b'':
            break
        time.sleep(.1)
        print(data.decode('ascii'), end='', flush=True)

You can read your file in pieces and then join these pieces together if you need one single byte string. But this will not be as memory efficient as reading the file with a single read:

BLOCKSIZE = 64*1024 # or some other value depending on the file size
with open(source, "rb") as infile:
    blocks = []
    while True:
        data = infile.read(BLOCKSIZE)
        if data == b'':
            break
        blocks.append(data)
# if you need the data in one piece (otherwise the pieces are in blocks):
data = b''.join(blocks)

Can I prevent a StreamReader from locking a text file whilst it is in use?

You can pass a FileStream to the StreamReader, and create the FileStream with the proper FileShare value. For instance:

using (var file = new FileStream (openFileDialog1.FileName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
using (var reader = new StreamReader (file, Encoding.Unicode)) {
}

File.ReadLines without locking it?

No... If you look with Reflector you'll see that in the end File.ReadLines opens a FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read, 0x1000, FileOptions.SequentialScan);

So Read-only share.

(it technically opens a StreamReader with the FileStream as described above)

I'll add that it seems to be child's play to make a static method to do it:

public static IEnumerable<string> ReadLines(string path)
{
    using (var fs = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite, 0x1000, FileOptions.SequentialScan))
    using (var sr = new StreamReader(fs, Encoding.UTF8))
    {
        string line;
        while ((line = sr.ReadLine()) != null)
        {
            yield return line;
        }
    }
}

This returns an IEnumerable<string> (something better if the file has many thousand of lines and you only need to parse them one at a time). If you need an array, call it as ReadLines("myfile").ToArray() using LINQ.

Please be aware that, logically, if the file changes "behind its back (of the method)", how will everything work is quite undefined (it IS probably technically defined, but the definition is probably quite long and complex)

Java: opening and reading from a file without locking it

Rebuilding tail is tricky due to some special cases like file truncation and (intermediate) deletion. To open the file without locking use StandardOpenOption.READ with the new Java file API like so:

try (InputStream is = Files.newInputStream(path, StandardOpenOption.READ)) {
    InputStreamReader reader = new InputStreamReader(is, fileEncoding);
    BufferedReader lineReader = new BufferedReader(reader);
    // Process all lines.
    String line;
    while ((line = lineReader.readLine()) != null) {
        // Line content content is in variable line.
    }
}

For my attempt to create a tail in Java see:

• Method examineFile(…) in https://github.com/AugustusKling/yield/blob/master/src/main/java/yield/input/file/FileMonitor.java
• The above is used by https://github.com/AugustusKling/yield/blob/master/src/main/java/yield/input/file/FileInput.java to create a tail operation. The queue.feed(lineContent) passes line content for processing by listeners and would equal your this.parse(…).

Feel free to take inspiration from that code or simply copy the parts you require. Let me know if you find any issues that I'm not aware of.

---

Related Topics