How to Read a Large (1 Gb) Txt File in .Net

How to read a large (1 GB) txt file in .NET?

If you are using .NET 4.0, try MemoryMappedFile which is a designed class for this scenario.

You can use StreamReader.ReadLine otherwise.

Reading and writing very large text files in C#

Not sure how much this will improve the performance, but surely, opening and closing the output file for every line that you want to write is not a good idea.

Instead open both files just one time and then write the line directly

using (StreamWriter file = new StreamWriter(@strFullFileName, true, System.Text.Encoding.UTF8))
using (StreamReader srStreamRdr = new StreamReader(strFileName))
{
    while ((strDataLine = srStreamRdr.ReadLine()) != null)
    {
        lngCurrNumRows++;

        if (lngCurrNumRows > 1)
           file.WriteLine(strDataRow);
    }
}

You could also remove the check on lngCurrNumRow simply making an empty read before entering the while loop

strDataLine = srStreamRdr.ReadLine();
if(strDataLine != null)
{
    while ((strDataLine = srStreamRdr.ReadLine()) != null)
    {
           file.WriteLine(strDataRow);
    }
}

Reading a large text file (over 4 million lines) and parsing each line in .NET

You probably get the exception at LogEntries.Add in ProcessLine, because you have so many log entries that this collection gets too large for memory.

So you should store the entries into database immediately without adding them to the list.

But you should read only one line, then process it, then read the next line and forget the previous one. File.ReadAllLines will read all lines at once into a string[] which will occupy the memory(or cause an OutOfMemoryException).

You could use a StreamReader os File.ReadLines instead.

How can I read, replace and write very large files?

There isn't much you can optimize about the I/O, most of the optimization should be on the string comparison to determine if the string should be replaced or not, basically you should do this

protected void ReplaceFile(string FilePath, string NewFilePath)
{
   using (StreamReader vReader = new StreamReader(FilePath))
   {
      using (StreamWriter vWriter = new StreamWriter(NewFilePath))
      {
         int vLineNumber = 0;
         while (!vReader.EndOfStream)
         {
            string vLine = vReader.ReadLine();
            vWriter.WriteLine(ReplaceLine(vLine, vLineNumber++));
         }
      }
   }
}
protected string ReplaceLine(string Line, int LineNumber )
{
   //Do your string replacement and 
   //return either the original string or the modified one
   return Line;
}

What is your criteria to find and replace a string?

How to open a large text file in C#

You can open the file and read it as a stream rather than loading everything into memory all at once.

From MSDN:

using System;
using System.IO;

class Test 
{
    public static void Main() 
    {
        try 
        {
            // Create an instance of StreamReader to read from a file.
            // The using statement also closes the StreamReader.
            using (StreamReader sr = new StreamReader("TestFile.txt")) 
            {
                String line;
                // Read and display lines from the file until the end of 
                // the file is reached.
                while ((line = sr.ReadLine()) != null) 
                {
                    Console.WriteLine(line);
                }
            }
        }
        catch (Exception e) 
        {
            // Let the user know what went wrong.
            Console.WriteLine("The file could not be read:");
            Console.WriteLine(e.Message);
        }
    }
}

How to read very large text file(in gb's) in .net c# and split into small files

Create a StreamReader for the large file and call ReadLine() in a loop.

Maintain a StreamWriter for the current output file.

For each line, check whether it's a header, and, if it is, open a new target file in the StreamWriter. If it isn't, just write that line to the current StreamWriter.

---

Related Topics