What's the Fastest Way to Read a Text File Line-By-Line

Faster Way to Read File Line by Line?

One possible approach that might be faster would be to read large chunks of the file in with fread(), split it by newlines and then process the lines. You'd have to take in account that the chunks may sever lines and you'd have to detect this and glue them back together.

Generally speaking the larger the chunk you can read in one go the faster your process should become. Within the limits of your available memory.

From fread() docs:

Note that fread() reads from the current position of the file pointer. Use ftell() to find the current position of the pointer and rewind() to rewind the pointer position.

Least Expensive and Fastest Way to Read Large Text Files with Multi line Events

The switch statement with its -File option allows you to read a file's lines one by one, which keeps memory usage low and constant; option -Regex allows the branch handlers to be regular expressions.

$eventLines = ''
switch -Regex -File 'E:\Scripts\Logs\First1000' {
  '^START--' { # Note that the regex is case-INsensitive
    if ($eventLines) { 
        # send previous events' lines to syslog
    }
    # Current line is the start of a new event.
    $eventLines = $_
  }
  default { 
    # Event-interior line, append it.
    $eventLines += [Envirionment]::NewLine + $_
  }
}
# Process last event.
if ($eventLines) { 
    # send last event's lines to syslog
}

Alternatively, you could modify your System.IO.File.ReadLines() solution analogously and use an if statement with the string-literal String.StartsWith() method, which is faster than regex matching - however, in my informal tests the switch solutions still appears to be faster overall.

$eventLines = ''
foreach ($line in [System.IO.File]::ReadLines('E:\Scripts\Logs\First1000')) {
  if ($line.StartsWith('START--')) {
    if ($eventLines) { 
        # send previous events' lines to syslog
    }
    # Current line is the start of a new event.
    $eventLines = $line
  } else {
    # Event-interior line, append it.
    $eventLines += [Environment]::NewLine + $line
  }
}
# Process last event.
if ($eventLines) { 
    # send last event's lines to syslog
}

Fastest way to read a text file of strings line by line

As I understand your question your objective is to read a file of words and insert each word into some data structure. You want this read+insertion to be as fast as possible. (I won't debate the rationale for or the wisdom of this, I'll just accept is as a requirement. :-) )
If my understanding is correct, then perhaps an alternative approach would be to write a utility program that will read the file of words, insert them into the data structure, and then serialize that data structure to a file (say BLOB.dat, for example). Then your main program will deserialize BLOB.dat into the data structure that you require. Essentially you pre-process the words file into some intermediate binary format that can be loaded into your data structure most efficiently. Or would this be cheating in your scenario??

Reading line by line through text file taking abnormally long time

A very quick and easy solution is using StringBuilder type instead of String type for variables line and fullStr. (see https://msdn.microsoft.com/en-us/library/ms172824.aspx).
Strings are immutable, which means every time you assign a value to line or fullStr variables, you are not really updating the value of variable that you have in the memory, instead you scrap the previous allocated memory and allocate a new memory space for the variable and assign the new value to the new memory space. This is a lot of overhead and affects the performance of your application.

---

Related Topics