How to read a text file reversely with iterator in C#
Reading text files backwards is really tricky unless you're using a fixed-size encoding (e.g. ASCII). When you've got variable-size encoding (such as UTF-8) you will keep having to check whether you're in the middle of a character or not when you fetch data.
There's nothing built into the framework, and I suspect you'd have to do separate hard coding for each variable-width encoding.
EDIT: This has been somewhat tested - but that's not to say it doesn't still have some subtle bugs around. It uses StreamUtil from MiscUtil, but I've included just the necessary (new) method from there at the bottom. Oh, and it needs refactoring - there's one pretty hefty method, as you'll see:
using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using System.Text;
namespace MiscUtil.IO
{
/// <summary>
/// Takes an encoding (defaulting to UTF-8) and a function which produces a seekable stream
/// (or a filename for convenience) and yields lines from the end of the stream backwards.
/// Only single byte encodings, and UTF-8 and Unicode, are supported. The stream
/// returned by the function must be seekable.
/// </summary>
public sealed class ReverseLineReader : IEnumerable<string>
{
/// <summary>
/// Buffer size to use by default. Classes with internal access can specify
/// a different buffer size - this is useful for testing.
/// </summary>
private const int DefaultBufferSize = 4096;
/// <summary>
/// Means of creating a Stream to read from.
/// </summary>
private readonly Func<Stream> streamSource;
/// <summary>
/// Encoding to use when converting bytes to text
/// </summary>
private readonly Encoding encoding;
/// <summary>
/// Size of buffer (in bytes) to read each time we read from the
/// stream. This must be at least as big as the maximum number of
/// bytes for a single character.
/// </summary>
private readonly int bufferSize;
/// <summary>
/// Function which, when given a position within a file and a byte, states whether
/// or not the byte represents the start of a character.
/// </summary>
private Func<long,byte,bool> characterStartDetector;
/// <summary>
/// Creates a LineReader from a stream source. The delegate is only
/// called when the enumerator is fetched. UTF-8 is used to decode
/// the stream into text.
/// </summary>
/// <param name="streamSource">Data source</param>
public ReverseLineReader(Func<Stream> streamSource)
: this(streamSource, Encoding.UTF8)
{
}
/// <summary>
/// Creates a LineReader from a filename. The file is only opened
/// (or even checked for existence) when the enumerator is fetched.
/// UTF8 is used to decode the file into text.
/// </summary>
/// <param name="filename">File to read from</param>
public ReverseLineReader(string filename)
: this(filename, Encoding.UTF8)
{
}
/// <summary>
/// Creates a LineReader from a filename. The file is only opened
/// (or even checked for existence) when the enumerator is fetched.
/// </summary>
/// <param name="filename">File to read from</param>
/// <param name="encoding">Encoding to use to decode the file into text</param>
public ReverseLineReader(string filename, Encoding encoding)
: this(() => File.OpenRead(filename), encoding)
{
}
/// <summary>
/// Creates a LineReader from a stream source. The delegate is only
/// called when the enumerator is fetched.
/// </summary>
/// <param name="streamSource">Data source</param>
/// <param name="encoding">Encoding to use to decode the stream into text</param>
public ReverseLineReader(Func<Stream> streamSource, Encoding encoding)
: this(streamSource, encoding, DefaultBufferSize)
{
}
internal ReverseLineReader(Func<Stream> streamSource, Encoding encoding, int bufferSize)
{
this.streamSource = streamSource;
this.encoding = encoding;
this.bufferSize = bufferSize;
if (encoding.IsSingleByte)
{
// For a single byte encoding, every byte is the start (and end) of a character
characterStartDetector = (pos, data) => true;
}
else if (encoding is UnicodeEncoding)
{
// For UTF-16, even-numbered positions are the start of a character.
// TODO: This assumes no surrogate pairs. More work required
// to handle that.
characterStartDetector = (pos, data) => (pos & 1) == 0;
}
else if (encoding is UTF8Encoding)
{
// For UTF-8, bytes with the top bit clear or the second bit set are the start of a character
// See http://www.cl.cam.ac.uk/~mgk25/unicode.html
characterStartDetector = (pos, data) => (data & 0x80) == 0 || (data & 0x40) != 0;
}
else
{
throw new ArgumentException("Only single byte, UTF-8 and Unicode encodings are permitted");
}
}
/// <summary>
/// Returns the enumerator reading strings backwards. If this method discovers that
/// the returned stream is either unreadable or unseekable, a NotSupportedException is thrown.
/// </summary>
public IEnumerator<string> GetEnumerator()
{
Stream stream = streamSource();
if (!stream.CanSeek)
{
stream.Dispose();
throw new NotSupportedException("Unable to seek within stream");
}
if (!stream.CanRead)
{
stream.Dispose();
throw new NotSupportedException("Unable to read within stream");
}
return GetEnumeratorImpl(stream);
}
private IEnumerator<string> GetEnumeratorImpl(Stream stream)
{
try
{
long position = stream.Length;
if (encoding is UnicodeEncoding && (position & 1) != 0)
{
throw new InvalidDataException("UTF-16 encoding provided, but stream has odd length.");
}
// Allow up to two bytes for data from the start of the previous
// read which didn't quite make it as full characters
byte[] buffer = new byte[bufferSize + 2];
char[] charBuffer = new char[encoding.GetMaxCharCount(buffer.Length)];
int leftOverData = 0;
String previousEnd = null;
// TextReader doesn't return an empty string if there's line break at the end
// of the data. Therefore we don't return an empty string if it's our *first*
// return.
bool firstYield = true;
// A line-feed at the start of the previous buffer means we need to swallow
// the carriage-return at the end of this buffer - hence this needs declaring
// way up here!
bool swallowCarriageReturn = false;
while (position > 0)
{
int bytesToRead = Math.Min(position > int.MaxValue ? bufferSize : (int)position, bufferSize);
position -= bytesToRead;
stream.Position = position;
StreamUtil.ReadExactly(stream, buffer, bytesToRead);
// If we haven't read a full buffer, but we had bytes left
// over from before, copy them to the end of the buffer
if (leftOverData > 0 && bytesToRead != bufferSize)
{
// Buffer.BlockCopy doesn't document its behaviour with respect
// to overlapping data: we *might* just have read 7 bytes instead of
// 8, and have two bytes to copy...
Array.Copy(buffer, bufferSize, buffer, bytesToRead, leftOverData);
}
// We've now *effectively* read this much data.
bytesToRead += leftOverData;
int firstCharPosition = 0;
while (!characterStartDetector(position + firstCharPosition, buffer[firstCharPosition]))
{
firstCharPosition++;
// Bad UTF-8 sequences could trigger this. For UTF-8 we should always
// see a valid character start in every 3 bytes, and if this is the start of the file
// so we've done a short read, we should have the character start
// somewhere in the usable buffer.
if (firstCharPosition == 3 || firstCharPosition == bytesToRead)
{
throw new InvalidDataException("Invalid UTF-8 data");
}
}
leftOverData = firstCharPosition;
int charsRead = encoding.GetChars(buffer, firstCharPosition, bytesToRead - firstCharPosition, charBuffer, 0);
int endExclusive = charsRead;
for (int i = charsRead - 1; i >= 0; i--)
{
char lookingAt = charBuffer[i];
if (swallowCarriageReturn)
{
swallowCarriageReturn = false;
if (lookingAt == '\r')
{
endExclusive--;
continue;
}
}
// Anything non-line-breaking, just keep looking backwards
if (lookingAt != '\n' && lookingAt != '\r')
{
continue;
}
// End of CRLF? Swallow the preceding CR
if (lookingAt == '\n')
{
swallowCarriageReturn = true;
}
int start = i + 1;
string bufferContents = new string(charBuffer, start, endExclusive - start);
endExclusive = i;
string stringToYield = previousEnd == null ? bufferContents : bufferContents + previousEnd;
if (!firstYield || stringToYield.Length != 0)
{
yield return stringToYield;
}
firstYield = false;
previousEnd = null;
}
previousEnd = endExclusive == 0 ? null : (new string(charBuffer, 0, endExclusive) + previousEnd);
// If we didn't decode the start of the array, put it at the end for next time
if (leftOverData != 0)
{
Buffer.BlockCopy(buffer, 0, buffer, bufferSize, leftOverData);
}
}
if (leftOverData != 0)
{
// At the start of the final buffer, we had the end of another character.
throw new InvalidDataException("Invalid UTF-8 data at start of stream");
}
if (firstYield && string.IsNullOrEmpty(previousEnd))
{
yield break;
}
yield return previousEnd ?? "";
}
finally
{
stream.Dispose();
}
}
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
}
}
// StreamUtil.cs:
public static class StreamUtil
{
public static void ReadExactly(Stream input, byte[] buffer, int bytesToRead)
{
int index = 0;
while (index < bytesToRead)
{
int read = input.Read(buffer, index, bytesToRead - index);
if (read == 0)
{
throw new EndOfStreamException
(String.Format("End of stream reached with {0} byte{1} left to read.",
bytesToRead - index,
bytesToRead - index == 1 ? "s" : ""));
}
index += read;
}
}
}
Feedback very welcome. This was fun :)
Reading large files backwards (from end to start) in C#
Try following code. The last line could be blank. Wasn't sure best way of handling the last line being blank.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
namespace GetFileReverse
{
class Program
{
const string FILENAME = @"c:\temp\test.txt";
static void Main(string[] args)
{
GetFileReverse getFileReverse = new GetFileReverse(FILENAME);
string line = "";
while ((line = getFileReverse.ReadLine()) != null)
{
Console.WriteLine(line);
}
}
}
public class GetFileReverse : IDisposable
{
const int BUFFER_SIZE = 1024;
private FileStream stream { get; set; }
private string data { get; set; }
public Boolean SOF { get; set; }
private long position { get; set; }
public GetFileReverse(string filename)
{
stream = File.OpenRead(filename);
if (stream != null)
{
position = stream.Seek(0, SeekOrigin.End);
SOF = false;
data = string.Empty;
}
else
{
SOF = true;
}
}
private byte[] ReadStream()
{
byte[] bytes = null;
int size = BUFFER_SIZE;
if (position != 0)
{
bytes = new byte[BUFFER_SIZE];
long oldPosition = position;
if (position >= BUFFER_SIZE)
{
position = stream.Seek(-1 * BUFFER_SIZE, SeekOrigin.Current);
}
else
{
position = stream.Seek(-1 * position, SeekOrigin.Current);
size = (int)(oldPosition - position);
bytes = new byte[size];
}
stream.Read(bytes, 0, size);
stream.Seek(-1 * size, SeekOrigin.Current);
}
return bytes;
}
public string ReadLine()
{
string line = "";
while (!SOF && (!data.Contains("\r\n")))
{
byte[] bytes = ReadStream();
if (bytes != null)
{
string temp = Encoding.UTF8.GetString(bytes);
data = data.Insert(0, temp);
}
SOF = position == 0;
}
int lastReturn = data.LastIndexOf("\r\n");
if (lastReturn == -1)
{
if (data.Length > 0)
{
line = data;
data = string.Empty;
}
else
{
line = null;
}
}
else
{
line = data.Substring(lastReturn + 2);
data = data.Remove(lastReturn);
}
return line;
}
public void Close()
{
stream.Close();
}
public void Dispose()
{
stream.Dispose();
data = string.Empty;
position = -1;
}
}
}
How to read a text file from last line to first line in c#
Try to reverse the collection of read lines
List<string> lines = new List<string>();
using (var sr = File.OpenText("C:\\test.txt"))
{
string line;
while ((line = sr.ReadLine()) != null)
{
lines.Add(line);
}
}
lines.Reverse();
Reading a text file from bottom up
public static void ReadFile()
{
List<string> lines = new List<string>();
using (var sr = new System.IO.StreamReader(@"C:\test\tempfile.txt"))
{
string line = null;
while((line = sr.ReadLine()) != null)
{
lines.Insert(0, line);
}
lines.RemoveRange(0, 20);
}
List<string> result = lines.Where((x, i) => i % 2 == 0).ToList();
//Results:
// Fourth-Read:2019/09/24, 12345, abcdefg
// Third-Read:2019/09/24, 12345, abcdefg
// Second-Read:2019/09/24, 12345, abcdefg
// First-Read:2019/09/24, 12345, abcdefg
}
Read from a file starting at the end, similar to tail
To read the last 1024 bytes:
using (var reader = new StreamReader("foo.txt"))
{
if (reader.BaseStream.Length > 1024)
{
reader.BaseStream.Seek(-1024, SeekOrigin.End);
}
string line;
while ((line = reader.ReadLine()) != null)
{
Console.WriteLine(line);
}
}
Reading the last n lines of a file
you can use this code:
List<string> Temp = new List<string>();
int SkipLinesNum = 8;
var GetAllFileToVar = File.ReadLines(@"C:\Sahbak\LinesToList.txt").Skip(SkipLinesNum);
for (int n = 0; n < GetAllFileToVar.Count(); n++)
{
Temp.Add(GetAllFileToVar.ElementAt(0));
}
now you have List of string called "Temp" that contains all lines from n=8.
you can edit you "n" number any time, and also get it from the user.
this is the most efficient method, where you do not load all file to memory.
Related Topics
What Are Some Good .Net Profilers
How to Load Dll (Module Could Not Be Found Hresult: 0X8007007E)
How to Specify a [Dllimport] Path At Runtime
Deserializing Polymorphic Json Classes Without Type Information Using Json.Net
How to Calculate Someone'S Age Based on a Datetime Type Birthday
What Are Good Ways to Prevent SQL Injection
Why Are Mutable Structs "Evil"
Reach Control from Another Page. Asp.Net
How to Dynamically Generate HTML Code Using .Net'S Webbrowser or Mshtml.Htmldocument
How to Handle Both a Single Item and an Array For the Same Property Using Json.Net
Cs0120: an Object Reference Is Required For the Nonstatic Field, Method, or Property 'Foo'
How to Convert HTML to Plain Text