Outofmemoryexception While Populating Memorystream: 256Mb Allocation on 16Gb System

OutOfMemoryException while populating MemoryStream: 256MB allocation on 16GB system

Short answer - dev server is 32bit process.

Long answer for "why just 256Mb?"

First of all, let's understand how it works.

MemoryStream has internal byte[] buffer to keep all the data. It cannot predict exact size of this buffer, so it just initializes it with some initial value.

Position and Length properties don't reflect actual buffer size - they are logical values to reflect how many bytes is written, and easily may be smaller than actual physical buffer size.

When this internal buffer can not fit all the data, it should be "re-sized", but in real life it means creating new buffer twice as size as previous one, and then copying data from old buffer to new buffer.

So, if the length of your buffer is 256Mb, and you need new data to be written, this means that .Net need to find yet another 512Mb block of data - having all the rest in place, so heap should be at least 768Mb on the moment of memory allocation when you receive OutOfMemory.

Also please notice that by default no single object, including arrays, in .Net can take more than 2Gb in size.

Ok, so here is the sample piece which simulates what's happening:

        byte[] buf = new byte[32768 - 10];

        for (; ; )
        {
            long newSize = (long)buf.Length * 2;
            Console.WriteLine(newSize);

            if (newSize > int.MaxValue)
            {
                Console.WriteLine("Now we reach the max 2Gb per single object, stopping");
                break;
            }

            var newbuf = new byte[newSize];
            Array.Copy(buf, newbuf, buf.Length);
            buf = newbuf;
        }

If it built in x64/AnyCPU and runs from console - everything is ok.

If it built across x86 - it fails in console.

If you put it to say Page_Load, built in x64, and open from VS.Net web server - it fails.

If you do the same with IIS - everything is ok.

Hope this helps.

Reading 200MB file from db throws Out Of Memory Exception

Ok so here's what's happening:

Because this is running on a 32-bit build, the maximum memory allocation is 2GB but I'm still coming nowhere near that threshold.

According to this stackoverflow post that is very similar to my situation, the .NET framework restricts objects to a limit of 256MB in memory.

So even though my file is only 200MB, byte[]s and MemoryStreams expand by powers of 2 until they reach the 256MB necessary. When they expand, they create a new instance of the appropriate size and copy the old data over to the new one, effectively multiplying the memory usage by 3 which causes the exception.

MSDN has an example of how to retrieve a large file using a FileStream, but instead of a FileStream, I use a static byte[] pre-initialized to the size of my data using this post.

Here is my final solution:

    public File GetFileViaFileIdGuid(Guid fileId)
    {
        File file = new File();
        string connectionString = ConfigurationManager.ConnectionStrings["Database"].ConnectionString;
        using (var sourceSqlConnection = new SqlConnection(connectionString))
        {
            using (SqlCommand sqlCommand = sourceSqlConnection.CreateCommand())
            {
                sqlCommand.CommandText = $"SELECT FileName, FileExtension, UploadedDateTime, DATALENGTH(Content) as [ContentLength] FROM dbo.[File] WHERE FileId = '{fileId}'";
                sqlCommand.CommandType = CommandType.Text;
                sqlCommand.CommandTimeout = 300;
                sourceSqlConnection.Open();

                var reader = sqlCommand.ExecuteReader();
                while (reader.Read())
                {
                    file.FileId = fileId;
                    file.FileExtension = reader["FileExtension"].ToString();
                    file.FileName = reader["FileName"].ToString();
                    file.UploadedDateTime = (DateTime)reader["UploadedDateTime"];
                    file.Content = new byte[Convert.ToInt32(reader["ContentLength"])];
                }

                reader.Close();
                sourceSqlConnection.Close();
            }
        }
        file.Content = GetFileContent(file.FileId, file.Content.Length);
        return file;
    }

And to fetch the Content:

    private byte[] GetFileContent(Guid fileId, int contentLength)
    {
        int outputSize = 1048576;
        int bufferSize = contentLength + outputSize;
        byte[] content = new byte[bufferSize];
        string connectionString = ConfigurationManager.ConnectionStrings["Database"].ConnectionString;

        using (SqlConnection sqlConnection = new SqlConnection(connectionString))
        {
            using (SqlCommand sqlCommand = sqlConnection.CreateCommand())
            {
                sqlCommand.CommandTimeout = 300;
                sqlCommand.CommandText = $"SELECT Content FROM dbo.[File] WHERE FileId = '{fileId}'";
                sqlConnection.Open();
                using (SqlDataReader reader = sqlCommand.ExecuteReader(CommandBehavior.SequentialAccess))
                {

                    while (reader.Read())
                    {
                        int startIndex = 0;
                        long returnValue = reader.GetBytes(0, startIndex, content, startIndex, outputSize);
                        while (returnValue == outputSize)
                        {
                            startIndex += outputSize;
                            returnValue = reader.GetBytes(0, startIndex, content, startIndex, outputSize);
                        }
                    }
                }

                sqlConnection.Close();
            }
        }
        return content;
    }

OutOfMemoryException while while opening OracleConnection

I would rewrite the code like this.

.
.
.
OracleConnection test = new OracleConnection();
.
.
.

        using (OracleConnection test = new OracleConnection("Data Source=; User ID=; Password=")) // <-- SQLconnection here
        {

            test.Open(); // <-- Only Open connection

            using (OracleDataReader reader = cmd.ExecuteReader())
            {
.
.
.
            }
         }

How to force an OutOfMemoryException in .Net

One example from MSDN..

The following example illustrates the OutOfMemoryException exception thrown by a call to the StringBuilder.Insert(Int32, String, Int32) method when the example tries to insert a string that would cause the object's Length property to exceed its maximum capacity

using System;
using System.Text;

public class Example
{
   public static void Main()
   {
      StringBuilder sb = new StringBuilder(15, 15);
      sb.Append("Substring #1 ");
      try {
         sb.Insert(0, "Substring #2 ", 1);
      }
      catch (OutOfMemoryException e) {
         Console.WriteLine("Out of Memory: {0}", e.Message);
      }
   }
}
// The example displays the following output:
//    Out of Memory: Insufficient memory to continue the execution of the program.

Further ,it says how to rectify the error.

Replace the call to the StringBuilder.StringBuilder(Int32, Int32) constructor with a call any other StringBuilder constructor overload. The maximum capacity of your StringBuilder object will be set to its default value, which is Int32.MaxValue.

Call the StringBuilder.StringBuilder(Int32, Int32) constructor with a maxCapacity value that is large enough to accommodate any expansions to the StringBuilder object.

C# OutOfMemoryException creating ZipOutputStream using SharpZipLib

I gave up trying to use MemoryStream even though being on a 64bit system with 16gb of memory I should have been safe on that side.

The relevant topic I found was: OutOfMemoryException while populating MemoryStream: 256MB allocation on 16GB system

And using a temporary file to write/read the data instead of memory.

Failed to allocate a managed memory buffer of 268435456 bytes. The amount of available memory may be low

As mentioned before I have changed the TransferMode to "Streamed" but I have also increased the maximum buffer size. Increased the timeout and maxAllowedContent to 300mb to allow larger files to be uploaded. Works great now.

---

Related Topics