Memorycache Does Not Obey Memory Limits in Configuration

MemoryCache does not obey memory limits in configuration

Wow, so I just spent entirely too much time digging around in the CLR with reflector, but I think I finally have a good handle on what's going on here.

The settings are being read in correctly, but there seems to be a deep-seated problem in the CLR itself that looks like it will render the memory limit setting essentially useless.

The following code is reflected out of the System.Runtime.Caching DLL, for the CacheMemoryMonitor class (there is a similar class that monitors physical memory and deals with the other setting, but this is the more important one):

protected override int GetCurrentPressure()
{
  int num = GC.CollectionCount(2);
  SRef ref2 = this._sizedRef;
  if ((num != this._gen2Count) && (ref2 != null))
  {
    this._gen2Count = num;
    this._idx ^= 1;
    this._cacheSizeSampleTimes[this._idx] = DateTime.UtcNow;
    this._cacheSizeSamples[this._idx] = ref2.ApproximateSize;
    IMemoryCacheManager manager = s_memoryCacheManager;
    if (manager != null)
    {
      manager.UpdateCacheSize(this._cacheSizeSamples[this._idx], this._memoryCache);
    }
  }
  if (this._memoryLimit <= 0L)
  {
    return 0;
  }
  long num2 = this._cacheSizeSamples[this._idx];
  if (num2 > this._memoryLimit)
  {
    num2 = this._memoryLimit;
  }
  return (int) ((num2 * 100L) / this._memoryLimit);
}

The first thing you might notice is that it doesn't even try to look at the size of the cache until after a Gen2 garbage collection, instead just falling back on the existing stored size value in cacheSizeSamples. So you won't ever be able to hit the target right on, but if the rest worked we would at least get a size measurement before we got in real trouble.

So assuming a Gen2 GC has occurred, we run into problem 2, which is that ref2.ApproximateSize does a horrible job of actually approximating the size of the cache. Slogging through CLR junk I found that this is a System.SizedReference, and this is what it's doing to get the value (IntPtr is a handle to the MemoryCache object itself):

[SecurityCritical]
[MethodImpl(MethodImplOptions.InternalCall)]
private static extern long GetApproximateSizeOfSizedRef(IntPtr h);

I'm assuming that extern declaration means that it goes diving into unmanaged windows land at this point, and I have no idea how to start finding out what it does there. From what I've observed though it does a horrible job of trying to approximate the size of the overall thing.

The third noticeable thing there is the call to manager.UpdateCacheSize which sounds like it should do something. Unfortunately in any normal sample of how this should work s_memoryCacheManager will always be null. The field is set from the public static member ObjectCache.Host. This is exposed for the user to mess with if he so chooses, and I was actually able to make this thing sort of work like it's supposed to by slopping together my own IMemoryCacheManager implementation, setting it to ObjectCache.Host, and then running the sample. At that point though, it seems like you might as well just make your own cache implementation and not even bother with all this stuff, especially since I have no idea if setting your own class to ObjectCache.Host (static, so it affects every one of these that might be out there in process) to measure the cache could mess up other things.

I have to believe that at least part of this (if not a couple parts) is just a straight up bug. It'd be nice to hear from someone at MS what the deal was with this thing.

TLDR version of this giant answer: Assume that CacheMemoryLimitMegabytes is completely busted at this point in time. You can set it to 10 MB, and then proceed to fill up the cache to ~2GB and blow an out of memory exception with no tripping of item removal.

What, exactly, do MemoryCache's memory limits mean?

I know this is late but I've done a lot of digging in the source code to try to understand what is going on and I have a fairly good idea now. I will say that MemoryCache is the worst documented class on MSDN, which kind of baffles me for something intended to be used by people trying to optimize their applications.

MemoryCache uses a special "sized reference" to measure the size of objects. It all looks like a giant hack in the memory cache source code involving reflection to wrap an internal type called "System.SizedReference", which from what I can tell causes the GC to set the size of the object graph it points to during gen 2 collections.

From my testing, this WILL include the size of parent objects, and thus all child objects referenced by the parent etc, BUT I've found that if you make references to parent objects weak references (i.e. via WeakReference or WeakReference<>) then it is no longer counted as part of the object graph, so that is what I do for all cache objects now.

I believe cache objects need to be completely self-contained or use weak references to other objects for the memory limit to work at all.

If you want to play with it yourself, just copy the code from SRef.cs, create an object graph and point a new SRef instance to it, and then call GC.Collect. After the collection the approximate size will be set to the size of the object graph.

Why do I need to call twice the Set on my size limited MemoryCache when I hit the size limit?

I downloaded, built and debugged the unit tests in Microsoft.Extensions.Caching.Memory; there seems to be no test that seems that truly covers this case.

The cause is: as soon as you try to add an item which would make the cache go over capacity, MemoryCache triggers a compaction in the background. This will evict the oldest (MRU) cache entries up until a certain difference. In this case, it tries to remove a total size of 1 of cache items, in your case "first", because that was accessed last.

However, since this compact cycle runs in the background, and the code in the SetEntry() method is already on the code path for a full cache, it continues without adding the item to the cache.

The next time it tries to, it succeeds.

Repro:

class Program
{
    private static MemoryCache _cache;
    private static MemoryCacheEntryOptions _options;

    static void Main(string[] args)
    {
        _cache = new MemoryCache(new MemoryCacheOptions
        {
            SizeLimit = 2
        });

        _options = new MemoryCacheEntryOptions
        {
            Size = 1
        };
        _options.PostEvictionCallbacks.Add(new PostEvictionCallbackRegistration
        {
            EvictionCallback = (key, value, reason, state) =>
            {
                if (reason == EvictionReason.Capacity)
                {
                    Console.WriteLine($"Evicting '{key}' for capacity");
                }
            }
        });
        
        Console.WriteLine(TestCache("first"));
        Console.WriteLine(TestCache("second"));
        Console.WriteLine(TestCache("third")); // starts compaction

        Thread.Sleep(1000);

        Console.WriteLine(TestCache("third"));
        Console.WriteLine(TestCache("third")); // now from cache
    }

    private static object TestCache(string id)
    {
        if (_cache.TryGetValue(id, out var cachedEntry))
        {
            return cachedEntry;
        }

        _cache.Set(id, $"{id} - cached", _options);
        return id;
    }
}

What do the size settings for MemoryCache mean?

I was able to hunt down some helpful documentation.

SizeLimit does not have units. Cached entries must specify size in whatever units they deem most appropriate if the cache size limit has been set. All users of a cache instance should use the same unit system. An entry will not be cached if the sum of the cached entry sizes exceeds the value specified by SizeLimit. If no cache size limit is set, the cache size set on the entry will be ignored.

It turns out that SizeLimit can function as the limit on the number of entries, not the size of those entries.

A quick sample app showed that with a SizeLimit of 1, the following:

var options = new MemoryCacheEntryOptions().SetSize(1);
cache.Set("test1", "12345", options);
cache.Set("test2", "12345", options);

var test1 = (string)cache.Get("test1");
var test2 = (string)cache.Get("test2");

test2 will be null.

In turn, SetSize() allows you to control exactly how much of your size limit an entry should take. For instance, in the following example:

var cache = new MemoryCache(new MemoryCacheOptions
{
    SizeLimit = 3,
});

var options1 = new MemoryCacheEntryOptions().SetSize(1);
var options2 = new MemoryCacheEntryOptions().SetSize(2);
cache.Set("test1", "12345", options1);
cache.Set("test2", "12345", options2);

var test1 = (string)cache.Get("test1");
var test2 = (string)cache.Get("test2");

both test1 and test2 will have been stored in the cache. But if SizeLimit is set to 2, only the first entry will be successfully stored.

Limitation of memory usage in Asp.net Core

This was written a year ago so I'm going to assume you're using v1.x.x of the Microsoft.Extensions.Caching.Memory package.

Since there isn't a SizeLimit property in the MemoryCacheOptions like v2.x.x, after digging around into the code for a while I found the following line of documentation.

https://github.com/aspnet/Caching/blob/rel/1.1.2/src/Microsoft.Extensions.Caching.Memory/MemoryCache.cs#L329

/// This is called after a Gen2 garbage collection. We assume this means there was memory pressure.
/// Remove at least 10% of the total entries (or estimated memory?).

Thus the package will eat up as much memory as the OS will allow your code to have. When it reaches that limit it will start compacting (evicting) cache entries.

With v2.x.x you can set the limit manually using SizeLimit property and you can even set the amount of compaction when the limit is hit CompactionPercentage.

MemoryCache absoluteExpiration and memory limit

Your code is equivalent to calling Add, like below:

cache.Add("cacheItem", testObject, null);

The added entry would have the default expiration time, which is infinite (i.e., it doesn't expire). See the MSDN on CacheItemPolicy.AbsoluteExpiration for details.

To answer the question about memory usage: (from CacheMemoryLimitMegabytes Property):

The default is zero, which indicates that MemoryCache instances manage their own memory based on the amount of memory that is installed on the computer.

I would say that it's safe to let the MemoryCache defaults decide how much memory to use, unless you're doing something really fancy.

---

Related Topics