Allocation Rate / GC.GetTotalAllocatedBytes() stops updating

[theolivenbaum]

Description

When monitoring the "Allocation Rate" event counter, we often see the value showing zeros for a very long time in our logs:

And it seems like it is due to the fact that GC.GetTotalAllocatedBytes() is returning a constant value (for example:
1198079260376 on a recent test). This is on a memory intensive application (container with >300GB RAM) with high memory allocation rate.

Any ideas if this is expected behaviour?

Reproduction Steps

N/A

Expected behavior

GC.GetTotalAllocatedBytes() updates with the latest value

Actual behavior

GC.GetTotalAllocatedBytes() stops updating

Regression?

As far as I know, this was not happening on .NET 9.0

Known Workarounds

No response

Configuration

No response

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

[theolivenbaum]

For reference, calling GC.GetTotalAllocatedBytes(true) does return a value that is still changing, and much lower than the value returned by GC.GetTotalAllocatedBytes():

GC.GetTotalAllocatedBytes(true) -> 723,187,864,416
GC.GetTotalAllocatedBytes() -> 1,198,079,260,376

The relevant source code seems to be here.

It doesn't seem to be an overflow issue as the variable used for accumulating the value is uint64_t (i.e. max value of 18,446,744,073,709,551,615)

[jkotas]

Any ideas if this is expected behaviour?

GetTotalAllocatedBytes returns an approximation. The raw approximation is computed without taking locks and thus it can fluctuate up and down. This fluctuation was confusing to people, and we have "fixed" the implementation to never return smaller value than what was returned before:

runtime/src/coreclr/vm/comutilnative.cpp

Lines 977 to 985 in c78f853

	uint64_t current_high = high_watermark;
	while (allocated_bytes > current_high)
	{
	uint64_t orig = InterlockedCompareExchange64((LONG64*)& high_watermark, allocated_bytes, current_high);
	if (orig == current_high)
	return allocated_bytes;
	current_high = orig;
	}

My guess is that the problem you are seeing is caused by a particularly bad high raw approximation that the extra logic in GetTotalAllocatedBytes cached and keeps returning. I expect that if you wait long enough GetTotalAllocatedBytes will start to return changing values again.

[jkotas]

particularly bad high raw approximation

I don't have ideas what can lead to it specifically and whether we can do something about it.

[theolivenbaum]

That does indeed happens, but from the two values above you can see it will take ~500GB of allocations to get back working. In practice it takes a very long time for it to start showing values again, and not so long for it to fail again.

Maybe a call to GC.GetTotalAllocatedBytes(true) should reset the value? This way we could detect this edge case in code and reset it automatically?

Or every N calls to GC.GetTotalAllocatedBytes() the cached value is reset?