When trying to allocate a huge amount of memory, our production system throws OutOfMemoryException. Well, there is not enough memory available, that's the reason. But what I don't understand is the folowing:
Using System.Diagnostics.PerformanceCounter("Memory", "Available MBytes") we see that the system still has over 600 MB available. However, when the system tries to allocate about 300 MB byte[], it gets OutOfMemoryException. I would expect, that 700MB of available memory means that the system is able to allocate almost all of these 700 MB in one continuous block. And if it is not able right now, it will be able after garbage collection finishes.
It seems to me that the garbage collector needs twice as much memory for its work than it allows to allocate. Is it so?
> When trying to allocate a huge amount of memory, our production system > throws OutOfMemoryException. Well, there is not enough memory available, > that's the reason. But what I don't understand is the folowing:
> Using System.Diagnostics.PerformanceCounter("Memory", "Available MBytes") we > see that the system still has over 600 MB available. However, when the > system tries to allocate about 300 MB byte[], it gets OutOfMemoryException. > I would expect, that 700MB of available memory means that the system is able > to allocate almost all of these 700 MB in one continuous block.
That's an incorrect expectation.
> And if it is > not able right now, it will be able after garbage collection finishes.
> It seems to me that the garbage collector needs twice as much memory for its > work than it allows to allocate. Is it so?
No. But it's not nearly as simple as you seem to think it is either.
First, "available MBytes" refers to physical RAM, which has practically nothing to do with how much memory your process can allocate. What's important to your own process is _virtual_ address space, and available disk space.
Within your process's virtual address space, you need a _contiguous_ block of address space large enough to contain your allocation. In addition, on the hard disk (where the virtual memory manager's swap file resides) you need enough (not necessarily contiguous) space to contain your allocation.
Most likely in your case, you are running out of virtual address space. You may in fact even have more than 300MB of virtual address space free, but due to fragmentation, no single contiguous block available that large.
The fact is, 300MB is a pretty large data structure for a 32-bit application. Except under the most ideal circumstances, I would expect an allocation that large to succeed once, rarely twice, at the most. Even under the best circumstances, you're not likely to get more than four successful allocations that size, and at that point, very little else will work.
This might be a checkerboading scenario. You're asking for one chunk of 300MB, but that won't work if the 600MB available is in (say) 600 chunks of 1MB. 700MB available does not mean it's all in one huge available chunk. -- Phil Wilson The Definitive Guide to Windows Installer http://www.apress.com/book/view/1590592972
"Martin Plechsmid" <S...@No.Mail> wrote in message
> When trying to allocate a huge amount of memory, our production system > throws OutOfMemoryException. Well, there is not enough memory available, > that's the reason. But what I don't understand is the folowing:
> Using System.Diagnostics.PerformanceCounter("Memory", "Available MBytes") > we see that the system still has over 600 MB available. However, when the > system tries to allocate about 300 MB byte[], it gets > OutOfMemoryException. I would expect, that 700MB of available memory means > that the system is able to allocate almost all of these 700 MB in one > continuous block. And if it is not able right now, it will be able after > garbage collection finishes.
> It seems to me that the garbage collector needs twice as much memory for > its work than it allows to allocate. Is it so?
> This might be a checkerboading scenario. You're asking for one chunk of > 300MB, but that won't work if the 600MB available is in (say) 600 chunks > of 1MB. 700MB available does not mean it's all in one huge available > chunk.
Yes, of course. But after garbage collection all separated chunks should join in one contiguous area, shouldn't they? Some GC algorithms work this way, but I don't know whether .NET is this case.
> First, "available MBytes" refers to physical RAM, which has practically > nothing to do with how much memory your process can allocate. What's > important to your own process is _virtual_ address space, and available > disk space.
I expect that the free virtual address space always contains free RAM. Thus 700MB of available RAM should mean at least 700MB of available virtual address space - correct?
> Within your process's virtual address space, you need a _contiguous_ block > of address space large enough to contain your allocation.
Yes, I know that. But I expect that garbage collection joins separated pieces of memory into one contiguous area. Maybe this is the incorrect expectation?
> Most likely in your case, you are running out of virtual address space. > You may in fact even have more than 300MB of virtual address space free, > but due to fragmentation, no single contiguous block available that large.
The production server has 8GB of physical RAM, and it is not used to its limits. I don't think that lack of memory resources is the cause, but I'll check it again.
> The fact is, 300MB is a pretty large data structure for a 32-bit > application. Except under the most ideal circumstances, I would expect an > allocation that large to succeed once, rarely twice, at the most. Even > under the best circumstances, you're not likely to get more than four > successful allocations that size, and at that point, very little else will > work.
According to our logs, at most one such huge structure is created at a time, but even then it has very short life. I.e. minutes before the structure is allocated again, and seconds before it's given available for disposal.
Martin Plechsmid wrote: >> First, "available MBytes" refers to physical RAM, which has practically >> nothing to do with how much memory your process can allocate. What's >> important to your own process is _virtual_ address space, and available >> disk space.
> I expect that the free virtual address space always contains free RAM. Thus > 700MB of available RAM should mean at least 700MB of available virtual > address space - correct?
No. Not even close to correct. A process's virtual address space has nothing to do with the amount of physical RAM available. There can be no physical RAM free, and yet plenty of virtual address space available to a process, and vice a versa.
Remember: physical RAM is a global resource, shared by all processes running on the computer, where that sharing is managed by the virtual memory manager. But to each process, the only thing they see is the _virtual_ address space, which is completely independent of the physical RAM. That's why it's called "_virtual_".
>> Within your process's virtual address space, you need a _contiguous_ block >> of address space large enough to contain your allocation.
> Yes, I know that. But I expect that garbage collection joins separated > pieces of memory into one contiguous area. Maybe this is the incorrect > expectation?
Firstly: for objects in the small object heap, garbage collection _sometimes_ will defragment the heap, by compacting it. But the large object heap is never compacted, and thus can become fragmented.
Secondly: the .NET heap compaction strategy has very little to do with compaction of the used virtual address space itself. The .NET heap is allocated from the virtual address space, but is managed separately from that. The .NET heap can be perfectly compacted, and yet your process can still have fragmentation in the virtual address space.
>> Most likely in your case, you are running out of virtual address space. >> You may in fact even have more than 300MB of virtual address space free, >> but due to fragmentation, no single contiguous block available that large.
> The production server has 8GB of physical RAM, and it is not used to its > limits. I don't think that lack of memory resources is the cause, but I'll > check it again.
The amount of physical RAM is irrelevant. The amount of disk space is barely relevant. From your description, everything suggests that it's the process's virtual address space that is getting fragmented, and no amount of free physical RAM or disk space will help that.
>> The fact is, 300MB is a pretty large data structure for a 32-bit >> application. Except under the most ideal circumstances, I would expect an >> allocation that large to succeed once, rarely twice, at the most. Even >> under the best circumstances, you're not likely to get more than four >> successful allocations that size, and at that point, very little else will >> work.
> According to our logs, at most one such huge structure is created at a time, > but even then it has very short life. I.e. minutes before the structure is > allocated again, and seconds before it's given available for disposal.
I think you misunderstand me: I mean "at most" during the entire lifetime of the process. If you're lucky, you won't have any new allocations in the virtual address space between the time that the large data structure is allocated and when it's freed. But most applications don't work that way, and it would be very easy for a single allocation of a single structure of that size to result in enough subsequent fragmentation to prevent future allocations of that size.
So far, you've said nothing at all about why you're allocating something this large, or why you're trying to do so in a 32-bit process. So all I can tell you is that the best answer to your problem is "don't do that". Either change your application so that it's a 64-bit application, or change your algorithm so you don't need a single 300MB data structure.
And in any case, you need to rid yourself of your incorrect assumptions about how memory management works. They are leading you down fruitless paths as you search for a solution to your problem.
>> I expect that the free virtual address space always contains free RAM. >> Thus 700MB of available RAM should mean at least 700MB of available >> virtual address space - correct?
> No. Not even close to correct. A process's virtual address space has > nothing to do with the amount of physical RAM available. There can be no > physical RAM free, and yet plenty of virtual address space available to a > process, and vice a versa.
Aha! I thought that the PerformanceCounter was giving me information about the memory of the .NET's GC heap (which is a part of the process memory)! Not an info about the system memory!
>>> Within your process's virtual address space, you need a _contiguous_ >>> block of address space large enough to contain your allocation.
>> Yes, I know that. But I expect that garbage collection joins separated >> pieces of memory into one contiguous area. Maybe this is the incorrect >> expectation?
And I thought that the GC pre-allocates a _contiguous_ block of memory from the system virtual memory, which it subsequently lends to managed objects. This was my other misunderstanding. I thought that the performance counter shows the amount of this pre-allocated memory available for .NET heap (thus part of the process memory).
> Firstly: for objects in the small object heap, garbage collection > _sometimes_ will defragment the heap, by compacting it. But the large > object heap is never compacted, and thus can become fragmented.
Some GC algorithms (e.g. the basic copying GC) always defragment the heap. But I do not know which algorithm is used by .NET. And if - as you say - the heap memory need not be contiguous, this can influence its fragmentation too. I believe you are right that the .NET GC algorithm need not defragment the heap.
> So far, you've said nothing at all about why you're allocating something > this large, or why you're trying to do so in a 32-bit process. So all I > can tell you is that the best answer to your problem is "don't do that". > Either change your application so that it's a 64-bit application, or > change your algorithm so you don't need a single 300MB data structure.
I simplified the problem to get answers to what I needed to learn (i.e. answers on memory management). The original code used the MemoryStream. The buffer of the stream was failing to extend when the buffer grew over certain size (140MB). MemoryStream internally uses set_Capacity() which in turn creates a new bigger (twice as big, i.e. 280MB) contiguous buffer. I already knew that we must get rid of allocating so huge memory stream. But I didn't understand the maths. Now I know we were comparing wrong numbers.
Martin Plechsmid wrote: >>> I expect that the free virtual address space always contains free RAM. >>> Thus 700MB of available RAM should mean at least 700MB of available >>> virtual address space - correct? >> No. Not even close to correct. A process's virtual address space has >> nothing to do with the amount of physical RAM available. There can be no >> physical RAM free, and yet plenty of virtual address space available to a >> process, and vice a versa.
> Aha! I thought that the PerformanceCounter was giving me information about > the memory of the .NET's GC heap (which is a part of the process memory)! > Not an info about the system memory!
It's documented to provide information about available physical memory. Physical memory is relevant only to the OS; normal processes never "see" it (obviously they use it, but only via virtual addresses).
> And I thought that the GC pre-allocates a _contiguous_ block of memory from > the system virtual memory, which it subsequently lends to managed objects. [...]
It does. It doesn't have to ask the OS for more memory for every managed allocation. But that's not the entire story.
> [...] >> Firstly: for objects in the small object heap, garbage collection >> _sometimes_ will defragment the heap, by compacting it. But the large >> object heap is never compacted, and thus can become fragmented.
> Some GC algorithms (e.g. the basic copying GC) always defragment the heap.
The .NET GC algorithm is more complex. It's main optimization is that it's a generational collector, and older generations in the small object heap not subject to a given collection operation are also not defragmented.
Furthermore (and this is what's important here, because you're dealing with a large object): the large object heap is separate from the small object heap, and it's _never_ defragmented.
> But I do not know which algorithm is used by .NET. And if - as you say - the > heap memory need not be contiguous, this can influence its fragmentation > too. I believe you are right that the .NET GC algorithm need not defragment > the heap. [...]
Allocations made from the OS to increase the heap size allocate single blocks at a time. Obviously, within each block, the available memory addresses are contiguous. There can be fragmentation temporarily in the small object heap, and in the large object heap, any fragmentation that occurs will persist until the objects that are in the middle of free space are themselves released. No data in the large object heap will be compacted.
For all intents and purposes, you can ignore the small object heap and its behavior. Even though it does get compacted on a regular basis, that doesn't matter. Your data isn't being allocated there. It's being allocated in a heap that never is compacted and so, depending on your exact allocation patterns, can become fragmented.
> Martin Plechsmid wrote: >>>> I expect that the free virtual address space always contains free RAM. >>>> Thus 700MB of available RAM should mean at least 700MB of available >>>> virtual address space - correct? >>> No. Not even close to correct. A process's virtual address space has >>> nothing to do with the amount of physical RAM available. There can be >>> no physical RAM free, and yet plenty of virtual address space available >>> to a process, and vice a versa.
>> Aha! I thought that the PerformanceCounter was giving me information >> about the memory of the .NET's GC heap (which is a part of the process >> memory)! Not an info about the system memory!
> It's documented to provide information about available physical memory. > Physical memory is relevant only to the OS; normal processes never "see" > it (obviously they use it, but only via virtual addresses).
>> And I thought that the GC pre-allocates a _contiguous_ block of memory >> from the system virtual memory, which it subsequently lends to managed >> objects. [...]
> It does. It doesn't have to ask the OS for more memory for every managed > allocation. But that's not the entire story.
>> [...] >>> Firstly: for objects in the small object heap, garbage collection >>> _sometimes_ will defragment the heap, by compacting it. But the large >>> object heap is never compacted, and thus can become fragmented.
>> Some GC algorithms (e.g. the basic copying GC) always defragment the >> heap.
> The .NET GC algorithm is more complex. It's main optimization is that > it's a generational collector, and older generations in the small object > heap not subject to a given collection operation are also not > defragmented.
> Furthermore (and this is what's important here, because you're dealing > with a large object): the large object heap is separate from the small > object heap, and it's _never_ defragmented.
>> But I do not know which algorithm is used by .NET. And if - as you say - >> the heap memory need not be contiguous, this can influence its >> fragmentation too. I believe you are right that the .NET GC algorithm >> need not defragment the heap. [...]
> Allocations made from the OS to increase the heap size allocate single > blocks at a time. Obviously, within each block, the available memory > addresses are contiguous. There can be fragmentation temporarily in the > small object heap, and in the large object heap, any fragmentation that > occurs will persist until the objects that are in the middle of free space > are themselves released. No data in the large object heap will be > compacted.
> For all intents and purposes, you can ignore the small object heap and its > behavior. Even though it does get compacted on a regular basis, that > doesn't matter. Your data isn't being allocated there. It's being > allocated in a heap that never is compacted and so, depending on your > exact allocation patterns, can become fragmented.
in addition to what others have already said, I can confirm that .NET memory management is not a trivial topic. If you need to access large amounts of memory, you might try to use memory mapped files. A while a go I wrote an article about this to Developer.com, search for it or let me know and I can share a link.
Additionally, I can share links to some good discussion about the issue you are seeing. Browse through these, and let us know if you need further assistance.
> in addition to what others have already said, I can confirm that .NET > memory management is not a trivial topic. If you need to access large > amounts of memory, you might try to use memory mapped files. A while a go > I wrote an article about this to Developer.com, search for it or let me > know and I can share a link.
> Additionally, I can share links to some good discussion about the issue > you are seeing. Browse through these, and let us know if you need further > assistance.
You're assuming that the Performance counter you're using is returning .NET memory. You need .NET counters for .NET memory. Run Perfmon, and look at the .NET CLR Memory counters, such as "Bytes in all Heaps". -- Phil Wilson The Definitive Guide to Windows Installer http://www.apress.com/book/view/1590592972
"Martin Plechsmid" <Do....@Mail.Me> wrote in message
>> This might be a checkerboading scenario. You're asking for one chunk of >> 300MB, but that won't work if the 600MB available is in (say) 600 chunks >> of 1MB. 700MB available does not mean it's all in one huge available >> chunk.
> Yes, of course. But after garbage collection all separated chunks should > join in one contiguous area, shouldn't they? Some GC algorithms work this > way, but I don't know whether .NET is this case.
|
