且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

内存不足读取具有8GB RAM的300MB文件

更新时间:2023-01-11 19:19:05

300 MB可能编码为UTF-8的XML-> 600 MB UTF-16字符串

ReadToEnd在内部使用StringBuilder,它返回sb.ToString().
仅此一项就导致至少1200 MB被分配在一个点上.还有一点缓冲和用于从UTF-8转换为UTF-16的内存.

如果在ASP.Net中执行此操作,则应用程序池通常具有默认的内存限制-通常会导致池被回收.

如果您在32位操作系统下运行,那么这8 GB根本无济于事,即使您在64位操作系统下运行,也必须确保该进程作为64位进程执行.在Visual Studio下运行它默认为32位执行...

我会尝试另一种处理300 MB xml文件的方法.

一个有效的基于SAX的方法可能会更好-我尚未测试过,但这也许值得一试:
建立声明性SAX框架:第1部分-简单的SAX到C#映射 [ ^ ]

***的问候
Espen Harlinn
300 MB XML likely encoding UTF-8 -> 600 MB UTF-16 string

ReadToEnd uses a StringBuilder internally, it returns sb.ToString().
That alone results in at least 1200 MB being allocated at one point. There is also a bit of buffering and the memory used to convert from UTF-8 to UTF-16.

If you are doing this in ASP.Net the application pool has usually a default memory limit - usually resulting in the pool being recycled.

If you are running under a 32-bit os those 8 GB will not help at all, and even if you are running under a 64-bit os, you have to make sure that the process is executed as a 64-bit process. Running it under Visual Studio defaults to 32-bit execution ...

I would try another method of processing a 300 MB xml file.

A working SAX based approach would probably be better - I haven''t tested this, but it is perhaps worth a try:
Towards a Declarative SAX Framework : Part 1 - A Simple SAX-to-C#-Mapping[^]

Best regards
Espen Harlinn


[ ^ ]可以为您提供帮助.
This[^] could help you.