且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

查询非常大的.out(平面文件)

更新时间:2023-02-19 22:23:27

根据平面文件的格式,我建议以下其中一项:

1)快速CSV阅读器 [用于平面文件的可移植且高效的通用解析器 [
Depending on the format of the flat file, I would recommend one of the following:

1) A Fast CSV Reader[^] - Handles delimited file formats

2) A Portable and Efficient Generic Parser for Flat Files[^] - Handles delimited and fixed width file formats

Both will allow you to get the data into memory in a quick way. Depending on what you''re doing with it though, you may prefer one of the methods Nijboer has already suggested.


您可以使用LINQ,但首先必须从文件,然后将其解析为列表,然后再使用它.另一个选择是使用SQLServer的批量导入功能,然后以这种方式将数据导入到表中.

http://msdn.microsoft.com/en-us/library/ms187042.aspx [ ^ ]

http://msdn.microsoft.com/en-us/library/ms188365.aspx [ ^ ]

由于记录数量和内存限制,将其读入内存可能是一个问题.当然,这确实取决于行的大小和您实际需要的数据,因为其余部分可以被丢弃.您还可以选择仅将where子句中所需的列保留在内存中,稍后再基于选择返回的记录/行号读取要选择的数据.

祝你好运!
You could use LINQ but you would first have to read the lines from the file and parse them into a list before you could use it. Another option is to use the bulk import function of SQLServer and import the data into a table that way.

http://msdn.microsoft.com/en-us/library/ms187042.aspx[^]

http://msdn.microsoft.com/en-us/library/ms188365.aspx[^]

Reading it into memory could be a problem because of the number of records and memory limitations. This does of course depend on the size of the lines and the data you actually need because the rest can be discarded. You could also choose to only hold in memory the columns you need in your where clause and later on read the data you want to select based on the record/line numbers the select returned.

Good luck!