且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

将大量数据加载到数据表中的有效方法

更新时间:2023-10-31 16:08:10

阅读以下内容: foxpro - 如何在SQL Server中导入DBF文件 - Stack Overflow [ ^ ]


您可以使用数据读取器使用SqlBulkCopy:

  const   string  sourceConnectionString =   abc; 
const string targetConnectionString = xyz;

使用 var sourceConnection = new OleDbConnection(sourceConnectionString))
使用 var targetConnection = new SqlConnection(targetConnectionString))
使用 var sourceCommand = new OleDbCommand( select * from dbftable,sourceConnection))
{
sourceConnection.Open();
targetConnection.Open();

使用 var reader = sourceCommand.ExecuteReader())
使用 var bulkCopy = new SqlBulkCopy (targetConnection))
{
bulkCopy.DestinationTableName = aTableName;
bulkCopy.BatchSize = 1000 ;
bulkCopy.WriteToServer(reader);
}
}


看看这个:

VFP:将VFP数据导入SQL Server [ ^ ]

如何将Visual Foxpro数据库转换为SQL Server数据库 - Stack Overflow [ ^ ]

将DBF文件导入SQL Server [ ^ ]

Hi I am loading DBF file into existing sql table in c# console application. First of all my dbf file having 8 ,50,000+ records i had load the whole DBF file into datatable, It's lead to out of memory exception when processing 400000+ records. So i have read a record at a time and add into the datatable as a row. It's worked fine untill 6,00,000+ records. After this record i am facing System.OutofMemoryException. Please anyone help me to load DBF file into sql table in c# console application. My code is here, I have tried lots of code optimizations but no use

 public void ConvertDBFtoSQL() {
     string connStr = @"Provider=VFPOLEDB.1;Data Source=dbf file path;Persist              Security Info=False; User ID=Admin;Password=;";
     using (OleDbConnection oleDbConnection = new OleDbConnection(connStr))
     {
         oleDbConnection.Open();
         string query = "select * from dbftable";
         OleDbCommand oleDbCommand = new OleDbCommand(query, oleDbConnection);
         oleDbCommand.CommandTimeout = 600000;
         OleDbDataReader oleDbDataReader = oleDbCommand.ExecuteReader();
         int fieldCount = oleDbDataReader.FieldCount;
         DataTable dataTable= CreateDataTable();
         string[] rows = new string[fieldCount];
         while (oleDbDataReader.Read())
         {                     
             using (dataTable)
             {
                 Array.Clear(rows, 0, rows.Length);
                 for (int i = 0; i < fieldCount; i++)
                 {
                     try
                     {
      rows[i] = oleDbDataReader.GetFieldType(i).Name.ToLower() == "datetime"?
"\"" + Convert.ToDateTime(oleDbDataReader.GetValue(i)).ToString("dd/MM/yyyy").Substring(0, 10) + "\"":
                         
                             (oleDbDataReader.GetValue(i).ToString().Contains(",") ? "" :"\"" + oleDbDataReader.GetValue(i).ToString().Trim() + "\"");                                 
                     }
                     catch (Exception ex) { }
                 }
                 string row = string.Join(",", rows);
                 dataTable.Rows.Add(new object[] { row });
             }
         }
     }
}



What I have tried:

First i tried dataTable.Load(oledbcommand.executereader()) it's fail with out of memory exception then i have tried with reading a row and add into a datatable.

Read the following : foxpro - How to import a DBF file in SQL Server - Stack Overflow[^]


You can use data reader with SqlBulkCopy:
const string sourceConnectionString = "abc";
const string targetConnectionString = "xyz";

using (var sourceConnection = new OleDbConnection(sourceConnectionString))
using (var targetConnection = new SqlConnection(targetConnectionString))
using (var sourceCommand = new OleDbCommand("select * from dbftable", sourceConnection))
{
	sourceConnection.Open();
	targetConnection.Open();
	
	using (var reader = sourceCommand.ExecuteReader())
	using (var bulkCopy = new SqlBulkCopy(targetConnection))
	{
		bulkCopy.DestinationTableName = "aTableName";
		bulkCopy.BatchSize = 1000;
		bulkCopy.WriteToServer(reader);
	}
}


Have a look at this:
VFP: Importing VFP data into SQL Server[^]
How to convert Visual Foxpro database into SQL Server database - Stack Overflow[^]
Importing DBF files into SQL Server[^]