且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

使用Python从Amazon S3将大尺寸的压缩JSON文件导入AWS RDS-PostgreSQL

更新时间:2022-11-06 23:03:31

问题是您试图将整个文件读入内存中。时间,如果文件确实太大,可能会导致内存不足。

The problem is that you try to read an entire file into memory at a time, which can cause you to run out of memory if the file is indeed too large.

您应该一次读取一行文件,因为每一行都在文件显然是JSON字符串,您可以直接在循环中处理每一行:

You should read the file one line at a time, and since each line in a file is apparently a JSON string, you can process each line directly in the loop:

with z.open(filename) as f:
    for line in f:
        insert_query(json.loads(line.decode('utf-8')))

您的 insert_query 函数应接受 data 作为参数,方式:

Your insert_query function should accept data as a parameter, by the way:

def insert_query(data):