且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

Google ndb 库中的内存泄漏

更新时间:2023-11-29 16:43:52

经过更多的调查,并在 google 工程师的帮助下,我找到了两个解释我的内存消耗的原因.

After more investigations, and with the help of a google engineer, I've found two explanation to my memory consumption.

上下文和线程

ndb.Context 是一个线程本地"对象,只有在线程中有新请求时才会清除.所以线程在请求​​之间保持它.一个 GAE 实例中可能存在许多线程,并且在第二次使用线程并清除上下文之前可能需要数百个请求.
这不是内存泄漏,但内存中的上下文大小可能会超过小型 GAE 实例中的可用内存.

ndb.Context is a "thread local" object and is only cleared when a new request come in the thread. So the thread hold on it between requests. Many threads may exist in a GAE instance and it may take hundreds of requests before a thread is used a second time and it's context cleared.
This is not a memory leak, but contexts size in memory may exceed the available memory in a small GAE instance.

解决方法:
您无法配置 GAE 实例中使用的线程数.因此,***使每个上下文尽可能小.避免上下文缓存,并在每次请求后清除它.

Workaround:
You can not configure the number of threads used in a GAE instance. So it is best to keep each context smallest possible. Avoid in-context cache, and clear it after each request.

事件队列

似乎 NDB 并不能保证在请求后清空事件队列.同样,这不是内存泄漏.但是它将 Futures 留在您的线程上下文中,您又回到了第一个问题.

It seems that NDB does not guarantee that event queue is emptied after a request. Again this is not a memory leak. But it leave Futures in your thread context, and you're back to the first problem.

解决方法:
@ndb.toplevel 包装所有使用 NDB 的代码.

Workaround:
Wrap all your code that use NDB with @ndb.toplevel.