且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

Python asyncio不显示任何错误

更新时间:2022-06-07 23:27:08

问题是您的consumer仅捕获两个非常具体的异常,在这种情况下,它们将任务标记为已完成.如果发生任何其他异常(例如与网络相关的异常),它将终止使用者.但是,run无法检测到此情况,正在等待queue.join()且消费者(有效)在后台运行.这就是您的程序挂起的原因-永远不会考虑排队的项目,并且永远不会完全处理队列.

The problem is that your consumer catches only two very specific exceptions, and in their case marks the task as done. If any other exception happens, such as a network-related exception, it will terminate the consumer. However, this is not detected by run, which is awaiting queue.join() with the consumer (effectively) running in the background. This is why your program hangs - queued items are never accounted for, and the queue is never fully processed.

有两种方法可以解决此问题,具体取决于程序遇到意外异常时要执行的操作.如果希望它继续运行,则可以向使用者添加一个包罗万象的except子句,例如:

There are two ways to fix this, depending on what you want your program to do when it encounters an unanticipated exception. If you want it to keep running, you can add a catch-all except clause to the consumer, e.g.:

        except Exception as e
            print('other error', e)
            queue.task_done()

替代方法是将未处理消费者异常传播到run.这必须进行显式安排,但是具有永远不允许静默传递异常的优点. (请参阅本文实现此目标的一种方法是同时等待queue.join()和消费者.由于使用者处于无限循环中,因此只有在发生异常的情况下它们才会完成.

The alternative is for an unhandled consumer exception to propagate to run. This must be arranged explicitly, but has the advantage of never allowing exceptions to pass silently. (See this article for a detailed treatment of the subject.) One way to achieve it is to wait for queue.join() and the consumers at the same time; since consumers are in an infinite loop, they will complete only in case of an exception.

    print('[Joining queue]')
    # wait for either `queue.join()` to complete or a consumer to raise
    done, _ = await asyncio.wait([queue.join(), *consumers],
                                 return_when=asyncio.FIRST_COMPLETED)
    consumers_raised = set(done) & set(consumers)
    if consumers_raised:
        await consumers_raised.pop()  # propagate the exception

问题:如何在asyncio中检测和处理异常?

Questions: how to detect and handle exceptions in asyncio?

异常通过await传播,并且通常像其他任何代码一样检测和处理.仅需要特殊处理才能捕获从consumer这样的后台"任务泄漏的异常.

Exceptions are propagated through await and normally detected and handled like in any other code. The special handling is only needed to catch exceptions that leak from a "background" task like the consumer.

如何在不中断队列的情况下重试?

how to retry without disrupting the Queue ?

您可以在except块中调用await queue.put((i, url)).该商品将被添加到队列的后面,供消费者领取.在那种情况下,您只需要第一个代码段,并且不想麻烦尝试将consumer中的异常传播到run.

You can call await queue.put((i, url)) in the except block. The item will be added to the back of the queue, to be picked up by a consumer. In that case you only need the first snippet, and don't want to bother with trying to propagate the exception in consumer to run.