且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

管道,多路复用和***缓冲

更新时间:2022-03-04 07:04:12

在启动时创建,1000项池,说。在BlockingCollection存储他们 - 一个池队列

Create a pool of items at startup, 1000, say. Store them on a BlockingCollection - a 'pool queue'.

供应商从池队列获取项目,从文件中加载它们,加载在序列号/不管,并将其提交给处理器线程池。

The supplier gets items from the pool queue, loads them from the file, loads in the sequence-number/whatever and submits them to the processors threadpool.

该处理器做自己的东西,并发送输出到多路复用器。多路复用器做它存储了序任何项目,直到较早的项目已处理的工作。

The processors do their stuff and sends the output to the multiplexer. The multiplexer does it job of storing any out-of-order items until earlier items have been processed.

当一个项目已通过任何复用器输出被完全消耗,它们将返回到由供应商重新使用池队列

When an item has been completely consumed by whatever the multiplexer outputs to, they are returned to the pool queue for re-use by the supplier.

如果一个人缓慢的项目'确实需要处理巨额,乱序集合中的多路复用器将在其他池中的线程通过成长为快速项的滑,但由于多路实际上并不是喂养其各色其输出,池队列没有被补充。

If one 'slow item' does require enormous amounts of processing, the out-of-order collection in the multiplexer will grow as the 'quick items' slip through on the other pool threads, but because the multiplexer is not actually feeding its items to its output, the pool queue is not being replenished.

在池清空,供应商将阻止它,将无法提供任何更多的项目。

When the pool empties, the supplier will block on it and will be unable to supply any more items.

快速项目剩余的处理池投入将得到处理,然后将处理停止除了慢件。供应商受阻,多路复用器具有[poolSize-1]其集合中的项目。没有额外的内存被使用,没有CPU是被浪费掉的,发生的唯一的事情就是'慢项目'的处理。

The 'quick items' remaining on the processing pool input will get processed and then processing will stop except for the 'slow item'. The supplier is blocked, the multiplexer has [poolSize-1] items in its collection. No extra memory is being used, no CPU is being wasted, the only thing happening is the processing of the 'slow item'.

在'慢项目终于完成,它会输出到多路复用器。

When the 'slow item' is finally done, it gets output to the multiplexer.

多路复用器现在可以输出所有[poolSize]在规定的顺序项目。由于这些项目被消耗,池被重新填平,供应商,现在可以从池中获取项目,运行在再次阅读了文件中的排队项目到处理器池。

The multiplexer can now output all [poolSize] items in the required sequential order. As these items are consumed, the pool gets filled up again and the supplier, now able to get items from the pool, runs on, again reading its file an queueing up items to the processor pool.

自动调节,不需要界缓冲区,无记忆失控。

Auto-regulating, no bounded buffers required, no memory runaway.

编辑:我的意思是'不要求有界缓冲区:)

I meant 'no bounded buffers required' :)

此外,没有GC含率 - 由于物品被重新使用时,它们并不需要G​​C'ing

Also, no GC holdups - since the items are re-used, they don't need GC'ing.