且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

如何扩大/缩小正在运行的 Flink 集群?

更新时间:2023-01-17 23:39:36

默认情况下,Apache Flink 不会根据任务管理器数量的变化重新缩放.

Apache Flink does not, by default, rescale in response to changes in the number of task managers.

关于 Flink 如何在 K8s 环境中重新缩放有多种方案.一种称为主动模式",是 Flink 知道它想要什么资源,并与 K8s 配合以相应地获取/释放资源.请参阅如何在本地部署 FlinkKubernetes 了解详情.

There are various schemes for how Flink rescales in a K8s environment. One, referred to as "active mode", is where Flink knows what resources it wants, and works with K8s to obtain/release resources accordingly. See how to deploy Flink natively on Kubernetes for details.

另一种方案,称为反应模式",正是您所希望的:Flink 扩展自身以使用任何可用的资源.有关详细信息,请参阅弹性缩放.

Another scheme, referred to as "reactive mode", is what you appear to be hoping for: Flink scales itself to use whatever resources have been made available. See Elastic Scaling for details.

您还可以在 Kubernetes 上部署 Flink,这实际上是独立部署的容器化版本.在这样的环境中,重新缩放涉及以下步骤:

You can also deploy Flink on Kubernetes in what is effectively a containerized version of a standalone deployment. In such an environment, rescaling involves these steps:

  1. 在获取保存点时停止作业
  2. 从保存点恢复作业,并为新集群安排了适当的大小

Flink Operations Playground 有一个分步示例,说明如何在基于 docker-compose 的部署中进行重新缩放.

The Flink Operations Playground has a step-by-step example of how to do rescaling in a docker-compose-based deployment.