且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

如何配置flink来理解Azure Data Lake文件系统?

更新时间:2023-11-29 13:42:46

Flink 能够连接到任何 Hadoop 兼容的文件系统(即实现 org.apache.hadoop.fs.FileSystem).请参阅此处的说明:https://ci.apache.org/projects/flink/flink-docs-release-0.8/example_connectors.html

Flink has the capability to connect to any Hadoop compatible file system (i.e that implements org.apache.hadoop.fs.FileSystem). See here for the explanation: https://ci.apache.org/projects/flink/flink-docs-release-0.8/example_connectors.html

在 core-site.xml 中,您应该添加特定于 ADLS 的配置.无论 Flink 代理在哪里运行,您还需要在类路径中使用 ADL jar.

In the core-site.xml, you should add the ADLS-specific configuration. You will also need the ADL jars in the class path whereever the Flink agents run.

除了适用于 Flink 之外,它与本博客中概述的概念基本相同.https://medium.com/azure-data-lake/connecting-your-own-hadoop-or-spark-to-azure-data-lake-store-93d426d6a5f4

It's basically the same concept as outlined in this blog, except adapted to Flink. https://medium.com/azure-data-lake/connecting-your-own-hadoop-or-spark-to-azure-data-lake-store-93d426d6a5f4