且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

在 Apache Beam 中从 GCS 读取文件

更新时间:2023-11-21 23:07:40

好的.如果您想简单地从 GCS 读取文件,而不是作为 PCollection 而是作为常规文件,并且如果您在使用 GCS Java 客户端库时遇到问题,您还可以使用 Apache Beam 文件系统 API:

OK. If you want to simply read files from GCS, not as a PCollection but as regular files, and if you are having trouble with the GCS Java client libraries, you can also use the Apache Beam FileSystems API:

首先,您需要确保在 beam-sdks-java-extensions-google-cloud-platform-core 上的 pom.xml 中有一个 Maven 依赖项> 其中包含 gs:// 文件系统的实现:

First, you need to make sure that you have a Maven dependency in your pom.xml on beam-sdks-java-extensions-google-cloud-platform-core which contains implementation of the gs:// filesystem:

<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-extensions-google-cloud-platform-core</artifactId>
</dependency>

然后设置 FileSystems API(默认设置在所有管道中,但如果您在管道外使用它,则需要手动完成).

Then set up the FileSystems API (it is set up by default in all pipelines, but if you're using it outside a pipeline, you need to do it manually).

PipelineOptions options = PipelineOptionsFactory.create();
// ...Optionally fill in options such as GCP credentials...
// (see GcpOptions class)
FileSystems.setDefaultPipelineOptions(options);

然后就可以使用了:

ReadableByteChannel chan = FileSystems.open(FileSystems.matchNewResource(
  "gs://path/to/your/file", false /* is_directory */));
try (InputStream stream = Channels.newInputStream(chan)) {
  // Use regular Java utilities to work with the input stream.
}