更新时间:2023-11-21 23:07:40
好的.如果您想简单地从 GCS 读取文件,而不是作为 PCollection 而是作为常规文件,并且如果您在使用 GCS Java 客户端库时遇到问题,您还可以使用 Apache Beam 文件系统 API:
OK. If you want to simply read files from GCS, not as a PCollection but as regular files, and if you are having trouble with the GCS Java client libraries, you can also use the Apache Beam FileSystems API:
首先,您需要确保在 beam-sdks-java-extensions-google-cloud-platform-core
上的 pom.xml
中有一个 Maven 依赖项> 其中包含 gs://
文件系统的实现:
First, you need to make sure that you have a Maven dependency in your pom.xml
on beam-sdks-java-extensions-google-cloud-platform-core
which contains implementation of the gs://
filesystem:
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-extensions-google-cloud-platform-core</artifactId>
</dependency>
然后设置 FileSystems API(默认设置在所有管道中,但如果您在管道外使用它,则需要手动完成).
Then set up the FileSystems API (it is set up by default in all pipelines, but if you're using it outside a pipeline, you need to do it manually).
PipelineOptions options = PipelineOptionsFactory.create();
// ...Optionally fill in options such as GCP credentials...
// (see GcpOptions class)
FileSystems.setDefaultPipelineOptions(options);
然后就可以使用了:
ReadableByteChannel chan = FileSystems.open(FileSystems.matchNewResource(
"gs://path/to/your/file", false /* is_directory */));
try (InputStream stream = Channels.newInputStream(chan)) {
// Use regular Java utilities to work with the input stream.
}