flume打包遇到的一些问题
1.ipc兼容性问题,线上使用2.3.0的hdfs,但是打包时默认为1.2.1的
1
2
3
4
|
08 Apr 2015 19:38:25,122 WARN [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:455) - HDFS IO error org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4 at org.apache.hadoop.ipc.Client.call(Client.java:1113)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
|
拷贝hadoop-core包即可,或者更改pom.xml文件
1
2
3
4
5
6
|
< properties >
< hadoop.version >1.2.1</ hadoop.version > //hadoop-core-1.2.1.jar 改为hadoop-core-2.3.0-mr1-cdh5.1.0.jar
< hbase.version >0.92.1</ hbase.version >
< hadoop.common.artifact.id >hadoop-core</ hadoop.common.artifact.id >
< thrift.version >0.7.0</ thrift.version >
</ properties >
|
或者打包时指定:
1
|
mvn clean install -Phadoop-2 |
2.没有跳过test时
1
|
org.apache.flume.auth.TestFlumeAuthenticator: org/apache/commons/io/Charsets (no class error) |
根目录下的pom.xml文件中更改为:
1
2
3
4
5
|
< dependency >
< groupId >commons-io</ groupId >
< artifactId >commons-io</ artifactId >
< version >2.4</ version > //2.1的包没有Charsets这个类
</ dependency >
|
如果手动下载jar包可以使用如下命令导入:
1
|
mvn install:install-file -DgroupId=commons-io -DartifactId=commons-io -Dversion=2.0.1 -Dpackaging=jar -Dfile=commons-io-2.4.jar |
3. 程序语言问题,因为在系统中使用了中文环境导致编译不通过
1)
1
2
3
4
5
6
|
@Test public void shouldUseUtcAsBasisForDateFormat() {
assertEquals("Coordinated Universal Time",
factory.fastDateFormat.getTimeZone().getDisplayName()); //请求英文,返回中文
}
... |
2)
1
2
3
4
5
6
|
Running org.apache.flume.source.twitter.TestTwitterSource Tests run: 2, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 0.247 sec <<< FAILURE! testCarrotDateFormatBug(org.apache.flume.source.twitter.TestTwitterSource) Time elapsed: 21 sec <<< ERROR! java.text.ParseException: Unparseable date: "Fri Oct 26 22:53:55 +0000 2012" at java.text.DateFormat.parse(DateFormat.java:357)
at org.apache.flume.source.twitter.TestTwitterSource.testCarrotDateFormatBug(TestTwitterSource.java:109)
|
代码:
1
2
3
4
5
|
@Test public void testCarrotDateFormatBug() throws Exception {
SimpleDateFormat formatterFrom = new SimpleDateFormat("EEE MMM dd HH:mm:ss Z yyyy"); //更改为new SimpleDateFormat("EEE MMM dd HH:mm:ss Z yyyy",Locale.US);
formatterFrom.parse("Fri Oct 26 22:53:55 +0000 2012");
}
|
或者简单点,更改语言设置或者直接skip test
1
|
mvn clean install -Phadoop-2 -DskipTests |
4.json问题
替换单个jar包时,因为依赖问题,包class not found:
1
2
3
4
5
|
Exception in thread "PollableSourceRunner-KafkaSource-kafka1" java.lang.NoClassDefFoundError: net/sf/ezmorph/Morpher at org.apache.flume.source.kafka.KafkaSourceUtil.getDateMessage(KafkaSourceUtil.java:117)
at org.apache.flume.source.kafka.KafkaSource.process(KafkaSource.java:123)
at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139)
at java.lang.Thread.run(Thread.java:745)
|
可以打成ensmbly包
1
2
3
4
5
6
7
|
flume-1.6.0/flume-ng-sources/flume-kafka-source/pom.xml < dependency >
< groupId >net.sf.json-lib</ groupId >
< artifactId >json-lib</ artifactId >
< version >2.4</ version >
< classifier >jdk15</ classifier >
</ dependency >
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
< plugin >
< artifactId >maven-assembly-plugin</ artifactId >
< version >2.4</ version >
< configuration >
< descriptorRefs >
< descriptorRef >jar-with-dependencies</ descriptorRef >
</ descriptorRefs >
</ configuration >
< executions >
< execution >
< id >make-assembly</ id >
< phase >package</ phase >
< goals >
< goal >single</ goal >
</ goals >
</ execution >
</ executions >
</ plugin >
|
或者拷贝jar包:
1
2
3
4
5
6
|
/Users/nizengguang/.m2/repository/net/sf/ezmorph/ezmorph/1.0.6/ezmorph-1.0.6.jar /Users/nizengguang/.m2/repository/net/sf/json-lib/json-lib/2.4/json-lib-2.4-jdk15.jar /Users/nizengguang/.m2/repository/commons-beanutils/commons-beanutils/1.8.0/commons-beanutils-1.8.0.jar /Users/nizengguang/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar /Users/nizengguang/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar /Users/nizengguang/.m2/repository/commons-lang/commons-lang/2.5/commons-lang-2.5.jar |