且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

ES索引瘦身 压缩——_source _all 均disable filed store为no,引入第三方DB存储原始数据,去掉pos倒排和doc_values,强制定期merge segments,将所有fileds合并为一个field big string

更新时间:2021-10-09 07:02:00

原始数据:835MB

 

ES 设置了_source _all disabled 且设置了仅仅存docs倒排
Wed Feb 22 11:58:27 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
435 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 11:59:43 CST 2017
done

不做任何压缩:
Wed Feb 22 14:56:04 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1656 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 14:57:54 CST 2017
done
curl -XPOST 'localhost:9200/hec_test3/_forcemerge'
bone@PEK1000074003:~/test_data/splunk_es_cmp$ du -sm ~/elasticsearch-2.4.1/data/
969 /home/bone/elasticsearch-2.4.1/data/


_all disabled:
Wed Feb 22 15:22:09 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1576 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 15:23:25 CST 2017
done
merge后:
787

_source disabled: field store为false 搜索不到东西了 如果不存DB的话
Wed Feb 22 15:26:06 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1288 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 15:27:45 CST 2017
done
merge后:
512


_source disabled: field store为yes
Wed Feb 22 15:33:28 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1583 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 15:34:45 CST 2017
done
merge后:
780


最终设置:
curl -XPUT 'http://localhost:9200/hec_test3' -d '
{
"mappings": {
"hec_type3": {
"_source": {
"enabled": false
},
"_all": {
"enabled": false
},
"properties": {
"event": {
"type": "string",
"include_in_all": false, 
"doc_values": false,
"index_options": "docs"
}
}

}
}
'
压缩配置:
index.codec: best_compression

bash test_es24.sh
Wed Feb 22 15:39:39 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
892 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 15:40:36 CST 2017
done
merge后:
202 MB

50个字段拆分后:
用上述最终设置的效果:
Wed Feb 22 16:02:59 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1423 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 16:05:04 CST 2017
done
merge后:686

不做任何配置:
Wed Feb 22 16:08:30 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
2641 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 16:11:54 CST 2017
done
merge后:
1650














本文转自张昺华-sky博客园博客,原文链接:http://www.cnblogs.com/bonelee/p/6430378.html,如需转载请自行联系原作者