且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

如何使用logstash将队列的内容发送到弹性搜索索引

更新时间:2023-02-18 23:23:46

p>很好的开始现在,您只需要输入每个输入,然后将事件转发到给定类型的相应输出,如下所示:

 输入{
rabbitmq {
host => 'rabbit'
durable => true
user => 'user'
queue => 'dev-user_trace'
password => 'pass'
type => 'trace'#< - add this
}
rabbitmq {
host => 'rabbit'
durable => true
user => 'user'
queue => 'min-price-queue'
password => 'pass'
type => 'price'#< - add this
}

}
filter {
}
output {
stdout {codec =&gt ; json}

if [type] =='traces'{#< - - check type
elasticsearch {
hosts => [host1:9200]
index => index1 - %{+ YYYY.MM.dd}
}
}

如果[type] =='price'{#< - check type
elasticsearch {
hosts => [host2:9200]
index => index2 - %{+ YYYY.MM.dd}
}
}
}

更新



以上是最通用的方法,因此您可以不同地配置两种输出。根据@pandaadb的建议,您还可以输入一个输出并定义一个类型,这将成为您的目标索引:

  input { 
rabbitmq {
host => 'rabbit'
durable => true
user => 'user'
queue => 'dev-user_trace'
password => 'pass'
type => 'index1'#< - 添加此
}
rabbitmq {
host => 'rabbit'
durable => true
user => 'user'
queue => 'min-price-queue'
password => 'pass'
type => 'index2'#< - 添加此
}

}
过滤器{
}
输出{
stdout {codec =&gt ; json}

elasticsearch {
hosts => [localhost:9200]
index => %{type} - %{+ YYYY.MM.dd}#< - 使用类型
}
}


I have a logstash up and running that consumes two rabbit queues and sends to an elasticsearch. This is my logstash.conf file:

input {
  rabbitmq {
    host => 'rabbit'
    durable => true
    user => 'user'
    queue => 'dev-user_trace'
    password => 'pass'
  }
  rabbitmq {
    host => 'rabbit'
    durable => true
    user => 'user'
    queue => 'min-price-queue'
    password => 'pass'
  }

}
filter{
}
output{
  stdout { codec => json}
    elasticsearch{
    hosts => ["elasticsearch"]
    index => "eventss-%{+YYYY.MM.dd}"
  }

}

Now I have another queue, but I want to send its content to a different elasticsearch index. My question is: how do I need to redirect specific entries to an specific index? Or do I need another logstash instance?

Thanks in advance.

Very good start. Now you simply need to "type" each input and then forward the events to the appropriate output given its type, like this:

input {
  rabbitmq {
    host => 'rabbit'
    durable => true
    user => 'user'
    queue => 'dev-user_trace'
    password => 'pass'
    type => 'traces'               # <-- add this
  }
  rabbitmq {
    host => 'rabbit'
    durable => true
    user => 'user'
    queue => 'min-price-queue'
    password => 'pass'
    type => 'prices'               # <-- add this
  }

}
filter{
}
output{
  stdout { codec => json}

  if [type] == 'traces' {          # <-- check type
     elasticsearch{
       hosts => ["host1:9200"]
       index => "index1-%{+YYYY.MM.dd}"
     }
  }

  if [type] == 'prices' {          # <-- check type
     elasticsearch{
       hosts => ["host2:9200"]
       index => "index2-%{+YYYY.MM.dd}"
     }
  }
}

UPDATE

The above is the most general approach so that you can configure both outputs differently. As suggested by @pandaadb, you can also have a single output and define a type that would be your target index:

input {
  rabbitmq {
    host => 'rabbit'
    durable => true
    user => 'user'
    queue => 'dev-user_trace'
    password => 'pass'
    type => 'index1'                    # <-- add this
  }
  rabbitmq {
    host => 'rabbit'
    durable => true
    user => 'user'
    queue => 'min-price-queue'
    password => 'pass'
    type => 'index2'                    # <-- add this
  }

}
filter{
}
output{
  stdout { codec => json}

  elasticsearch{
    hosts => ["localhost:9200"]
    index => "%{type}-%{+YYYY.MM.dd}"   # <-- use type here
  }
}