By | February 24, 2016

Logstash and Rabbitmq Example

In this tutorial we will discuss about some easy steps of sending log messages to RabbitMQ queue, this is a Part 1 tutorial, In part 2, I will discuss about how to store this messages from rabbitmq to hadoop (hdfs) using logstash-output-webhdfs plugin

Let’s start our easy step tutorial

Prerequisites

  1. Logstash 2.x or higher – Refer my Logstash Installation Tutorial of this
  2. Rabbitmq

 
 

Let’s say we need to read and send catalina (tomcat) messages to rabbitmq queue

Logstash agent configuration (Client)

Create configuration file in – vi /etc/logstash/conf.d/client.conf

input {
file  {
    path => "/var/log/tomcat/catalina.out"
    type => "toodey"
 }
}
filter{
}
output{
      rabbitmq {
          key => "toodeykey"
          exchange => "toodeyex"
          exchange_type => "direct"
          user => "rabbitmquser"
          password => "rabbitmqpass"
          host => "127.0.0.1"
          port => 5672
          durable => true
          persistent => true
      }
}

Create queue on RabbitMQ GUI and assigned key and exchange to that queue, logstash need this key and exchange value later for reading logs at server side.

To create queue in RabbitMQ login to RabbitMQ GUI and click on queues tab

Got to exchanges tab to create exchange, enter exchange name and select type direct

Now bind the tomcatqueue with tomcatex for this enter queue name and key name and click on bind button.

On top you will see one diagram created for reference.

Now it’s time to start logstash client, if you have followed my previous Logstash Installation tutorial then it’s just a simple command to start logstash

Type – service logstash start

You will find logs in /var/log/logstash default location.

To see messages immediately into RabbitMQ queue, edit the catalina.out file or wait until this file gets updated.

To edit this file

vi /var/log/tomcat/catalina.out and delete some lines from this file and got to RabbitMQ GUI again, you will see the messages are populated into queue.

In Part 2, we will discuss how to read messages from this rabbitmq queue and store it into Hadoop using logstash-webhdfs-output plugin.




Leave a Reply

Your email address will not be published. Required fields are marked *