Logstash and Rabbitmq Example
In this tutorial we will discuss about some easy steps of sending log messages to RabbitMQ queue, this is a Part 1 tutorial, In part 2, I will discuss about how to store this messages from rabbitmq to hadoop (hdfs) using logstash-output-webhdfs plugin
Let’s start our easy step tutorial
- Logstash 2.x or higher – Refer my Logstash Installation Tutorial of this
Let’s say we need to read and send catalina (tomcat) messages to rabbitmq queue
Logstash agent configuration (Client)
Create configuration file in – vi /etc/logstash/conf.d/client.conf
path => "/var/log/tomcat/catalina.out"
type => "toodey"
key => "toodeykey"
exchange => "toodeyex"
exchange_type => "direct"
user => "rabbitmquser"
password => "rabbitmqpass"
host => "127.0.0.1"
port => 5672
durable => true
persistent => true
Create queue on RabbitMQ GUI and assigned key and exchange to that queue, logstash need this key and exchange value later for reading logs at server side.
To create queue in RabbitMQ login to RabbitMQ GUI and click on queues tab
Got to exchanges tab to create exchange, enter exchange name and select type direct
Now bind the tomcatqueue with tomcatex for this enter queue name and key name and click on bind button.
On top you will see one diagram created for reference.
Now it’s time to start logstash client, if you have followed my previous Logstash Installation tutorial then it’s just a simple command to start logstash
Type – service logstash start
You will find logs in /var/log/logstash default location.
To see messages immediately into RabbitMQ queue, edit the catalina.out file or wait until this file gets updated.
To edit this file
vi /var/log/tomcat/catalina.out and delete some lines from this file and got to RabbitMQ GUI again, you will see the messages are populated into queue.
In Part 2, we will discuss how to read messages from this rabbitmq queue and store it into Hadoop using logstash-webhdfs-output plugin.