By | July 21, 2015

 Log analysis using Logstash ElasticSearch and Kibana

Introduction

Consider 192.168.120.158, 192.168.120.159, 192.168.120.160 is my 3 node cluster in above case.

192.168.120.158 – Server Machine Ubuntu 10.04LTS.
192.168.120.159 – Client Machine Ubuntu 10.04LTS.
192.168.120.160 – Client Machine Ubuntu 10.04LTS.

Architecture

 

log ana 1

Prerequisites

Now, let’s collect the require software materials for our log analysis process.

192.168.120.158 (Server Machine) :

  • Elasticsearch
  • Kibana 3
  • Redis
  • Apache Tomcat (For Kibana deployment)
  • Logstash Jar (Logstash Indexer Conf. file)
  • Java 6 or above

192.168.120.159 (Client Machine) :

  • Logstash Jar (Logstash Shipper Conf. File)
  • Java 6 or above

192.168.120.160 (Client Machine) :

  • Logstash Jar (Logstash Shipper Conf. File)
  • Java 6 or above.

Installation

Software Installation on 192.168.120.158 (Server Machine) :

Note – I hope you have already installed Apache Tomcat and Java if not get it up and running fast and then go further.

Step 1 – Download Elasticsearch – http://www.elasticsearch.org/download/

  • Paste downloaded tar in lets say i have pasted it in /usr/myallserversoft/
  • Extract the tar “$ tar -xvf Elastic….tar.gz”
  • Perfect you have installed the ES on server machine successfully but will do some configuration changes later in ES configuration file. 

Step 2 – Download Kibana 3 – http://www.elasticsearch.org/overview/kibana/installation/

  • Paste tar in /usr/myallserversoft/apache7/webapps/kibana
  • Extract the tar “$ tar -zxvf Kibana3…tar.gz”
  • Great will do some configuration changes later

Step 3 – Download Redis – http://redis.io/download

  • Paste tar in /usr/myserversoft
  • Extract the tar “$ tar -zxvf Redis.xX…tar.gz”
  • Good work will do some configuration later
  • Step 4 –  Download Logstash Jar – http://logstash.net/
  • Paste jar in /usr/myjar/

Great! we will do some configuration later.

Configuration

Step 1 – Elasticsearch configuration (Server Machine)

  • Go to /usr/myserversoft/elasticsearch-.0.20.x/config
  • Open and edit elasticsearch.yml file
  • Remove # from cluster.name property – Ex. cluster.name = Master158.
  • Remove # from node.name property – Ex. node.name = Master_Node.
  • Done for ES configuration.

Note – If you are getting error something like “zen.ping.multicast or unicast” please comment me here or paste your error code here i will help you to solve this error.

Step 2 – Kibana 3 Configuration (Server Machine)

  • Go to /usr/myserversoft/apache_tomcat/webapps/kibana
  • Edit config.js file
  • Replace – elasticsearch: “http://”+window.location.hostname+”:9200″
  • Ex. – elasticsearch: “http://”+”192.168.120.158″+”:9200″
  • Here .158 is my server IP and 9200 is port where elasticsearch is running.
  • Ok cool . . ! now done with  Kibana 3 configuration

Step 3 – Logstash Indexer

  • Make a separate directory for Logstash configuration
  • Ex. – /usr/mylogstash_configurationdir/
  • Make file – logstash-indexer.conf  and write below configuration

  • Save file, the logstash indexer is ready for accepting the messages from redis server
  • here we are reading log messages from redis, stdout is for our satisfaction it will print the log messages on console and elasticsearch tag will store all the log messages.




Step 4 – Logstash Shipper configuration for client machines 192.168.120.159, 192.168.120.160

  • Make a separate configuration directories on both the client machine and type following configuration

  • Here, we are accepting the input through command line and after that message will be send to redis server on server machine where redis is running
  • Save this file on both the client machine

[spacer height=”20px”]

DEMO - WeW . . .now its time to start everything Let's see this in Part 2

[spacer height=”20px”]

Leave a Reply

Your email address will not be published. Required fields are marked *