ELK.stack.on.my.laptop

ELK Stack

Combining Elastic search, Logstash, and Kibana (ELK) gives you an easy way to analyze logged data. The typical pipeline goes like this.

• Some tasks output logs into files.
• Logstash monitors those files.
• Logstash translates those logs into records.
• Logstash saves the records into elastic search.
• Elastic search indexes the logged records.
• Kibana allows you to query the elastic search engine

It is a powerful combination, so I wanted to give it a try.

I followed the steps outlined in http://aarvik.dk/a-bit-on-elasticsearch-logstash-kibana-the-elk-stack/ . You can obviously do the same, but here are my notes on the process (I did this on my Macbook Air).

Get the files

curl -OL https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.0.tar.gz
curl -OL https://download.elasticsearch.org/logstash/logstash/logstash-1.4.0.tar.gz
curl -OL https://download.elasticsearch.org/kibana/kibana/kibana-3.0.0.tar.gz

It is amazing that all of this is completely free to download and try out.

Testing each of the downloads

# Start your (elastic search) engines ... 
elasticsearch-1.3.4/bin/elasticsearch
curl -X GET localhost:9200
curl -X POST localhost:9200/person/1 -d '{ "info" : {"height" : 2, "width" : 20 } }'

# Testing logstash with command line arguments:
logstash-1.4.0/bin/logstash -e 'input { stdin { } } output { stdout { codec => rubydebug } }'
# or run with a config file
logstash-1.4.0/bin/logstash -f ./logstash.conf

I used the same logstash config file listed in the reference page I noted above. However, I did change one of the inputs to read from /var/log/system.log . At this point, I just wanted it get input from a file I knew was geting updates.

Setting up a local web development server - apache

In order to use kibana, you need a web server. Luckily apache is pretty much ready to use on Macbooks. I followed steps outline in http://ole.michelsen.dk/blog/setup-local-web-server-apache-php-macos-x-mavericks.html. Update The setup is slightly different for Yosemite.

I followed that site to get my server started, until my browser said "It works". Then I configured kibana to use my elastic search engine. To do this, update kibana/config.js to have this line:

elasticsearch: "http://localhost:9200",

This replaces -->

elasticsearch: "http://"+window.location.hostname+":9200",

This works because everything is on one machine. Finally, I moved the kibana folder over to be read.

mkdir ~/Sites
cp -r kibana-3.0.0 ~/Sites/
sudo apachectl start

You should now be able to visit your kibana dashboard at http://localhost/~username/kibana

Conclusion

Getting the ELK stack up and running was really easy. Hopefully I can think of a way to put this cool technology to good use.

Resources I used ...

• http://aarvik.dk/a-bit-on-elasticsearch-logstash-kibana-the-elk-stack/
• http://ole.michelsen.dk/blog/setup-local-web-server-apache-php-macos-x-mavericks.html