Elasticsearch, Fluentd, and Kibana
Install fluentd, elasticsearch and kibana
Fluentd 2.3.1
Step 0: Before Installation
This will prevent many unnecessary problems.
Before Installing Fluentd
You MUST set up your environment according to the steps below before installing Fluentd. Failing to do so will be the cause of many unnecessary problems.
Set Up NTP
Increase Max # of File Descriptors
Please increase the maximum number of file descriptors. You can check the current number using the ulimit -n command.
$ ulimit -n
65535
If your console shows 1024, it is insufficient. Please add following lines to your /etc/security/limits.conf file and reboot your machine.
root soft nofile 65536
root hard nofile 65536
* soft nofile 65536
* hard nofile 65536
Optimize Network Kernel Parameters
For high load environments consisting of many Fluentd instances, please add these parameters to your /etc/sysctl.conf file. Please either type sysctl -p or reboot your node to have the changes take effect. If your environment doesn’t have a problem with TCP_WAIT, then these changes are not needed.
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240 65535
Step 1: Install from rpm Repository
CentOS and RHEL 5, 6 and 7 are currently supported. Amazon Linux RPM is built on 2015.03 environment.
Executing install-redhat-td-agent2.sh will automatically install td-agent on your machine. This shell script registers a new rpm repository at /etc/yum.repos.d/td.repo and installs the td-agent rpm package.
$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh
Step2: Launch Daemon
The /etc/init.d/td-agent script is provided to start, stop, or restart the agent.
$ /etc/init.d/td-agent start
Starting td-agent: [ OK ]
$ /etc/init.d/td-agent status
td-agent (pid 21678) is running...
The following commands are supported:
$ /etc/init.d/td-agent start
$ /etc/init.d/td-agent stop
$ /etc/init.d/td-agent restart
$ /etc/init.d/td-agent status
Please make sure your configuration file is located at /etc/td-agent/td-agent.conf.
# systemctl status td-agent.service
● td-agent.service - LSB: data collector for Treasure Data
Loaded: loaded (/etc/rc.d/init.d/td-agent)
Active: inactive (dead)
Docs: man:systemd-sysv-generator(8)
# systemctl enable td-agent.service
td-agent.service is not a native service, redirecting to /sbin/chkconfig.
Executing /sbin/chkconfig td-agent on
# /sbin/chkconfig td-agent on
Step3: Install elasticsearch plgin
/usr/sbin/td-agent-gem install fluent-plugin-elasticsearch
Elasticsearch 2.3.3
wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.3.3/elasticsearch-2.3.3.rpm
# yum install elasticsearch-2.3.3.rpm
Examining elasticsearch-2.3.3.rpm: elasticsearch-2.3.3-1.noarch
Marking elasticsearch-2.3.3.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package elasticsearch.noarch 0:2.3.3-1 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=============================================================================================================================================================================================
Package Arch Version Repository Size
=============================================================================================================================================================================================
Installing:
elasticsearch noarch 2.3.3-1 /elasticsearch-2.3.3 29 M
Transaction Summary
=============================================================================================================================================================================================
Install 1 Package
Total size: 29 M
Installed size: 29 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Installing : elasticsearch-2.3.3-1.noarch 1/1
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
sudo systemctl start elasticsearch.service
Verifying : elasticsearch-2.3.3-1.noarch 1/1
Installed:
elasticsearch.noarch 0:2.3.3-1
Complete!
# systemctl enable elasticsearch.service
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
# systemctl start elasticsearch.service
Kibana 4.5.1
Kibana 4.5.x requires Elasticsearch 2.3.x
Installing Kibana with yum
Download and install the public signing key:
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
Create a file named kibana.repo in the /etc/yum.repos.d/ directory with the following contents:
[kibana-4.5]
name=Kibana repository for 4.5.x packages
baseurl=http://packages.elastic.co/kibana/4.5/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
Install Kibana by running the following command:
yum install kibana
Configure Kibana to automatically start during bootup. If your distribution is using the System V version of init (check with ps -p 1), run the following command:
chkconfig --add kibana
If your distribution is using systemd, run the following commands instead:
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable kibana.service
Kibana is now running on port 5601.
Referfance
http://docs.fluentd.org/articles/free-alternative-to-splunk-by-fluentd
https://www.digitalocean.com/community/tutorials/elasticsearch-fluentd-and-kibana-open-source-log-search-and-visualization
https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html
http://masasuzu.hatenablog.jp/entry/20120520/1337518231
http://chi15036-blog.logdown.com/posts/297025-elasticsearch-fluentd-kibana4-installation-details-processes-in-ubuntu1404
http://docs.fluentd.org/articles/common-log-formats