Building A Simple Graylog Server on CentOS 7 For Log Analysis

In today’s world of constant, ongoing cyber attacks, administrators are seeking new and innovative ways to analyze existing log data to learn what is normal behavior and uncover compromise. Until recently, the tools that were available to centralize log storage and analysis tended to be:

  1. Non-existent
  2. Moderately Priced But Feature-poor or
  3. Not Affordable for Most Businesses
  4. Difficult to Deploy and Manage

Software like Kiwi Syslog Server, while reliable and easy to deploy, don’t provide much in the way of robust search and analysis features. Other more robust services like Splunk are feature-rich but become very expensive to implement for most enterprises. The recent introduction of the Elk stack and tools like Graylog, have made it possible for everyone to access the tools needed to gather useful information from their log data.

This post outline the basic steps needed to get a Graylog server up, running and ingesting logs for analysis. Graylog has OVA images available for VMware or Virtualbox but their VMs are built on Ubuntu. I don’t recommend using Ubuntu for any serious server deployment beyond testing (why is a discussion for another day) and building your own VM from scratch will give you insight into to deploying and configuring the various components required for an operational Graylog cluster so I am outlining the steps for installing on CentOS 7, the community RedHat distribution. The configuration steps are sufficient to get a test VM up and running. Later posts will address optimizing each component for production / enterprise deployment.

Here are the basic steps:

  1. Download the CentOS 7 Installer ISO
  2. Install CentOS Minimal on a VM
  3. Install MongoDB – stores configuration and meta data
  4. Install Elasticsearch – stores and indexes data ingested by Graylog. Graylog currently does not work with Elasticsearch 2.0 so we will install 1.7.x
  5. Install Graylog Server – Collects logs from various sources, formats and hands off to Elasticsearch
  6. Install Graylog Web – Provides Web interface for managing Graylog
  7. Configure Graylog Input(s)
  8. Install NXLog or Graylog Collector on a Windows or Linux server
  9. Configure the Log Shipper
  10. Enjoy!

Download CentOS 7

I prefer to use the minimal install of CentOS 7 as it comes with the are minimum required to run Linux. Visit the CentOS ISO Download site, choose a mirror and download the correct ISO image. As of the writing of this post, the correct ISO image is called CentOS-7-x86_64-Minimal-1503-01.iso. Save this image to a location that you can access when building a virtual machine.

Building A CentOS 7 Virtual Machine

The virtual machine that we are going to build is going to handle all the services required for a single node, all-in-one Graylog server. This is intended to be used for basic testing and to get you familiar with the configuration and use of a Graylog deployment. In production, one would separate services and build in redundancy. A recommended deployment for enterprise use will be discussed in a future post.

I prefer to use VMware Workstation or a full VMware environment for my VM deployments but you can use Virtualbox and everything should work just fine. I am using VMware Workstation version 11.

  1. Open VMware Workstation
  2. Create a New Virtual Machine
  3. Choose “Custom (advanced) >> Next
  4. Choose “Workstation 11.0” for hardware compatibility  >> Next
  5. Choose “Installer disc image file (iso)”, click the “Browse” button and open the location of the CentOS 7 ISO that you downloaded previously
  6. Click “Open” >> Next
  7. Choose “Linux” as the Guest Operating System and select the “CentOS 64-bit” from the Version drop-down >> Next
  8. Give your VM a name (ex. GRAYLOG-TEST)
  9. Set the location to store the VM >> Next
  10. Select “Number of processors” and “Number of cores per processor”. More = better performance >> Next
  11. Set the “Memory for the Virtual Machine” to a value your hardware supports. 4 GB or more  >> Next
  12. Select “Used bridged networking” for the Network Type >> Next
  13. Leave the “LSI Logic” as the selected controller >> Next
  14. Leave “Virtual Disk Type” as SCSI >> Next
  15. Choose “Create a new virtual disk” >> Next
  16. Set the “Maximum disk size (GB)” to 40 or 60 GB and be sure to split into multiple files (default) >> Next
  17. Leave the “Disk File” as the suggested default >> Next
  18. Finish

Your VM will be created and should boot immediately into the CentOS 7 installer. Follow the installer steps to get a base minimal install of CentOS 7 on the VM. Configure your VM with the automated partitioning and be sure to enable your network adapter and configure a static IP or allow DHCP on the interface. Do not install any added software components. Once installed and booted, log in and do the following:

  1. Make sure you can connect to you LAN and get out to the Internet. If you are unable to resolve DNS, edit the /etc/resolv.conf file and add the following
    # vi /etc/resolv.conf
    
    nameserver 8.8.8.8

    Save the file and test pinging to a site on the Internet by name.

  2. Update the OS
    # yum upgrade
  3. Install a few useful tools
    # yum -y install net-tools
    # yum -y install nano
  4. Install Prerequisites
    # yum -y install java

NOTE: In a working environment, we will want to install the Oracle Java packages as they are more stable and better at garbage collection than the open source packages.

Installing Elasticsearch

Elasticsearch just released version 2.0. Unfortunately, Graylog is not yet compatible with Elasticsearch 2.0 so we will configure our VM to install Elasticsearch version 1.7 for compatibility.

First we will import the Elasticsearch GPG key

# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

Next, we add the correct Elasticsearch Repository.

# nano /etc/yum.repos.d/elasticsearch.repo

The file will be empty. Paste in the following:

[elasticsearch-1.7]
name=Elasticsearch repository for 1.7.x packages
baseurl=http://packages.elastic.co/elasticsearch/1.7/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

 Save the file by pressing CTRL+O <Enter>. CTRL+X to exit

Now we an install Elasticsearch.

# yum -y install elasticsearch

Set Elasticsearch to start when the system starts

# systemctl daemon-reload
# systemctl enable elasticsearch-service

The only configuration setting we care about on this test VM for Elasticsearch is the name of the cluster. Open the /etc/elasticsearch/elasticsearch.yml file and set the cluster.name value to “graylog2

# nano /etc/elasticsearch/elasticsearch.yml

cluster.name: graylog2

For security, you should disable dynamic scripts by appending the following to the end of the elasticsearch.yml file:

script.disable_dynamic: true

Once the basic configuration is complete, restart the elasticsearch service.

# systemctl restart elasticsearch.service

 Once the elasticsearch service has had sufficient time to restart, you can check the status of your cluster using CURL.

# curl -X GET http://localhost:9200

{
  "status" : 200,
  "name" : "Franz Kafka",
  "cluster_name" : "graylog2",
  "version" : {
    "number" : "1.7.3",
    "build_hash" : "05d4530971ef0ea46d0f4fa6ee64dbc8df659682",
    "build_timestamp" : "2015-10-15T09:14:17Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

 If you want to see more details about the status of you cluster, use the command below. Elasticsearch must show status “green” in order to work with a single node Graylog install.

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "graylog2",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 4,
  "active_shards" : 4,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0
}

 Installing MongoDB

 MongoDB can be installed in RPM format. In order to automatically download, it is necessary to add the repository needed to install MongoDB.

# nano /etc/yum.repos.d/mongodb-org-3.0.repo

[mongodb-org-3.0]
name=MongoDB Repository
baseurl=http://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.0/x86_64/
gpgcheck=0
enabled=1

CTRL+O to save and CTRL+X to exit.

Install MongoDB

# yum -y install mongodb-org

The above command installs

  1. mongodb-org-server
  2. mongodb-org-mongos
  3. mongodb-org-shell
  4. mongodb-org-tools

CentOS 7 installs SELinux by default so we need to install the following package to allow us to manage elements of the SELinux policy:

# yum -y install policycoreutils-python

Run the following command to configure SELinux to allow MongoDB to start.

# semanage port -a -t mongod_port_t -p tcp 27017

Enable the MongoDB service.

# systemctl enable mongod

Start MongoDB service.

# systemctl start mongodb

 Install Graylog Server

Graylog server accepts and processes incoming log messages, storing the data in Elasticsearch indexes for visualization and analysis using the Graylog web services or Kibana.

First, use the following to install the correct repository for Graylog:

# rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-1.2-repository-el7_latest.rpm

Now we can install the latest Graylog server.

# yum -y install graylog-server

Once installed, there are a few minimal configuration changes that we need to make.

We will need to generate a secret password used for services in a cluster to exchange information and the default “root user” password for the API and Web interface.

First, generate a password secret of at least 64 characters. We can use pwgen but we will need install the EPEL repository and the correct package.If you have another password generator, you can use that and just copy and paste the password into the config file.

Install EPEL

# cd /tmp
# rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

Now EPEL should appear in your repo list

# yum repolist

Install the pwgen package

# yum -y install pwgen

Generate a secret

# pwgen -N 1 -s 96
y2396e28v6cka9pa6mb8vk63kpaca69272238fzgb6xc44eyfgh4y7h3hk7mf666vvxg6at9786p3a69hd42a63mfz3qtyh7

Copy the password into a notepad window.

Generate the hash for the default root user password (admin account) for Graylog server and Graylog web

# echo -n TypeYourPasswordHere | sha256sum
769b1626b7b1c3c124a05dc1f48b98d9c86807b8453927d664db20817e946d91

Copy the hash to the same notepad window

Edit the configuration file.

# nano /etc/graylog/server/server.conf

Copy the generated password into the password_secret value shown below

password_secret = y2396e28v6cka9pa6mb8vk63kpaca69272238fzgb6xc44eyfgh4y7h3hk7mf666vvxg6at9786p3a69hd42a63mfz3qtyh7

Add the hash to the root_password_sha2 value

root_password_sha2 = 769b1626b7b1c3c124a05dc1f48b98d9c86807b8453927d664db20817e946d91

Set to admin user email address

root_email = "user@mydomain.com"

Set the admin user timezone

root_timezone = America/Chicago

Graylog will try to find the Elasticsearch nodes automatically using multicast but that is not the optimal manner so we will set this ping unicast hosts and set it to the IP address of our  VM. This is done by setting the following values:

elasticsearch_http_enabled = false
elasticsearch_discovery_zen_ping_unicast_hosts = x.x.x.x:9300

Where x.x.x.x is the IP address of the VM.

Next, we need to set the VM to be the master node (even though we have one Graylog server, this is required).

is_master = true

Now we set the number of messages per index. Several smaller indexes are better than one large one.

elasticsearch_max_docs_per_index = 20000000

We also need to tell the sever how many indexes to keep open and keep stored.

elasticsearch_max_number_of_indices = 20

Shards are used by Elasticsearch to spread indices across nodes. We only have one node here so set shards to 1.

elasticsearch_shards = 1

Replicas are used to keep indices running when a node fails. We only have one node so set replicas to 0.

elasticsearch_replicas = 0

MongoDB is installed locally so we don’t need any authentication information.

mongodb_useauth = false

Save the changes to the config file (CRTL+O, CTRL+X) and we are ready to enable and restart the Graylog server service.

# systemctl enable graylog-server
# systemctl restart graylog-server

We can check the status of the service by tailing the log.

# tail -f -n 20 /var/log/graylog-server/server.log

You can also check the status using systemctl.

# systemctl status graylog-server

Install Graylog Web

We are going to install our Web interface for Graylog (aka Graylog web) on the same server as our Graylog server and other components. This is definitely not the recommended deployment method for a production instance but will get us up and running and able to test the viability of including Graylog in our production environments.

We have already installed the necessary repo on our VM so we can move ahead with the install.

# yum -y install graylog-web

Now we edit the configuration file and set a few parameters.

# nan0 /etc/graylog/web/web.conf

Set the server URIs (the Web APIs for the Graylog servers in our cluster). In this case, the localhost only.

graylog2-server.uris = "http://127.0.0.1:12900"

Set the application secret to the same value you set for the password_secret in the Graylog server configurations.

application.secret ="y2396e28v6cka9pa6mb8vk63kpaca69272238fzgb6xc44eyfgh4y7h3hk7mf666vvxg6at9786p3a69hd42a63mfz3qtyh7"

Save the configuration file. Enable and restart the Graylog web service.

# systemctl enable graylog-web
# systemctl restart graylog-web

You can check the status of the Graylog web service using systemctl.

# systemctl status graylog-web

Adjust the Firewall to Allow Connection to Web Interface

We only have to enable the firewall to allow access to the Web interface from the outside. All the other services are local in this VM install. The Web interface listens on port 9000.

# firewall-cmd --permanent --zone=public --add-port=9000/tcp
# firewall-cmd --reload

If everything is running correctly, you should now be able to connect to the Graylog web interface at http://x.x.x.x:9000. Log in with the username admin and the password you used to generate your sha256sum value.

Once logged in, you will be directed to the default search page.

We will address configuring inputs and sending data to your test Graylog server in upcoming posts.

Please keep in mind, this post covers a basic install on a single VM for testing and familiarization only. I will create some other posts relating to my experience in deploying larger Elasticsearch and Graylog clusters. I have operational clusters that can handle loads of 100K plus messages per second without issue and will detail some of the tricks I have uncovered as well as specific configuration optimizations and input configurations in upcoming posts.

Leave a Reply

Your email address will not be published. Required fields are marked *