Build Live Dashboard Using Open Source Tools

Table of contents:

Introduction : 2

What is Kibana? : 2

Integrate elasticsearch into Kibana: 4

Kibana access: 4

Kibana visualization: 5

Auto-Refresh your dashboard : 8

Lets see how it works: 8

How to publish your visualization: 9

Introduction:

We are all familiar about Elasticsearch, it is an open-source and json based search engine used for free text searching and indexing. It stores data in a structure which helps to examine large volume of data quickly. In this blog, you’ll get to know about Kibana, its advantages, how to install it, how to integrate Kibana with Elasticsearch, create dashboard in Kibana and publish live dashboard for application.

What is Kibana?

Kibana is an open-source tool used mainly for data visualization. Kibana can communicate, look at, search data which is stored in Elasticsearch indices. Kibana also helps to carry out data analysis and lets us to understand data in a variety of charts, tables and maps. Elasticsearch makes Kibana as an excellent choice for data visualization. 

Installation:

Let’s assume that you’re in a Linux based environment. Assuming that you also have Elasticsearch and JDK 8 or above installed. Before installing Kibana check compatibility with Elasticsearch version.

Let’s get on with downloading Kibana using the command below:

Step 1: Download the tar file. 

wget https://artifacts.elastic.co/downloads/Kibana/Kibana-6.7.1-linux-x86_64.tar.gz

Step 2: Unzip the tar file.

tar -xzf Kibana-6.7.1-linux-x86_64.tar.gz 

Before starting Kibana server make sure that Elasticsearch is running. Kibana should be started only when Elasticsearch is running.

Step 3: Start the Kibana server.

cd Kibana/
./bin/Kibana

You can access it at http://localhost:5601 on your web browser. To confirm everything is working fine, type http://localhost:5601 in your browser and you should see something like this.

Integrate Elasticsearch into Kibana: 

In order to configure Kibana, you have to first create an index in Elasticsearch. There are multiple ways to connect Elasticsearch (one of the easy methods is curl). Here we use curl for indexing data into Elasticsearch via terminal.

curl  -H “Content-Type: application/json” -XPOST ‘localhost:9200/companydatabase/employees/_bulk?pretty&refresh’  –data-binary “@test.json”

Access Kibana:           

When we access Kibana for the first time, we must define an index pattern that matches the name of one or more indices in Elasticsearch.  We can add index pattern by using management tab (left panel bottom option).

An index pattern can match a name of the simple index and use wild card (*) to match the multiple indices.

As I mentioned above, name of index pattern should match the one or more indices in Elasticsearch. Here I create index pattern named as company database which I indexed already.

Kibana visualization:

Visualize option enables us to represent the data in Elasticsearch in variety of formats like bar chart, pie chart, heat map, etc.

Here I am going to use the company database index for basic visualization. *As I said before in order to create visualization, we need to create index pattern in management tool.

We can create visualization by clicking visualize option in the left panel and then select visualization type.

Here I choose pie chart format. It displays the following diagram as shown below

Now I click split slices options to segregate the data based on specific field. Here I want to classify the data based on salary, so select Terms option under aggregation.

Finally select the field on which the data want to be classified and select the size also. For Example: I select salary and type the size as 100(to display 100 slices in the chart). Then click apply changes method.

Auto-Refresh your dashboard

For manual refresh there is an option named REFRESH in the top right corner.

For auto-refresh there is an option auto-refresh in same top right corner. By default, auto-refresh is off in Kibana. we must turn it on by clicking. Then it displays many options, we can select the timing as per SLA.

Here I select 5 seconds. After selecting 5 seconds visualize get refreshed once for every 5 seconds.

Let’s see how it works:

Let’s update the salary of one id from 5800 to 3000 by using the following curl command.

curl -X POST “localhost:9200/companydatabase/employees/ijIcU2oBPbUtXEGa4A22/_update” -H ‘Content-Type: application/json’ -d’ {     “doc” : {         “Salary” : “3000”     },     “doc_as_upsert” : true } ‘

The following image illustrate changes get reflected in the visualization.

How to publish your visualization:

  • Select share option on top of the right corner. 
  • It displays two options Embed code and Permalinks. 
  • Click on Embed code. 
  • It shows  copyiframecode icon. 
  • Save the code in HTML format. 

we can publish visualization by using copyiframecode. By clicking this icon, it copies the code. Then we can use the copied code to the downstream applications dashboard to see real time visualization.

For an example I’m creating a simple HTML page to see if my viz is refreshing or not. 

vi visualization.html

<!DOCTYPE html> 

<html

<body

The copied code should be paste here

</body

</html>  

Save the file and open it in the any web browser, to see your data.

Once we make updates/changes to the data in the index, those changes will be reflected immediately in the already published visualization (in the next 5 seconds).

Let’s change the salary from 3000 to 2000 for the same id,

curl -XPOST “localhost:9200/companydatabase/employees/ijIcU2oBPbUtXEGa4A22/_update” -H ‘Content-Type: application/json’ -d’
{
  “doc” : {
        “Salary” : “2000”
    },
    “doc_as_upsert” : true
}

See the changes in your web browsers page:

Author Bio

Priya –

A passionate big data developer, having hands on experience in end to end Bigdata application, starting from spawning cluster on-cloud or on-premises, developing and testing the integration of the application. Loves learning new technologies and troubleshooting the issues in the existing implementation.

Mahalakshmi –

A big data developer with good experience in Big data applications. She has lots of interest in understanding and working on new technology. Interested in analyzing and solving the issues in the application.

James

James have a highly skilled & experienced in software development company. He has wide experience in IT industries to develop creative business system based on Microsoft dynamics CRM.

James has 6 posts and counting. See all posts by James

Avatar

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.