Build Live Dashboard Using Open Source Tools

Table of contents:

Introduction : 2

What is Kibana? : 2

Integrate elasticsearch into Kibana: 4

Kibana access: 4

Kibana visualization: 5

Auto-Refresh your dashboard : 8

Lets see how it works: 8

Introduction:

We are all familiar with Elasticsearch, it is an open-source and json based search engine used for free text searching and indexing. It stores data in a structure which helps to examine large volume of data quickly. In this blog, you’ll get to know about Kibana, its advantages, how to install it, how to integrate Kibana with Elasticsearch, create dashboard in Kibana and publish live dashboard for application.

What is Kibana?

Kibana is an open-source tool used mainly for data visualization. Kibana can communicate, look at, search data which is stored in Elasticsearch indices. Kibana also helps to carry out data analysis and lets us to understand data in a variety of charts, tables and maps. Elasticsearch makes Kibana as an excellent choice for data visualization. 

Installation:

Let’s assume that you’re in a Linux based environment. Assuming that you also have Elasticsearch and JDK 8 or above installed. Before installing Kibana check compatibility with Elasticsearch version.

Let’s get on with downloading Kibana using the command below:

Step 1: Download the tar file. 

wget https://artifacts.elastic.co/downloads/Kibana/Kibana-6.7.1-linux-x86_64.tar.gz

Step 2: Unzip the tar file.

tar -xzf Kibana-6.7.1-linux-x86_64.tar.gz 

Before starting Kibana server make sure that Elasticsearch is running. Kibana should be started only when Elasticsearch is running.

Step 3: Start the Kibana server.

cd Kibana/
./bin/Kibana

You can access it at http://localhost:5601 on your web browser. To confirm everything is working fine, type http://localhost:5601 in your browser and you should see something like this.

Tdf6 12zmuI34jUAMdSEJJTik ztk4K2n8G34V7hkp5 ex2ZDxA QmYtTe V8OTAVe

Integrate Elasticsearch into Kibana: 

In order to configure Kibana, you have to first create an index in Elasticsearch. There are multiple ways to connect Elasticsearch (one of the easy methods is curl). Here we use curl for indexing data into Elasticsearch via terminal.

curl  -H “Content-Type: application/json” -XPOST ‘localhost:9200/companydatabase/employees/_bulk?pretty&refresh’  –data-binary “@test.json”

blank

Access Kibana:           

When we access Kibana for the first time, we must define an index pattern that matches the name of one or more indices in Elasticsearch.  We can add index pattern by using management tab (left panel bottom option).

An index pattern can match a name of the simple index and use wild card (*) to match the multiple indices.

As I mentioned above, name of index pattern should match the one or more indices in Elasticsearch. Here I create index pattern named as company database which I indexed already.

RIeaOQ PLJ4wM s3qo1WUhbglZYUtjcAvSPGKSWFpV0vAm5g IQeI 2H0EYJzxG9vIvWyHOYAqDNHBsrzWM1q6Dn6RrI1QTnYAsRkoTFtLvzmT0rsmhaJTyIw ZYTytBLW 7kewlruaG8oF0

Kibana visualization:

Visualize option enables us to represent the data in Elasticsearch in variety of formats like bar chart, pie chart, heat map, etc.

Here I am going to use the company database index for basic visualization. *As I said before in order to create visualization, we need to create index pattern in management tool.

We can create visualization by clicking visualize option in the left panel and then select visualization type.

blank

Here I choose pie chart format. It displays the following diagram as shown below

r46Hn6r6HGHDpsP91anjQc fF1KGP9lFA9 eR2LhUtl9NpccJSg035HNeElXgtzc1uIYqfmrF554yWM jrcUbI8yr5AsvGV9TAU7oaRDK3d ztsdPHxtQsLlsHHR7kcgJNvrkxyx1 VHXLf3

Now I click split slices options to segregate the data based on specific field. Here I want to classify the data based on salary, so select Terms option under aggregation.

QVp6Xde62mE5JvqPiOcwE0eYUDuWGjlx4cx1tKuJFdSu6dqrpMnxe30o6 lC9oPsr4HNRxE7VK hz2Ods2qGix3hLhDWQza73kh7Y38WsYHRmh fEWZEhFezJ8B8YwuuhgoLlU5UIiNMovqm

Finally select the field on which the data want to be classified and select the size also. For Example: I select salary and type the size as 100(to display 100 slices in the chart). Then click apply changes method.

IbNbeOgzYxAmveP GbcX61jZoEJC6yS0kgX4mA7Ik8Q6alKeOP6LjhzQG9rlxmruIYm MA35dgonnyNU9MDXCAa2XsK4KnwrSp5PM9onuC1 InLs52ujnOIVi0KGjquA3sfuYGkbgp 5Yrsu

Auto-Refresh your dashboard

For manual refresh there is an option named REFRESH in the top right corner.

For auto-refresh there is an option auto-refresh in same top right corner. By default, auto-refresh is off in Kibana. we must turn it on by clicking. Then it displays many options, we can select the timing as per SLA.

Here I select 5 seconds. After selecting 5 seconds visualize get refreshed once for every 5 seconds.

blank

Let’s see how it works:

Let’s update the salary of one id from 5800 to 3000 by using the following curl command.

curl -X POST “localhost:9200/companydatabase/employees/ijIcU2oBPbUtXEGa4A22/_update” -H ‘Content-Type: application/json’ -d’ {     “doc” : {         “Salary” : “3000”     },     “doc_as_upsert” : true } ‘

The following image illustrate changes get reflected in the visualization.

4xqQgn1VqRs5JX4XvmdkerUArjr8ZbdL2v0dk4h9rD

How to publish your visualization:

  • Select share option on top of the right corner. 
  • It displays two options Embed code and Permalinks. 
  • Click on Embed code. 
  • It shows  copyiframecode icon. 
  • Save the code in HTML format. 

we can publish visualization by using copyiframecode. By clicking this icon, it copies the code. Then we can use the copied code to the downstream applications dashboard to see real time visualization.

For an example I’m creating a simple HTML page to see if my viz is refreshing or not. 

vi visualization.html

<!DOCTYPE html> 

<html

<body

The copied code should be paste here

</body

</html>  

Save the file and open it in the any web browser, to see your data.

r JnufQlFYBzzzp5x5liPXGonQjAOQwwhk19veGjnn 9S 5gnxThN XY4 a 2Z8as1qWimqrGfFCWQwVaNju2npHXoq711km2z5oVnbWFwSLkiiSdRl3Z7P1pytSrNS45W0vEwfTudcb0Uut

Once we make updates/changes to the data in the index, those changes will be reflected immediately in the already published visualization (in the next 5 seconds).

Let’s change the salary from 3000 to 2000 for the same id,

curl -XPOST “localhost:9200/companydatabase/employees/ijIcU2oBPbUtXEGa4A22/_update” -H ‘Content-Type: application/json’ -d’
{
  “doc” : {
        “Salary” : “2000”
    },
    “doc_as_upsert” : true
}

See the changes in your web browsers page:

CSuBqCn40fBbdMmABrvymKJGbmNIqKhoOZN smlNnMa 7AVOyb0vtAJz5ms BMOaoegD5s eSOuzUWykwlGYFSPTZ9XP92yMZ0stXZsFEny P8iJKzivcjVdPKQlqfgV Df7 DACHud5s9zz

Author Bio

Priya –

A passionate big data developer, having hands on experience in end to end Bigdata application, starting from spawning cluster on-cloud or on-premises, developing and testing the integration of the application. Loves learning new technologies and troubleshooting the issues in the existing implementation.

Mahalakshmi –

A big data developer with good experience in Big data applications. She has lots of interest in understanding and working on new technology. Interested in analyzing and solving the issues in the application.

Share This:

Yashpal

Professional blogger and content writer. I like to share the latest information topics on technology, science, health, social media trends and many more.

Yashpal has 58 posts and counting. See all posts by Yashpal

blank

Leave a Reply

Your email address will not be published. Required fields are marked *