Need faster, data driven insights on your data? Supercharge your enterprise with ElasticSearch

Introduction
Organizations are constantly struggling for faster access to information that can help them make better decisions. The dynamic nature of many businesses has made it imperative for them to acquire solutions that can help move the needle on decision making. This is where a need for faster and flexible reporting becomes a necessity and that can really help provide data driven insights into meaningful KPIs thus empowering businesses to make quick and efficient decisions.

Many enterprises use relational databases for running their core business as it provides good transactional capabilities. However, when it comes to using a transactional relational database for reporting, the same becomes a bottleneck. Operations involving bulk retrieval of data from a traditional Database can really hurt the performance. This is where a NOSQL database like Elasticsearch comes to the rescue.

Use Case
We will use one of our customers in the Insurance industry that creates new plans for Insurance companies as an example to demonstrate the value of using Elasticsearch. These plans are created for Insurance companies and then an approval from respective State Government needs to be obtained before being released into the market. All the plans are stored in a No SQL database and data retrieval using the reporting tool was slow because of performance bottlenecks with the No SQL database. The plan data was migrated to Elastic Search in order to enable faster access to data.

We have used Elastic Search in order to implement a reporting solution for our customer. The data source is a No SQL database from which we extract the plan details. The data is then transformed and loaded to the index. An index is like a ‘database’ in a relational database. The reports are then generated from these indexes and submitted to the US government for approvals.

Technical Implementation
Elasticsearch is an open source, distributed, RESTful, full-text search engine built on top of Apache Lucene. Elasticsearch is developed in JAVA. It uses schema free JSON documents and comes with extensive REST APIs for storing and searching the data. The primary way of interacting with Elasticsearch is using the REST API over HTTP. We can use any of the popular HTTP clients, such as Curl or Postman.

Curl is a command line-based client available on most operating systems while Postman is an UI-based HTTP client available for major operating systems. The figure below shows how the Elasticsearch query works.

Elasticsearch has its own Query Domain Specific Language, where you specify the query in JSON format. The documents in the index are distributed across multiple shards, that are further distributed across multiple nodes. This ensures there is redundancy and it increases the query capacity. Elasticsearch will automatically rebalance the entire cluster by adding or removing the shards based on the need.

The following APIs (GET, POST, PUT, DELETE) were used to query, extract and load plan details into Elasticsearch.

GET API:
This API returns the results that matches the query. We can either specify in the request body or pass it as a string. In the below screenshot we are extracting the plan details from Marklogic.

POST API:
Posts the data into the Index. In the below example, we are inserting the details for one of the plan. (http://localhost:9200/index/type/id/_update)