PART2: ELK Stack Big Picture
What, Why, When!
Agenda
- Understand the basic architecture of the ELK stack.
- Understand the use cases of the ELK stack.
Introduction
ELK stack consists of Elasticsearch, Logstash, and Kibana. As discussed in the last blog, Elasticsearch is a search engine, and Logstash is a data ingestion pipeline used to sync the master database with Elasticsearch. Kibana is used to visualize and query the data stored in the Elasticsearch.
Architecture
As shown in the figure, the ELK stack interacts with the Relational database and collects the data for performing the search, filter, and sort operations.
- Logstash connects with the relational database through a database-specific JDBC plugin, queries the data from the database, processes the data, then loads the data to the specified Elasticsearch index.
- Elasticsearch stores data as JSON documents and parses the data into a chunk of words to form an inverted index for delivering data faster when the user sends a search request.
- Kibana is a client interface that connects with Elasticsearch and visualizes the data, providing a dev tool to perform CRUD operations on Elasticsearch.
Use cases
Imagine you have a shopping application that searches the products based on the name or brand and filters the results based on the color, size, price, etc.
Currently, your application connects to a Relational database like PostgreSQL, MySQL, Oracle, etc.
Millions of users are searching for the products on your shopping site.
Your current architecture is Struggling to deliver the data on time and stay on pace. That is where the ELK stack comes into the picture.
Likes the name, Elasticsearch is a search engine based on Apache Lucene. You will need to sync the relevant data from the relational database with Elasticsearch using Logstash and connect your shopping site with Elasticsearch.
When a user sends a search query and filters to your application, the request sent to the server pass to Elasticsearch. Elasticsearch returns the search results based on the filters back to the server. The server processes the results and sends them back to the browser faster than any relational database would do.
To get the live data synchronization with Elasticsearch, Logstash would scan the specified tables periodically and update the Elasticsearch with new/modified records.
Summary
Now that we know how the components of ELK Stack work together. Let’s dive more into Elasticsearch in the upcoming articles.
About me
I’m a software engineer with more than two years of experience.
I’m passionate about technologies and always stay on top of trends to connect with readers. You can reach me on Twitter and LinkedIn.
To know more about me, please visit here.
Please follow me to get the latest content from me.
