PART2: ELK Stack Big Picture
- Understand the basic architecture of the ELK stack.
- Understand the use cases of the ELK stack.
ELK stack consists of Elasticsearch, Logstash, and Kibana. As discussed in the last blog, Elasticsearch is a search engine, and Logstash is a data ingestion pipeline used to sync the master database with Elasticsearch. Kibana is used to visualize and query the data stored in the Elasticsearch.
As shown in the figure, the ELK stack interacts with the Relational database and collects the data for performing the search, filter, and sort operations.
- Logstash connects with the relational database through a database-specific JDBC plugin, queries the data from the database, processes the data, then loads the data to the specified Elasticsearch index.
- Elasticsearch stores data as JSON documents and parses the data into a chunk of words to form an inverted index for delivering data faster when the user sends a search request.
- Kibana is a client interface that connects with Elasticsearch and visualizes the data, providing a dev tool to perform CRUD operations on Elasticsearch.
Imagine you have a shopping application that searches the products based on the name or brand and filters the results based on the color, size, price, etc.
Currently, your application connects to a Relational database like PostgreSQL, MySQL, Oracle, etc.
Millions of users are searching for the products on your shopping site.
Your current architecture is Struggling to deliver the data on time and stay on pace. That is where the ELK stack comes into the picture.
Likes the name, Elasticsearch is a search engine based on Apache Lucene. You will need to sync the relevant data from the relational database with Elasticsearch using Logstash and connect your shopping site with Elasticsearch.
When a user sends a search query and filters to your application, the request sent to the server pass to Elasticsearch. Elasticsearch returns the search results based on the filters back to the server. The server processes the results and sends them back to the browser faster than any relational database would do.
To get the live data synchronization with Elasticsearch, Logstash would scan the specified tables periodically and update the Elasticsearch with new/modified records.
Now that we know how the components of ELK Stack work together. Let’s dive more into Elasticsearch in the upcoming articles.
I’m a software engineer with more than two years of experience.
To know more about me, please visit here.
Please follow me to get the latest content from me.