Building Analytics Engine Using Akka, Kafka & ElasticSearch


In this blog , I will share my experience on building scalable, distributed and fault-tolerant  Analytics engine using Scala, Akka, Play, Kafka and ElasticSearch.

I would like to take you through the journey of  building an analytics engine which was primarily used for text analysis. The inputs were structured, unstructured and semi-structured data and we were doing a lot of data crunching using it. The Analytics engine was accessible by the rest-client and web-client(Built In with engine)  as shown in below diagram.

Here is a quick overview on technology stack :

  1. Play Framework  as Rest Server & Web Application  (Play is MVC  framework based on  lightweight, stateless and web friendly architecture.)
  2. Akka cluster as processing engine.(Akka is a toolkit and runtime for building highly concurrent,distributed, and resilient message driven applications on the JVM.)
  3. ClusterClient (It was contributed module) for communication with Akka cluster. It used to run on rest…

View original post 987 more words