Any big data infrastructure operating at scale, requires the following technologies:
- Enterprise Search
- Enterprise Messaging
Managing these three verticals is a mammoth task.
When you have your big data infrastructure scaling according to business needs, you need to choose management technologies that are common/applicable across multiple areas. This way you minimize the number of complementary technologies in operation at your big data infrastructure.
One such technology used in management and coordination is Apache Zookeeper.
When you use the following technologies in your big data infrastructure, you can use Apache Zookeeper for coordination:
- Apache Solr for Enterprise Search
- Apache Kafka for Enterprise Messaging
As depicted in the diagram, Zookeeper can be the central pivot for managing your big data infrastructure.
Please do not hesitate to review my Introduction to Zookeeper post.