Apache Kafka seems to be the standard solution in event-driven architecture nowadays, but we should focus on whether it is the right choice for our needs.
This blog summarizes my original post on Medium.com.
With the rise of microservices and the API led approach, it seems as if we’ve forgotten about the importance of messaging systems, and we have had to rediscover them. So, we’re coming back to event-driven architecture to help us decouple consumers and producers and move from orchestration to choreography.
No single messaging solution is perfect for every set of requirements. In today’s world, with so many different variables for the different kinds of applications, one size does not fit all. So you should stop thinking about which messaging solution is the best one, and think more about which one serves your architecture best and fulfills both technical and business requirements.
Apache Kafka is a fantastic messaging solution, a performant, production-ready solution with impressive capabilities for replay and a powerful API to ease integration. However, Apache Kafka has some challenges in today’s cloud-native world because it doesn’t follow all of its rules. LinkedIn created Kafka in 2011, before containers and other cloud technologies were common.
Apache Pulsar, another open-source messaging project that Yahoo! created, is starting to challenge Kafka’s dominance in the market. Pulsar’s capabilities include:
- Pulsar functions: Easily deploy lightweight compute logic using developer-friendly APIs without needing to run a stream processing engine
- Proven in production: Apache Pulsar has run in production at Yahoo scale for over three years, with millions of messages per second across millions of topics
- Horizontally scalable: Seamlessly expand capacity to hundreds of nodes
- Low latency with durability: Designed for low publish latency (< 5ms) at scale with strong durability guarantees
- Geo-replication: Designed for configurable replication between data centers across multiple geographic regions
- Multi-tenancy: Built from the ground up as a multi-tenant system. Supports Isolation, Authentication, Authorization, and Quotas
- Persistent storage: Persistent message storage based on Apache BookKeeper. Provides IO-level isolation between write and read operations
- Client libraries: Flexible messaging models with high-level APIs for Java, C++, Python and GO
- Operability: REST Admin API for provisioning, administration, tools, and monitoring. Deploy on bare metal or Kubernetes.
In its design, Apache Pulsar addresses some of the main weaknesses of Apache Kafka with geo-replication and its cloud-native approach. But, Apache Pulsar is only one of several other commercial and open-source messaging options that you can choose for your organization.
To learn more about Apache Kafka, Apache Pulsar, and other commercial and open-source enterprise messaging options read our whitepaper, which sets out their pros, cons, and information about their complexity and cost of ownership to help you choose the type that best fits your requirements.
