Today Many companies in startup world are completely dependent on AWS infrastructure. Its a good strategy since you do not have to manage your own infrastructure and saves you from lot of headache.
Today we will discuss a bit about brokers available in AWS infrastructure. AWS has mainly 2 types of broker offering
a. SQS (Simple queue service) – More like ActiveMQ, RabbitMQ
b. Kinesis (Distributed, fault tolerant, highly scalable message broker) – less features but optimized for ingesting and delivering massive number of events at extremely low latency.
Design of Kinesis is inspired by Linked-in donated Kafka. Linked in processes billions of events per day using Kafka and it’s apache top level project which is being used in many highly scalable architecture.
I want to focus in this post on some of the key differences between Kinesis and Kafka. As stated in the beginning working with AWS infrastructure is a good thing but over-reliance on AWS infrastructure has some major problems.
a. You are vendor locked-in so tomorrow if you want to shift to Digital Ocean or even own infrastructure you will not be able to do so.
b. You are limited by the restrictions put by AWS like how many transactions you can do per unit of time
so, in the light of above 2 points I will try to explain where Kafka should be used instead of RabbitMQ and in-place of Kinesis
RabbitMQ Pros:
- Simple to install and manage
- Excellent routing capabilities based on rules
- Decent performance
- Cloud Installation also available (CloudAMQP)
Cons:
- Not-distributed
- Unable to scale to high loads (due to non-distributed nature)
Kafka Pros:
- Amazingly fast reads and writes (due to sequential reads and writes only)
- Does one thing and one thing only i.e. to transfer messages reliably
- Does provide load balancing of consumers via partitions of topic so real parallel-processing no ifs and buts
- No restrictions on transaction numbers unlike Kinesis
Cons:
- Complicated to setup cluster compared to rabbitmq
- Dependency on Zookeeper
- No Routing
So bottom line
- Use RabbitMQ for any simple use case
- Use Kafka if you want insane scalability and you are ready to put effort in learning kafka topics and partitions
- Use Kinesis if setting up kafka is not your cup of tea
| Kafka | Kinesis | RabbitMQ | |
| Routing | Basic (Topic Based) | Basic (Topic Based) | Advanced (Exchange based) |
| Throughput | Extremely high | Extremely high | |
| Latency | Depends on region (Not available in some regions hence Http call) | Very low | High (Compared to other 2) |
| Ease of implementation | Moderate..but setting up cluster requires effort | Moderate (but identifying number of shards can be tough) | Easy |
| Restrictions on transactions | None | 5 reads per seconds and 1000 write/sec/shard | None |
| Types of applications | High throughput | High throughput | Low to medium throughput |
As always drop me an email if still confused about your use case
Happy Coding !!
A geek's blog 