It was brought to my attention that I mispelled Kinesis. I’ve updated the text to reflect this change. Thx for bearing with me.
Change how you look at Kafka vs Kinesis. Kinesis isn’t all it’s meant to be.
Details of this comparison are based on the usage predictions given in the chart below. Average message sizes are assumed to be between 1K and 5K. EC2 Pricing is based on 1 year full up front reserved pricing in USD.
Pricing
For small workloads, Kinesis is much more cost effective, but it can easily exceed the cost of an EC2 cluster built for running Kafka for larger and more scalable workloads. An estimated cost for the minimum Kinesis specification for the predicted workload is approximately $240 per month for messages of 1K in size, but around $565 per month for messages averaging 5K in size. This represents an increase of 13 shards.
Kafka, priced per node, can cost about $440 monthly for a 3-node cluster using i3.large nodes, and $585 for a 5-node cluster of the same type. (i3 nodes are optimized for I/O which will be best suited for Kafka nodes). A 3-node cluster can easily handle several million of messages per hour.
Based on these figures, Kinesis is arguably more cost effective when the total number of shards is less than 15.
Streams and Shards vs. Topics and Partitions
In Kinesis, a stream is essentially a single named pipeline for transmitting data. It would be analogous to a Kafka topic. Similarly, a Shard is equivalent to a Partition. Message sequence is only guaranteed for Shards and Partitions.
Provisioning and scaling
While the basic architecture of these two solutions is similar, the pricing considerations of Kinesis force you to funnel your data through a limited number of shards. With a Kafka solution, you will not be charged based on the number of partitions or topics you have. For each Stream, you must have at least one shard, and parallelism for processing messages on the stream is limited to the number of shards you have. While you can have multiple independent consumer types, instances of the same consumer type is limited to the number of shards on the stream.
This difference is best illustrated with the following scenario:
Assume that the user experience stream is currently allocated with 1 shard. On a given day, a new release is deployed that increases the number of user experience events such that one shard is not sufficient. The shard count is scaled and a new consumer must also be deployed that is configured to handle the new shard. Each shard is essentially a new “sub-stream” that requires an independent consumer. This means that you will end up overprovisioning consumers if you overprovision shards.
For Kafka, assume the same stream (called a “topic” in Kafka). A topic in Kafka would likely be overprovisioned for partitions, since adding partitions to existing topics can be intrusive. This overprovisioning predicts this scenario and prevents it from being an issue. Since Kafka auto-balances over consumers in a consumer group, the number of consumers can be scaled on demand without deploying any configuration changes.
Kinesis scaling is also throttled such that you are limited on your scaling operations. The current rules are that you cannot:
- scale more than twice per rolling 24 hour period
- scale up above double your current shard count
- scale down below half your current shard count
- scale up above 200 shards in a stream
- scale a stream with more than 200 shards down unless the result is less than 200 shards
- scale up above the shard limits for your account
This means that if your current shard count is 1, you cannot scale to more than 4 within a 24 hour period. While this appears to be an unlikely scenario, new features or usage bursts that trigger an order of magnitude increase in messages could cripple the system until the shards can be scaled appropriately.
Scaling Kafka is as easy as adding another Kafka node to the cluster. If you have adopted an “infrastructure as code” strategy, this is as simple as updating a script and running it.
In Kafka, it is also possible to increase the number of partitions on a topic, but this is not an operation meant to be done on a frequent basis. It is also unnecessary in most cases if you overprovision partitions on your topics. 30 is a good number of partitions for Kafka, since it allows you to scale consumers to 2, 3, 5, 6, 10, 15 and 30 and have equal load across consumers for a well-balanced topic.
Flexibility
With Kinesis, you will feel the constraints of limiting the number of streams and shards to reduce the total shard count and keep it cost effective. A Kafka cluster will not be limited by these factors, rather only by the total number of messages across all topics. It is also a common pattern in Kafka to have a consumer simply transform a message from one format to another and republish on another topic. This may be useful as an aggregation pattern for transforming data messages from CDC into more structured business event messages. It can also be useful to filter specific business messages from the business event topic into less detailed topics that don’t require every transaction. For example, one topic could only contain sales, another may have only new inventory notifications.
Retention Period
Message streams are typically for real-time updates and maintaining messages over a period of time for replay is not it’s general use case. However, it is common practice to use streams for disaster recovery and for initializing new microservices.
Kinesis has a maximum retention period of 7 days, which is sufficient for a disaster recovery scenario, but likely not enough for initializing a new service. Kafka’s retention policy is configurable and can be essentially indefinite. (While “indefinite” isn’t an actual value, the configuration settings allow values that can be years … even thousands of years).
Software Development Life Cycle
Developing for Kinesis will introduce new challenges. Additional streams will be needed for development teams or sometimes individual developers. Depending on the number of teams and developers, this could drastically increase the cost of Kinesis.
Developers on Kafka can run a single node cluster on their development machines as Docker containers. This means each developer can have their own instance without any impact to others.
Operations and Management
Kinesis, being a hosted service, has very limited operations and management cost. Ideally it should be near none. The required focus on cost management implies strict monitoring of shard performance to scale appropriately in a reasonable time for the business needs. This could be scripted; however, the imposed scaling restrictions could still impact the ability to react quickly enough.
Since Kafka is deployed to EC2 instances, it implies more management than a hosted service. Adopting an “infrastructure as code” strategy will minimize the impact and can provide much of the same benefits of a hosted service. If the Kafka nodes are built using scripts, patching can be as simple as terminating a node and allowing it to be rebuilt from a newer base image. Kafka’s built-in cluster replication will ensure that no data is lost. This can be applied one node at a time until the cluster is completely patched. This can also be automated so that the cluster is completely refreshed on a schedule you dictate.
Summary
