Apache Kafka Bits and Pieces
In this article I like to list some resources that you might find useful when starting with Apache Kafka on AWS. This could be with Amazon MSK, but also for all other Kafka projects.
- I recorded a new web cast that shows many new feature of Amazon Managed Streaming for Kafka with short live demos: cluster creation and security, custom configurations, integration with CloudWatch, resizing of broker storage and much much more. You can get the slides here.
- The recording is online now:
- After watching the recordings, I recommend to invest some time reading the Amazon MSK documentation.
- kafkacat is a great open-source tool that does not depend on Java and does not require you to download Apache Kafka. You can easily install or build it.
- Amazon Kinesis can be an option for streaming data on AWS.
- This LinkedIn Blog Posting from 2013 is good read. Learn how it all started.
- How many Kafka Partitions per Broker should you configure?
- Get a head start with Event Sourcing, M. Fowler
- Learn about CQRS
- During my webcast at the AWS EMEA Web Days, you asked a number of questions. I repeat them here. Maybe they are useful to more of you:
Q: What’s the difference between a message broker and a commit log in the scope of Kafka?
A:
commit log = high level, conceptual data structure (in reality it is often a partitioned, replicated commit log).
message broker = a message broker is a Kafka process assigned to a machine. the logical commit log is partitioned across various brokers for increased throughput.
___Q: What does MSK stand for ?
A: It stands for Amazon Managed Streaming for Apache Kafka. It is the managed “Apache Kafka Service” that AWS offers.
___Q: Hoping to hear about migrating an existing on-prem Kafka cluster to MSK, any tooling?
A: I won’t have the time to talk about it in this webcast unfortunately. You can mirror or migrate your cluster using MirrorMaker, which is part of Apache Kafka. For example, you can use it to migrate your Apache Kafka cluster to Amazon MSK or to migrate from one MSK cluster to another. For more details, please check the Amazon MSK documentation at https://docs.aws.amazon.com/msk/latest/developerguide/migration.html
___
