Pub/Sub with Kafka and Microservices.

In your Microservices architecture , there may be a need to call different services from your service/application. So how does the communication happens between these microservices? There are mainly two ways to communicate with other microservices.
1. RPC(Remote Procedure Call) : Basically it is request/response model and synchronous. When client make request to the service ,the service should respond with response back. Best example to implement RPC between microservices is REST.
2. Messaging : The second best way to implement communication between microservices is Messaging. It can be synchronous and async too.

So here Kafka can be used as a messaging protocol and can make microservices communicate with each other.
KAFKA : Kafka is Pub/Sub streaming platform where you can push messages to the kafka topic usually this is achieved by applications called Producer. Then the data can be consumed by applications called Consumers. Kafka uses the binary TCP protocol for data transmission so the performance will be solid.
Producer : API offered by kafka to publish messages to the kafka.
Consumer: API offered by kafka to access data from kafka and process.

Kafka itself is a big topic to explore, what we covered here is not even introduction.But i have given below sample application on how to send JSON message to the kafka server.

How to send JSON data accessed from database to the kafka using Springboot.
For this application i used Springboot,Spring Data JPA,postgres,kafka server,zookeeper server. You can download kafka from here.

Start zookeeper server:
“bin/zookeeper-server-start.sh config/zookeeper.properties”
So after executing the above command from terminal you will see astack of logs as below.It means zookeeper started. Zookeeper used to manage Kafka brokers and other config related data.

Start Kafka server:
“bin/kafka-server-start.sh config/server.properties”

Publish message:
Here we use spring based java rest api to publish the data to the kafka using GET request. http://localhost:8080/publish
We need to start the spring boot application using “mvn spring-boot:run”
You can directly clone the project and run as spring-boot app from https://github.com/shivakumarksk/kafkapub-sub

Now hit the GET end point given above from Postman client/browser.

We will see String message return “Published message successfully”

If we consume the message from kafka uisng terminal command
“bin/kafka-console-consumer.sh — bootstrap-server localhost:9092 — topic events-rest — from-beginning”

Total one JSON message processed.

So this the simple Helloworld kind of application to publish the data to the kafka. Going forward complex topics will be discussed on kafka and Microservices.