Getting Started with Kafka Integration in Node.js

Integrating Kafka with Node.js can be done using a Kafka client library. Here’s a step-by-step guide to integrating Kafka with Node.js:

1. Install the Kafka client library for Node.js. The most popular library is kafkajs, which you can install using NPM:

npm install kafkajs

2. Create a Kafka producer to publish messages to Kafka. Here’s an example:

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092']
});
const producer = kafka.producer();
async function publishMessage(topic, message) {
  await producer.connect();
  await producer.send({
    topic: topic,
    messages: [
      { value: message }
    ]
  });
  await producer.disconnect();
}

In this example, we create a Kafka producer using the kafkajs library, which connects to a Kafka broker running on localhost:9092. We then define a function publishMessage that takes a topic and a message, and sends the message to Kafka.

3. Create a Kafka consumer to receive messages from Kafka. Here’s an example:

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092']
});
const consumer = kafka.consumer({ groupId: 'test-group' });
async function consumeMessages(topic) {
  await consumer.connect();
  await consumer.subscribe({ topic });
  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      console.log({
        key: message.key.toString(),
        value: message.value.toString(),
        headers: message.headers,
        topic: topic,
        partition: partition
      });
    }
  });
}

In this example, we create a Kafka consumer using the kafkajs library, which connects to a Kafka broker running on localhost:9092. We then define a function consumeMessages that takes a topic and listens for messages on that topic. When a message is received, the eachMessage callback function is called, which logs the message to the console.

4. Call the publishMessage function to send messages to Kafka:

publishMessage('my-topic', 'Hello, Kafka!');

This sends a message with the value “Hello, Kafka!” to the “my-topic” topic in Kafka.

5. Call the consumeMessages function to receive messages from Kafka:

consumeMessages('my-topic');

This listens for messages on the “my-topic” topic in Kafka and logs them to the console.

That’s it! With these steps, you should be able to integrate Kafka with Node.js using the kafkajs library.