The technical principles of the Apache Kafka framework and the explanation of the Java class library implementation
Apache Kafka is a high -performance, scalable distributed flow processing platform, which is widely used in the construction of real -time data flow applications.This article will explain the technical principles of the Apache Kafka framework and the implementation of the Java library.
Technical principle:
1. Data release and subscription model: Kafka uses the release and subscription model for data transmission.Data occurrence is called a producer, and the data is published to one or more topics (Topic) of the Kafka cluster; data receivers are called consumers, subscribe to data from the designated topic and process it.The theme is the classification of messages, and each message includes a key value pair.
2. Distributed storage: Kafka uses a distributed storage architecture to store data scattered on multiple nodes.Each theme is divided into multiple partitions, and the order of storing data in each partition is guaranteed, but the order of data of the entire theme is not guaranteed.Partitions can be replicated on multiple servers to improve reliability.
3. Producer API: The producer API allows the application to publish the message to the Kafka cluster.The message sent by the producer is added to a partition of the theme. You can select a specific partition according to the key pair of keys, or you can use rotation or random way to distribute the message to each partition on average.
4. Consumer API: Consumer API is used to read data from the Kafka cluster and process it.Consumers can subscribe to one or more themes and pull data from each partition.Consumers track their consumption progress by regularly sending offset (offset) requests to the server.
5. Broker and cluster: Kafka cluster consists of multiple server nodes (Broker). Each broker is an independent Kafka server.Each broker is responsible for processing the client's request, storage, and replication data, and the Broker in the cluster can communicate with each other and perform load balancing.After the client is connected with any broker, you can communicate with the entire cluster.
Java class library implementation:
The following is the example code of producers and consumers using Kafka's Java class library:
1. Producer implementation:
import org.apache.kafka.clients.producer.*;
import java.util.Properties;
public class KafkaProducerExample {
public static void main(String[] args) {
Properties props = new Properties();
props.put ("Bootstrap.servers", "LocalHost: 9092"); // kafka cluster address
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer<>(props);
String topic = "my_topic";
String message = "Hello Kafka!";
ProducerRecord<String, String> record = new ProducerRecord<>(topic, message);
producer.send(record, new Callback() {
@Override
public void onCompletion(RecordMetadata metadata, Exception e) {
if (e != null) {
e.printStackTrace();
} else {
System.out.println("Message sent to partition " + metadata.partition()
+ ", offset " + metadata.offset());
}
}
});
producer.close();
}
}
The above example creates a producer and sends the message to the theme of "My_topic".
2. Consumer implementation:
import org.apache.kafka.clients.consumer.*;
import java.util.Arrays;
import java.util.Properties;
public class KafkaConsumerExample {
public static void main(String[] args) {
Properties props = new Properties();
props.put ("Bootstrap.servers", "LocalHost: 9092"); // kafka cluster address
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("group.id", "my_group");
Consumer<String, String> consumer = new KafkaConsumer<>(props);
String topic = "my_topic";
consumer.subscribe(Arrays.asList(topic));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records) {
System.out.println("Received message: " + record.value()
+ ", from partition " + record.partition()
+ ", offset " + record.offset());
}
}
}
}
The above example creates a consumer, subscribing to the theme of "My_topic", and receive messages from it.
This article briefly introduces the technical principles of the Apache Kafka framework and the example of using the Java class library to achieve producers and consumers.By understanding the basic concepts and usage methods of Kafka, you can better understand and apply the framework.
