Kafka for IoT: High-Throughput Streaming with React Clients and CouchDB Persistence

Apache Kafka provides a high-throughput, fault-tolerant streaming backbone for ingesting telemetry from IoT devices, processing events in real time, and persisting data to databases like CouchDB. This specification outlines performance characteristics, architecture design, and includes a React Native producer example for fleet tracking.

---

Performance Capabilities

• Throughput:
  • Single broker: ~200,000 messages/sec (~1 KB payload).
  • 5-broker cluster: scales linearly to 1–2M messages/sec.
• Latency: under 10 ms in LAN deployments, under 50 ms across regions.
• Durability: Replication factor 3 with acks=all ensures no message loss.
• Retention: Topics configurable for days or weeks, enabling replay and backfill.

---

Core Components

1. IoT Device (Producer)
   Publishes telemetry (e.g., GPS data) to Kafka.

2. Kafka Brokers (Backbone)
   Provide distributed, durable message transport.

3. React Client App (Consumer)
   Receives updates through a WebSocket/REST bridge for live dashboards.

4. CouchDB Sink
   Stores JSON documents for querying, replication, and offline-first apps.

---

Example Use Case: Fleet Tracking

• Scale: 5,000 vehicles × 1 update/sec → 300,000 messages/min.
• Workflow:
  • Each vehicle publishes fleet.position events with location & speed.
  • Kafka brokers partition data by vehicle_id to preserve ordering.
  • React dashboard shows live fleet movement.
  • CouchDB stores event logs for compliance and historical analysis.

---

Custom Producer Example (React Native)

A React Native app can act as a Kafka producer (through an API gateway). Below is a simplified example for a fleet.position event:

// fleetProducer.ts
import Geolocation, { GeoPosition } from "react-native-geolocation-service";

async function getPosition(): Promise<GeoPosition> {
  return new Promise((resolve, reject) => {
    Geolocation.getCurrentPosition(resolve, reject, {
      enableHighAccuracy: true,
      timeout: 10000,
    });
  });
}

export async function sendFleetPosition(device_id: string, token: string) {
  const pos = await getPosition();

  const event = {
    device_id,
    ts: new Date().toISOString(),
    use_case: "fleet.position",
    schema: "v1",
    payload: {
      lat: pos.coords.latitude,
      lon: pos.coords.longitude,
      speed_kmh: (pos.coords.speed ?? 0) * 3.6, // m/s → km/h
      heading_deg: pos.coords.heading ?? undefined,
      status: "delivering",
    },
  };

  await fetch("https://your-gateway.example.com/ingest", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "Authorization": `Bearer ${token}`,
    },
    body: JSON.stringify({ events: [event] }),
  });
}

How it works:

• The mobile app captures GPS data.
• It wraps the data in a fleet.position event.
• The event is POSTed to a secure ingest API.
• The gateway publishes to a Kafka topic (fleet.updates.v1).
• Kafka guarantees ordering per vehicle (partitioned by device_id).
• CouchDB consumes events for durable storage.

---

Benefits

• Real-Time Responsiveness: under 50 ms from device → dashboard.
• Scalable Messaging: Millions of messages per minute across fleets.
• Durable Persistence: CouchDB provides JSON-based storage and replication.
• Cross-Platform Apps: React Native clients run on iOS and Android with the same code.

---

Kafka Topics (CLI Example)

# Fleet positions: 12 partitions for ~50k devices at 1 msg/s (tune to your load)
kafka-topics.sh --create --topic fleet.updates.v1 --partitions 12 --replication-factor 3 \
  --config cleanup.policy=delete --config retention.ms=604800000  # 7 days

kafka-topics.sh --create --topic coldchain.temp.v1 --partitions 6 --replication-factor 3 \
  --config retention.ms=259200000  # 3 days

kafka-topics.sh --create --topic driver.panic.v1 --partitions 3 --replication-factor 3 \
  --config retention.ms=1209600000 # 14 days

---

(Optional) Kafka → CouchDB Sink

If you want the events to persist automatically into CouchDB, you can configure a Kafka Connect sink. Example (pseudo-config):

{
  "name": "couchdb-sink",
  "config": {
    "connector.class": "com.github.couchdb.KafkaCouchDBSinkConnector",
    "tasks.max": "2",
    "topics": "fleet.updates.v1,coldchain.temp.v1,driver.panic.v1",
    "couchdb.url": "http://admin:password@couchdb:5984",
    "couchdb.db.mapping": "fleet.updates.v1:fleet_updates,coldchain.temp.v1:coldchain_temp,driver.panic.v1:driver_panic",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter"
  }
}

Note: There isn’t an official CouchDB sink in Kafka Connect core. Options:

• Use a community CouchDB connector.
• Or land events into a compacted topic and run a Kafka Streams / ksqlDB / Flink job that writes to CouchDB via its HTTP API.

---

Architecture Flow

Vehicle Device (Producer) → Kafka Brokers (Streaming Backbone) → React Dashboard (Consumer via WebSocket/REST) → CouchDB (Durable Storage & Analytics)

This pattern scales to tens of thousands of IoT devices, supports instant dashboards, and provides long-term event persistence.

---