Part 4 of the Apache Kafka Series
In the previous three parts of this series, we covered why Apache Kafka exists and why event-driven architecture matters, how Kafka’s internal design delivers scale and resilience, and how Kafka achieves exceptional throughput through log-structured storage, the page cache, and zero-copy transfers. We now move up the stack.
Kafka on its own handles reliable, high-throughput event transport. But in production, that is only half the picture. Most organisations need to get data into Kafka, process and enrich it, and deliver it to downstream systems – all without writing mountains of custom integration code.
This fourth part explores three components that transform Kafka from a simple message broker into a real-time data platform: Kafka Connect, Kafka Streams, and ksqlDB.
1. When to Use Which Tool
Kafka Connect moves data between Kafka and external systems. Kafka Streams handles complex, stateful processing in Java applications, while ksqlDB provides SQL-based stream processing. Use Connect for integration, Streams for custom business logic, and ksqlDB for rapid SQL transformations and operational views. Most production platforms combine all three.
| Need | Recommended Tool |
| Connect a database, S3, or search engine to Kafka | Kafka Connect |
| Move data between systems without custom code | Kafka Connect |
| Apply simple field mappings or filtering during ingestion | Kafka Connect (SMTs) |
| Write stateful application logic over event streams | Kafka Streams |
| Compute aggregations, joins, or windowed results | Kafka Streams |
| Enrich events from a continuously updated reference data set | Kafka Streams |
| Prototype a real-time SQL view quickly | ksqlDB |
| Expose a materialised table for an operational dashboard | ksqlDB |
| Apply lightweight filtering or routing with SQL syntax | ksqlDB |
2. Kafka as a Platform, Not Just a Queue
In Part 1, we established that Kafka is not „just another queue“. Its commit-log design enables retention, replayability, and decoupling at a level traditional brokers cannot match.
But Kafka on its own does not solve the full data lifecycle. In practice, organisations need:
- a way to ingest data from databases, object stores, and search engines,
- a way to process events – filter, enrich, aggregate, and join,
- a way to deliver processed data to warehouses, dashboards, or downstream services,
- a way to query streaming data without writing a Java application.
Kafka Connect, Kafka Streams, and ksqlDB each address one of these layers. Together they form an ecosystem that makes Kafka a serious operational data platform, not merely a transport layer.
Kafka Connect integrates source and destination systems, Kafka Streams executes stateful application logic, and ksqlDB exposes streaming transformations through SQL.
3. Kafka Connect: Integration Without Custom Code
What It Is
Kafka Connect is a framework for connecting Kafka to external systems. It runs as a distributed service on top of a Kafka cluster and executes connectors – standardised plug-ins that know how to read from or write to a specific external system.
The two connector types are:
- Source connectors – pull data into Kafka from external systems such as databases, file systems, APIs.
- Sink connectors – push data out of Kafka to external destinations such as data warehouses, object stores, search engines.
Typical Use Cases
A well-operated Kafka Connect cluster handles integrations that would otherwise require custom producer orconsumer applications:
- PostgreSQL → Kafka (change data capture via a source connector),
- Kafka → Amazon S3 (archiving events for long-term storage or batch analytics),
- Kafka → Elasticsearch / OpenSearch (near-real-time search indexing),
- Kafka → Snowflake or BigQuery (streaming analytics pipelines),
- Kafka → another database (data synchronisation across systems).
The connector ecosystem is large. Confluent, Debezium, and the broader open-source community maintain hundreds of production-ready connectors, significantly reducing the amount of custom integration code organisations need to write.
Why This Matters Architecturally
The main architectural benefit is standardisation. Instead of maintaining bespoke producer and consumer applications for each integration, engineering teams manage a configuration-driven layer with consistent monitoring, error handling, and restart behaviour.
This also improves operational predictability. A connector’s health, lag, and throughput are surfaced through the same monitoring stack as the rest of the Kafka cluster. When a sink connector falls behind, it appears as consumer lag – a familiar signal that engineering teams already know how to respond to.
Expert insight: Kafka Connect is not a data transformation engine. For complex transformations, it is better to route raw events into a topic, apply transformations using Kafka Streams or a dedicated processing layer, and then sink the enriched data. Attempting to use Single Message Transforms (SMTs) as a substitute for proper stream processing leads to unmaintainable connector configurations.
What Kafka Connect Does Not Replace
Kafka Connect is not a substitute for architectural thinking. Connector-based integrations still require:
- a well-defined schema strategy (how events are structured and versioned),
- error-handling logic (what happens when a message cannot be delivered),
- monitoring and alerting (connector lag, task failures, rebalance events),
- security configuration (credentials, TLS, topic-level access control),
- governance (who owns each connector, where data flows).
No-code integration reduces development effort; it does not eliminate the need for architectural discipline.
4. Kafka Streams: Stateful Stream Processing in Java
What It Is
Kafka Streams is a Java library – not a separate cluster – that allows applications to consume events from Kafka topics, process them, and produce results back into Kafka. Because it is a library rather than a framework, it runs inside your existing Java application without requiring any additional infrastructure components.
This is a deliberate design choice. Kafka Streams is simpler to deploy, version, and scale than a separate processing cluster. It scales horizontally by running multiple instances of the same application, with Kafka automatically distributing partitions across them.
Moving Beyond Data Transfer
In Part 2, we described Kafka as a distributed commit log. Connect handles moving data into and out of that log. But many real-world scenarios require more than simply moving data – they require deciding what to do with it.
This is where Kafka Streams becomes relevant. It supports:
- Filtering – dropping events that do not match a condition,
- Transformations – reshaping events from one schema to another,
- Enrichment – joining a stream against a reference dataset (e.g. customer profile),
- Aggregations – computing counts, sums, or averages over a set of events,
- Joins – combining two event streams based on a shared key,
- Windowing – grouping events into time-based buckets (tumbling, hopping, session windows),
- Stateful computation – maintaining a continuously updated view of the current state.
The Role of State
The distinction between stateless and stateful processing is fundamental to understanding where Kafka Streams adds value.
A stateless transformation sees only the current event. It can filter or reformat, but it cannot count, compare, or remember. Stateful processing maintains a local state store – backed by a Kafka changelog topic – that survives restarts and is replicated for fault tolerance.
Without state, stream processing is a pipeline. With state, it becomes a decision system.
Expert insight: Kafka Streams state stores are backed by RocksDB by default, persisted locally on each application instance and mirrored to a compacted changelog topic. This means that after a failure, the rebuilt state comes from Kafka rather than a remote database – keeping recovery fast and operational dependencies low. The trade-off is disk usage on the processing nodes, so audit this carefully before deploying stateful topologies in production.
When to Use Kafka Streams
Kafka Streams is the right tool when:
- processing logic is complex enough to warrant an application (not just configuration),
- results need to feed to be fed back into Kafka for further processing,
- stateful computation – aggregation, joins, windowing – is required,
- the team is comfortable with Java and wants tight integration with their existing services.
5. ksqlDB: SQL Over Streams
What It Is
ksqlDB is a streaming database built on top of Kafka Streams. It exposes Kafka topics as SQL-queryable streams and tables, allowing engineers to write continuous queries in familiar SQL syntax rather than Java code.
Under the hood, ksqlDB compiles SQL statements into Kafka Streams topologies and runs them as persistent server-side processes. The results materialise back into Kafka topics and, for table queries, into in-memory or RocksDB-backed state.
What It Is Good For
ksqlDB is effective for:
- rapid prototyping of real-time logic without writing a full application,
- simple to moderate aggregations – counts, averages, grouped summaries over time windows,
- materialised views – continuously updated tables that represent current state,
- stream filtering and routing – selecting subsets of a topic and writing them elsewhere,
- analytics and operational queries – exposing real-time metrics to dashboards or alerts.
It is particularly useful in organisations where not all teams have Java expertise. A data engineer or analyst comfortable with SQL can build meaningful real-time pipelines without involving a back-end developer.
What ksqlDB Is Not
ksqlDB is not a relational database and should not be treated as one.
- It does not support arbitrary ad-hoc queries over historical data.
- It does not replace a data warehouse for retrospective analysis.
- It is not appropriate for transactional workloads.
- Complex business logic – multi-step enrichment, sophisticated branching, custom state – is better expressed in Kafka Streams.
Expert insight: ksqlDB materialised state is derived from Kafka topics and local state stores. Dropping or replacing a table may remove its local materialisation and terminate the associated query, but it does not necessarily delete the underlying Kafka topic. Recovery therefore depends on topic retention, changelog availability, and the deployment procedure. Treat queries, schemas, and topics as versioned production assets.
6. Practical Use Cases
FinTech: Real-Time Transaction Risk Scoring
A payment processor receives hundreds of transactions per second. Each transaction needs to be scored for risk before the payment is authorised.
The architecture looks like this:
- Payment events flow into a Kafka topic via a source connector or producer application.
- A Kafka Streams application reads each event and enriches it with the customer’s recent transaction history, maintained in a local state store that is updated continuously.
- The application computes a risk score based on velocity, geography and behavioural patterns.
- High-risk transactions are routed to an alert topic for manual review or automatic blocking.
- Scores are written to a database via a Kafka Connect sink connector for downstream reporting.
The result is a risk decision completed within milliseconds of the transaction being submitted, without blocking the payment system and without requiring the risk engine to call external services synchronously.
Logistics: Real-Time Shipment Tracking
A logistics operator tracks thousands of vehicles, parcels, and warehouse events simultaneously.
- GPS updates, scanner events, and depot status changes flow into dedicated Kafka topics.
- Kafka Streams aggregates these events per shipment, combining vehicle position, warehouse scans, and estimated time of arrival into a continuously updated delivery state.
- ksqlDB exposes a materialised table of current shipment status for operational dashboards.
- A Kafka Connect sink connector delivers late or anomalous shipment events to an operations alerting system.
The operational team sees the current shipment state in real time. Exception handling – missed pickups, route deviations, delayed deliveries – is detected automatically without waiting for batch reports.
The three tools are complementary, not competing. Production systems frequently combine all three: Connect for ingestion, Streams for business logic, and ksqlDB for operational visibility.
8. Common Mistakes
Treating Kafka Connect as a full ETL engine. Connect is designed for transport and lightweight transformation. Using it to perform complex business logic through chained SMTs results in configurations that are difficult to test, debug, and maintain.
Writing a custom connector before evaluating existing ones. The connector ecosystem is extensive and building a custom connector is an ongoing maintenance commitment. Evaluate available connectors thoroughly before committing to custom development.
Ignoring schema management. Events that change shape without a versioning strategy will break downstream consumers. Schema Registry and Avro or Protobuf schemas are standard practice in production deployments.
Under-investing in monitoring. Connector task failures, consumer group lag, and Streams topology errors require dedicated alerting. Without it, pipeline failures are discovered through downstream data quality issues rather than infrastructure signals.
Using ksqlDB as a substitute for a relational database. ksqlDB is a continuous query engine over streams. It is not designed for ad-hoc historical queries, complex transactional workloads, or full SQL join semantics.
Optimising for throughput before validating the data flow design. Getting the data model, topic structure, and processing logic right first makes subsequent performance tuning straightforward. Optimising a poorly designed topology is far more costly.
Key takeaways
- Kafka itself handles transport. Kafka Connect, Kafka Streams, and ksqlDB extend it into a complete real-time data platform.
- Kafka Connect solves integration – connecting Kafka to external systems through standardised, configuration-driven connectors without custom code.
- Kafka Streams solves processing – enabling stateful computations, enrichments, aggregations, and joins within a Java application that scales with Kafka’s partition model.
- ksqlDB solves accessibility – exposing Kafka topics as SQL-queryable streams and tables for teams who need real-time logic without writing Java.
- The three tools are complementary. Production architectures routinely combine all three within the same pipeline.
- Integration without a schema strategy, monitoring plan, and governance model creates operational risk regardless of the tooling chosen.
Conclusion
Apache Kafka was designed to excel at one thing: reliable, high-throughput, decoupled event transport. Kafka Connect, Kafka Streams, and ksqlDB build on that foundation to form a complete platform for real-time data integration, processing, and querying.
Together, they enable organisations to move from isolated batch pipelines to continuous, event-driven data flows that respond to what is happening in the business as it happens – not hours later.
Choosing the right tool for each layer matters. Connect is not a processing engine. Streams is not a reporting database. ksqlDB is not a general-purpose SQL system. Understanding these boundaries is what separates architectures that remain maintainable at scale from those that accumulate technical debt.
In Part 5, we will examine the intersection of Kafka and databases in depth – specifically how Kafka integrates with PostgreSQL through Change Data Capture (CDC), how Debezium turns database transaction logs into event streams, how the Outbox Pattern solves the dual-write problem, and what modern data architectures look like when databases and Kafka operate as complementary layers rather than competing technologies.
Baremon helps organisations design, implement, and operate Apache Kafka deployments – from initial architecture through to production-grade stream-processing pipelines. Contact us if you are evaluating Kafka Connect, Kafka Streams, or ksqlDB for your platform.
