In the era of real-time enterprises, the combination of Apache Kafka and Kubernetes has become the gold standard for data streaming. However, for many organizations, managing Kafka at scale in a private or hybrid cloud environment introduces significant operational complexity.
This is where the synergy between Confluent for Kubernetes (CFK) and Red Hat OpenShift changes the game. By combining Confluent’s cloud-native automation with OpenShift’s hardened enterprise security, platform teams can finally shift their focus from “managing infrastructure” to “delivering data products.”
Phase 1: The “Pre-Flight” Checklist
OpenShift’s security model is stricter than standard K8s. Before you touch a Helm chart, you must verify:
- Security Context Constraints (SCC): Confluent pods require specific UIDs. You must install the confluent-operator SCC so pods aren’t blocked by the restricted policy.
- Storage Classes: Kafka requires high-performance block storage (RWO). Ensure your OpenShift cluster has a storage class like thin (vSphere), gp3 (AWS), or ocs-storagecluster-ceph-rbd.
- Operator Versions: Ensure you are using CFK 2.7+ and Confluent Platform 7.4+ to fully support KRaft on Kubernetes.
Phase 2: Deployment Step-by-Step
Deploying Confluent for Kubernetes (CFK) on OpenShift is a transition from manual configuration to Declarative Operations. Instead of managing individual brokers, you define the desired state of your entire data ecosystem, and the Operator ensures the cluster matches that state.
1. Establishing the “Control Plane” (The Operator)
The first step is establishing the Confluent Operator. Think of this as a “Digital SRE” living inside your OpenShift cluster.
- Role: It watches your Custom Resources (CRs) and handles the heavy lifting—automated rolling upgrades, scaling partitions, and managing certificates.
- Governance: On OpenShift, the Operator must be granted specific permissions via Security Context Constraints (SCC) to allow it to manage specialized pods (like Kafka) that require non-root user IDs.
2. The KRaft Controller Quorum (The New Brain)
With the industry move away from ZooKeeper, your deployment now starts with a KRaft Controller Quorum.
- Purpose: This group of pods handles metadata and leader elections.
- Architecture: Typically deployed as a 3-node quorum to ensure high availability. By separating Controllers from Brokers, you gain better isolation and faster recovery times during node failures.
3. The Kafka Broker Tier (The Data Plane)
The Brokers are the workhorses of your platform. In an OpenShift environment, their deployment logic shifts:
- Storage Mapping: Each broker is mapped to a Persistent Volume Claim (PVC). On OpenShift, the Operator dynamically provisions these using your cluster’s defined StorageClass (e.g., OCS, EBS, or Azure Disk).
- Anti-Affinity: To ensure maximum uptime, the deployment strategy uses Pod Anti-Affinity rules. This prevents multiple Kafka brokers from landing on the same physical worker node, protecting you against hardware failure.
4. Automated Network Exposure (Routes)
Exposing Kafka to clients outside the OpenShift cluster is traditionally a manual headache involving NodePorts or LoadBalancers.
- The CFK Advantage: The Operator integrates directly with the OpenShift Ingress/Router.
- Connectivity: It automatically generates a Bootstrap Route for initial discovery and Individual Broker Routes for direct data transfer, all secured via SNI (Server Name Indication) over TLS.
Phase 3: Bringing in the Ecosystem
A real-time platform is more than just Kafka. We need the full suite:
- Schema Registry: Essential for data governance and versioning.
- Confluent Control Center (C3): Your “Single Pane of Glass” for monitoring and management.
- Kafka Connect: For seamless integration with databases (S3, Snowflake, MongoDB).
- ksqlDB: For real-time stream processing.
Installation Tip: Deploy these as individual Custom Resources (CRs). CFK handles the “wiring”—automatically connecting Schema Registry and Connect to your Kafka brokers using internal Kubernetes services.
Key Takeaways for the Transition
- Goodbye ZooKeeper: KRaft eliminates the “split-brain” issues often found in ZK and simplifies your firewall/network rules.
- Declarative Management: Every component (Kafka, Connect, ksqlDB) is managed as Code (GitOps).
- OpenShift Native: By using Routes and SCCs, Confluent feels like a native part of the OpenShift ecosystem, not an “add-on.”
Conclusion
Ready to start your deployment? Check out the official Confluent Documentation and the OpenShift Security Examples to get your environment “production-ready.”
Author: Kundana Maddi, Solution Engineer at Alephys.