From ksqlDB to Flink SQL Without the Guesswork: Meet ksql2flink

Data Streaming · Apache Flink · Confluent Many teams that built their real-time pipelines on Confluent ksqlDB are now looking at Apache Flink SQL — whether to move to open-source Flink, Confluent Platform’s Flink (CFK/CMF), or Confluent Cloud for Flink. On paper the two look almost the same: both are SQL over Kafka topics. In practice, the migration stalls on the small details — and those small details are exactly where pipelines silently break. ksql2flink is an tested command-line tool that does this migration for you. It reads your ksqlDB streams, tables, queries, windows, and joins, and writes out ready-to-run Flink SQL — and, just as importantly, it tells you honestly about the parts a human still needs to decide. Why this migration is harder than it looks ksqlDB and Flink SQL differ in the places that matter most for correctness: Translate any of these by hand across dozens of statements and you will make a mistake somewhere. That’s the problem ksql2flink was built to remove. The core idea: convert what’s safe, flag what isn’t The tool follows one rule: nothing crashes, and nothing vanishes. Every ksqlDB object that has a well-defined Flink equivalent is converted automatically. Anything that needs human judgement — a pull query, a custom function, a pinned schema ID — is not guessed at. It lands in a migration report with a concrete next step. You always know exactly what was translated and what still needs your eyes. A quick example Here’s a typical ksqlDB pipeline — a stream and a windowed aggregate: ksql2flink turns the stream into a Flink CREATE TABLE, complete with the event-time column and watermark that ksqlDB implied: And it splits the single ksqlDB CREATE TABLE AS into the two statements Flink needs — a sink table plus an INSERT INTO — rewriting the window into a table function and adding the window bounds to the upsert key: Every one of those decisions — the watermark, the EXCEPT_KEY, the window bounds in the primary key — is a place where a hand migration typically goes wrong. Getting started in three commands Install it, then point it at a folder of .ksql files: Pick your target with a single flag: You can also migrate straight from a running ksqlDB cluster — and that path is strictly read-only. It only issues LIST/DESCRIBE and GET /info, never a CREATE, DROP, INSERT, or TERMINATE. Your production cluster is never touched. What it converts (and what it flags) Converted automatically: streams and tables, CREATE … AS SELECT (into sink DDL + INSERT), tumbling / hopping / session windows, stream-stream, stream-table and table-table joins, PARTITION BY, EXPLODE, custom types, aggregates, and the JSON, DELIMITED, Avro, JSON Schema and Protobuf formats. Flagged with an action (never silently wrong): pull queries, CREATE CONNECTOR, custom UDFs (for which it generates Java and PyFlink stubs to fill in), lifecycle statements, and pinned schema IDs. Each run ends with a MIGRATION_REPORT.md that gives you object counts, a per-object status table, a coverage percentage, and a checklist of manual actions. Verified against a real cluster This isn’t a paper exercise. Every mapping rule was executed against a real Confluent Platform 8.x + Flink 1.20 cluster — real Kafka with mTLS, a real Schema Registry, and real running Flink jobs — including a genuine cutover where the ksqlDB queries were terminated and the Flink jobs continued the pipelines correctly, verified record by record. It supports ksqlDB versions 7.2.x through 8.2.x, detects the server version automatically, and warns (rather than fails) outside that range. The bottom line Migrating from ksqlDB to Flink SQL doesn’t have to be a slow, error-prone rewrite. ksql2flink handles the mechanical translation, encodes the tricky semantics as tested rules, and hands you an honest report of everything that still needs a human. You spend your time on the decisions that matter — not on hunting down a mis-serialized key at 2 a.m. Want help planning a ksqlDB-to-Flink migration for your platform? Get in touch with the Alephys team. Author:Siva Munaga, Solution Architect (and developer of ksql2flink)  

From ksqlDB to Flink SQL Without the Guesswork: Meet ksql2flink Read More »