Cloudera

Data Streaming · Apache Flink · Confluent Many teams that built their real-time pipelines on Confluent ksqlDB are now looking at Apache Flink SQL — whether to move to open-source Flink, Confluent Platform’s Flink (CFK/CMF), or Confluent Cloud for Flink. On paper the two look almost the same: both are SQL over Kafka topics. In practice, the migration stalls on the small details — and those small details are exactly where pipelines silently break. ksql2flink is an tested command-line tool that does this migration for you. It reads your ksqlDB streams, tables, queries, windows, and joins, and writes out ready-to-run Flink SQL — and, just as importantly, it tells you honestly about the parts a human still needs to decide. Why this migration is harder than it looks ksqlDB and Flink SQL differ in the places that matter most for correctness: Translate any of these by hand across dozens of statements and you will make a mistake somewhere. That’s the problem ksql2flink was built to remove. The core idea: convert what’s safe, flag what isn’t The tool follows one rule: nothing crashes, and nothing vanishes. Every ksqlDB object that has a well-defined Flink equivalent is converted automatically. Anything that needs human judgement — a pull query, a custom function, a pinned schema ID — is not guessed at. It lands in a migration report with a concrete next step. You always know exactly what was translated and what still needs your eyes. A quick example Here’s a typical ksqlDB pipeline — a stream and a windowed aggregate: ksql2flink turns the stream into a Flink CREATE TABLE, complete with the event-time column and watermark that ksqlDB implied: And it splits the single ksqlDB CREATE TABLE AS into the two statements Flink needs — a sink table plus an INSERT INTO — rewriting the window into a table function and adding the window bounds to the upsert key: Every one of those decisions — the watermark, the EXCEPT_KEY, the window bounds in the primary key — is a place where a hand migration typically goes wrong. Getting started in three commands Install it, then point it at a folder of .ksql files: Pick your target with a single flag: You can also migrate straight from a running ksqlDB cluster — and that path is strictly read-only. It only issues LIST/DESCRIBE and GET /info, never a CREATE, DROP, INSERT, or TERMINATE. Your production cluster is never touched. What it converts (and what it flags) Converted automatically: streams and tables, CREATE … AS SELECT (into sink DDL + INSERT), tumbling / hopping / session windows, stream-stream, stream-table and table-table joins, PARTITION BY, EXPLODE, custom types, aggregates, and the JSON, DELIMITED, Avro, JSON Schema and Protobuf formats. Flagged with an action (never silently wrong): pull queries, CREATE CONNECTOR, custom UDFs (for which it generates Java and PyFlink stubs to fill in), lifecycle statements, and pinned schema IDs. Each run ends with a MIGRATION_REPORT.md that gives you object counts, a per-object status table, a coverage percentage, and a checklist of manual actions. Verified against a real cluster This isn’t a paper exercise. Every mapping rule was executed against a real Confluent Platform 8.x + Flink 1.20 cluster — real Kafka with mTLS, a real Schema Registry, and real running Flink jobs — including a genuine cutover where the ksqlDB queries were terminated and the Flink jobs continued the pipelines correctly, verified record by record. It supports ksqlDB versions 7.2.x through 8.2.x, detects the server version automatically, and warns (rather than fails) outside that range. The bottom line Migrating from ksqlDB to Flink SQL doesn’t have to be a slow, error-prone rewrite. ksql2flink handles the mechanical translation, encodes the tricky semantics as tested rules, and hands you an honest report of everything that still needs a human. You spend your time on the decisions that matter — not on hunting down a mis-serialized key at 2 a.m. Want help planning a ksqlDB-to-Flink migration for your platform? Get in touch with the Alephys team. Author:Siva Munaga, Solution Architect (and developer of ksql2flink)  

Introduction Organizations using CDH for their Big Data requirements typically rely on Cloudera Navigator for features like search, auditing, and data lifecycle management. However, with the advent of CDP (Cloudera Data Platform), Apache Atlas replaces Navigator, offering enhanced data discovery, cataloging, metadata management, and data governance. In this guide, we will explore the differences between Cloudera Navigator and Apache Atlas, explain why an organization may need these tools, and outline the steps for migrating from Navigator to Atlas. What is Cloudera Navigator? Cloudera Navigator is the tool that powers data discovery, lineage tracking, auditing, and policy management within CDH. It helps businesses efficiently manage large datasets, ensuring regulatory compliance, data governance, and data security. Why Do Organizations Use Cloudera Navigator? Self-Service Data Access: Enables business users to find and access data efficiently. Auditing and Security: Tracks all data access attempts, ensuring security and compliance. Provenance and Integrity: Allows tracing data back to its source to ensure data accuracy and trustworthiness. What is Apache Atlas? Apache Atlas, introduced in CDP, enhances data governance, offering rich metadata management, data classification, and lineage tracking. Key Features of Apache Atlas: Data Classification: Classify data entities with labels (e.g., PII, Sensitive). Lineage Tracking: Visualize the flow of data through its transformations. Business Glossary: Create and manage definitions for business terms, enabling common understanding across teams. Why Switch to Atlas? Organizations migrating to CDP benefit from the advanced governance capabilities provided by Atlas: Enhanced Metadata Management: Covering broader data entities and sources. Modern Data Governance: Better support for emerging data governance needs. Better Integration: Works seamlessly with CDP components like Apache Ranger for auditing and security. Comparison of Cloudera Navigator and Apache Atlas Feature Cloudera Navigator Atlas Metadata Entities HDFS, S3, Hive, Impala, Yarn, Spark, Pig, Sqoop HDFS, S3, Hive, Impala, Spark, HBase, Kafka Custom Metadata Yes Yes Lineage Yes Yes Tags Yes Yes Audit Yes No** (Handled by Ranger in CDP) Key Notes for Migration: HDFS Entities in Atlas are only referenced by services like Hive. Sqoop, Pig, MapReduce, Oozie, and YARN metadata are not migrated to Atlas. Audits are managed by Apache Ranger in CDP. Steps for Sidecar Migration from Navigator to Atlas 1. Pre-Requisites: Ensure the last Navigator purge is complete. Check disk space: For every million entities, allocate 100MB of disk space. 2. Extracting Metadata from Navigator Log into the Navigator host. Ensure JAVA_HOME and java.tmp.dir are configured correctly. Locate the cnav.sh script (typically at /opt/cloudera/cm-agent/service/navigator/cnav.sh). Run the script with the following options: nohup sh /path/to/cnav.sh -n http://<Navigator Hostname>:7187 -u <user> -p <password> -c <Cluster Name> -o <output.zip> For error handling, use the repair option: nohup sh /path/to/cnav.sh -r ON -n http://<Navigator Hostname>:7187 -u <user> -p <password> -c <Cluster Name> -o <output.zip> & 3. Transforming Metadata for Atlas Locate the nav2atlas.sh script (typically at /opt/cloudera/parcels/CDH/lib/atlas/tools/nav2atlas/nav2atlas.sh). Set JAVA_HOME and update the atlas-application.properties file with the following atlas.nav2atlas.backing.store.temp.directory=/var/lib/atlas/tmp Run the transformation script: nohup /path/to/nav2atlas.sh -cn cm -f /path/to/cnavoutput.zip -o /path/to/nav2atlasoutput.zip 4. Loading Data into Atlas Increase the Java Heap size for HBase hbase_reginserver_java_heapsize to 31Gb Increase the Java Heap size for Solr solr_java_heapsize to 31Gb Increase the Java Heap size for Atlas atlas_max_heapsize to 31Gb Set Atlas to Migration mode by adding the following properties in conf/atlas-application.properties_role_safety_valve atlas.migration.data.filename=<full path to the nav2atlas output file.zip> (If multiple files are generated by the nav2atlas.sh script you can use a regex and import all at once) atlas.migration.mode.batch.size=3000 atlas.migartion.mode.workers=32 atlas.patch.numWorkers=32 atlas.patch.batchSize=300 Restart Atlas service to start import Check the logs from /var/log/atlas/application.log file After the Load is done Once the Migration is complete you can bring Atlas out of migration mode by taking out the properties that were added to load the data in our previous step Once Atlas is out of migration mode you can verify the number of entities migrated and also some samples for the migrated entities. There might be a few entities dropped because of some missing parameters in the source cluster Conclusion Migrating from Cloudera Navigator to Apache Atlas offers improved data governance and cataloging features, crucial for modern data-driven organizations. By following the steps outlined, organizations can smoothly transition their metadata management while maintaining compliance and audit-readiness. Authored by Hruday Kumar Settipalle, Solution Architect at Alephys.