About the Role:
We are seeking a highly skilled Databricks Data Engineer to design, build, and optimize our next-generation data architecture. In this role, you will be instrumental in modernizing our data infrastructure, transitioning legacy ETL workloads to highly scalable, cloud-native solutions, and ensuring our data assets are secure, discoverable, and performant. You will leverage the latest capabilities of the Databricks platform—from declarative pipelines to advanced governance—to deliver high-quality data products to the business.
Key Responsibilities
Pipeline Engineering: Design, build, and maintain robust, scalable ETL/ELT pipelines using PySpark and Spark SQL to process large volumes of structured and semi-structured data.
Declarative Frameworks: Implement and manage Spark Declarative Pipelines (Delta Live Tables) to simplify pipeline development, automate data quality checks, and streamline operations.
Legacy Modernization: Lead the migration of complex workloads from legacy enterprise ETL systems into modernized, scalable PySpark architectures.
Data Governance & Security: Architect and enforce centralized data governance, access controls, auditing, and data lineage tracking across the organization utilizing Unity Catalog.
CI/CD & Automation: Automate the deployment lifecycle of data pipelines, notebooks, and infrastructure using Databricks Asset Bundles (DABs) integrated with enterprise CI/CD workflows.
Performance Optimization: Profile, tune, and optimize complex PySpark jobs and Spark SQL queries for maximum performance and cost-efficiency.
Required Skills & Qualifications
Experience: 5+ years of experience in Data Engineering, with a heavy focus on the Databricks Data Intelligence Platform.
Core Languages: Expert-level proficiency in Python (PySpark) and advanced Spark SQL.
Databricks Ecosystem: Deep hands-on experience with modern Databricks features, specifically Spark Declarative Pipelines and Unity Catalog for centralized governance.
DevOps / DataOps: Proven ability to implement CI/CD pipelines for data assets using Declarative Asset Bundles (DABs), Git, and automation tools (e.g., GitHub Actions, Jenkins, or GitLab CI).
Architecture & Migration: Strong understanding of distributed computing principles and experience migrating legacy on-premise ETL logic (e.g., DataStage, Informatica) to cloud-native Spark environments.
Problem Solving: Strong analytical skills with the ability to troubleshoot complex data processing issues and optimize massive data transformations.
Nice-to-Haves
Experience with event-driven data ingestion architectures and storage-layer triggers (e.g., S3).
Familiarity with cloud infrastructure (AWS, Azure, or GCP) and infrastructure-as-code (Terraform).
Experience building and designing internal frameworks to accelerate data pipeline development.