Sr. Databricks Data Engineer at Halvik in McLean, VA, USA

Job description

• The Senior Databricks Data Engineer / Technical Lead will support the U.S. Department of Transportation (DOT) data modernization effort by designing, building, migrating, and operating secure, scalable data pipelines in a Databricks Lakehouse. This is a highly hands-on role focused on legacy and external-source migration, metadata-driven ingestion, Medallion Architecture, data quality and reconciliation, production operations, and transition to the AWS/OneDOT environment.

Responsibilities

• Strategic Execution: Lead the design and implementation of reusable ingestion and transformation patterns for legacy and external data sources into Databricks, including support for migration to the AWS/OneDOT environment.
• Operational Oversight: Build and operate Bronze, Silver, and Gold pipelines; manage batch and incremental loads, orchestration, dependencies, retries/recovery, monitoring, alerting, and production support.
• Collaboration: Partner with the Technical Project Manager, Data Architects, business/data SMEs, BI developers, Cloud/DevSecOps, security, and governance teams to translate requirements into implementable data solutions.
• Technical Performance: Develop and optimize Python, PySpark, and SQL workloads using Databricks, Delta Lake, Unity Catalog, Auto Loader and/or Lakeflow capabilities, and Databricks Jobs/Workflows with a focus on reliability, performance, and cost.
• Data Quality & Governance: Implement source-to-target reconciliation, data validation, schema evolution, lineage, access controls, documentation, and reusable engineering standards for production data pipelines.

Requirements

• Education: Bachelor's degree in computer science, Data Engineering, Information Technology, Engineering, or a related field; equivalent relevant experience may be considered.
• Experience: Minimum of 7 years of professional data engineering experience, including significant hands-on experience with Databricks/Spark and production data pipelines.
• Technical Proficiency: Advanced Python, PySpark, and SQL skills; strong experience with Databricks, Delta Lake, AWS/S3, batch/file ingestion (CSV/JSON), REST APIs, Git/CI/CD, and data pipeline orchestration.
• Compliance: Must meet applicable DOT contract security and suitability requirements and demonstrate secure handling of PII and other sensitive data, including access controls and secrets management.
• Certifications: Relevant Databricks and/or AWS data engineering certification preferred; equivalent demonstrated hands-on experience will be considered.
• Demonstrated success modernizing legacy data platforms (Sybase/SAP IQ or similar) into Databricks, including source-to-target reconciliation, cutover, and production stabilization.
• Experience with Unity Catalog, metadata/configuration-driven ingestion, Auto Loader/Lakeflow, Databricks Jobs/Workflows, lineage/cataloging tools, and scalable multi-source ingestion frameworks.
• Federal/DOT delivery experience; familiarity with Agile/Kanban, production O&M, cloud cost optimization, and data governance. Streaming and AI/ML data pipeline exposure is a plus.

Skills

Required

• Advanced Python
• PySpark
• SQL
• Databricks
• Delta Lake
• AWS/S3
• batch/file ingestion (CSV/JSON)
• REST APIs
• Git/CI/CD
• data pipeline orchestration
• access controls and secrets management

Preferred

• Unity Catalog
• metadata/configuration-driven ingestion
• Auto Loader/Lakeflow
• Databricks Jobs/Workflows
• lineage/cataloging tools
• Agile/Kanban
• production O&M
• cloud cost optimization

Benefits

• Company-supported medical, dental, vision, life, STD, and LTD insurance
• Benefits include 11 federal holidays and PTO
• Eligible employees may receive performance-based incentives in recognition of individual and/or team achievements.
• 401(k) with company matching
• Flexible Spending Accounts for commuter, medical, and dependent care expenses
• Tuition Assistance
• Charitable Contribution matching
• Halvik's pay range for this job level is a general guideline only and not a guarantee of compensation or salary. Additional factors considered in extending an offer include (but are not limited to) responsibilities of the job, education, experience, knowledge, skills, and abilities, as well as internal equity, alignment with market data, applicable bargaining agreement (if any), or other law.