Lead Data Engineer, Consumer at Versant Media in New York, NY, United States

Job description

• We are seeking a hands-on Data Product Engineering Lead to lead engineering resources to deliver trusted, scalable, and reusable data products from source systems through the Silver layer of our enterprise data platform.
• This role is accountable for turning product and business needs into reliable, governed data capabilities: ingesting source data, applying standardized transformations, implementing data-quality controls, and publishing documented Silver-layer datasets ready for downstream analytics, reporting, AI, and operational use.
• The Data Product Engineering Lead works closely with Solution Architects, Data Modelers, Product Managers, domain experts, platform teams, and governance partners to ensure each data and digital product is built on a scalable data foundation.
• Product teams receive reliable Silver-layer data products on predictable timelines.
• New sources and data products are delivered faster because teams reuse proven patterns and shared components.
• Silver-layer data is trusted: complete, reconciled, monitored, documented, and traceable to authoritative sources.
• Data model and architecture standards are implemented consistently across domains.
• Pipeline failures, data-quality defects, and late-breaking schema changes are detected early and resolved quickly.
• The distributed engineering team operates as one accountable delivery unit rather than as a ticket-fulfillment function.

Responsibilities

• Lead delivery of source-to-Silver data products
• Lead and develop a distributed/offshore team of Data Engineers responsible for delivery from source ingestion through curated Silver-layer data products.
• Translate product roadmaps and requirements into actionable engineering plans, milestones, estimates, dependencies, and delivery commitments.
• Design and oversee ingestion, transformation, standardization, orchestration, and publication of data from operational, SaaS, file, streaming, API, and partner sources.
• Ensure Silver-layer data products are validated, standardized, documented, reusable, performant, secure, and ready for Gold-layer analytics and AI consumption.
• Establish repeatable patterns for batch, incremental, change-data-capture, and event-driven processing.
• Build scalable engineering foundations
• Implement reusable data-engineering frameworks, templates, libraries, and pipeline patterns that reduce repeated effort across products and domains.
• Apply modern engineering practices, including source control, peer review, automated testing, CI/CD, observability, release management, and incident remediation.
• Define and maintain standards for naming, partitioning, schema evolution, error handling, replay/recovery, performance, cost management, and documentation.
• Partner with the Data Platform team to use approved workspace, compute, storage, security, and deployment patterns.
• Identify technical debt and lead pragmatic improvements that increase delivery speed, reliability, and maintainability.
• Partner across architecture, modeling, and product
• Work with Data Modelers to implement canonical entities, conformed dimensions, data contracts, source-to-target mappings, and enterprise modeling standards.
• Work with Product Managers and domain leaders to clarify intended outcomes, source-system realities, priority use cases, and acceptance criteria.
• Coordinate dependencies with source-system owners, platform teams, analytics teams, and external partners.
• Ensure trusted, governed data
• Establish data profiling, reconciliation, quality testing, freshness monitoring, lineage, and alerting for every delivered data product.
• Ensure source-to-Silver traceability, including authoritative source identification, transformation logic, ownership, metadata, and data-quality expectations.
• Implement appropriate access controls, sensitive-data classification, masking, retention, and regional/data-residency requirements.
• Drive resolution of data defects, schema changes, pipeline failures, and quality issues through clear ownership and service-level expectations.
• Lead the team and delivery operating model
• Set clear priorities, technical direction, delivery expectations, and quality standards for the engineering team.
• Coach engineers in data engineering, cloud development, testing, observability, and product-oriented delivery practices.
• Create a healthy onshore/offshore delivery model with defined handoffs, overlap hours, documentation standards, ceremonies, and escalation paths.
• Communicate delivery progress, risks, tradeoffs, and decisions clearly to technical and business stakeholders.
• Build a culture of ownership, continuous improvement, and reliable execution.

Requirements

• Required qualifications
• 8+ years of data engineering, software engineering, or data-platform experience, including 2+ years leading engineers, technical workstreams, or large-scale delivery.
• Demonstrated experience designing and delivering enterprise-scale ingestion and transformation pipelines.
• Strong hands-on expertise in SQL, Python, Spark/PySpark, and modern ELT/ETL patterns.
• Experience with Databricks, Delta Lake, or a comparable lakehouse platform; experience with cloud storage, orchestration, and CI/CD.
• Strong understanding of Bronze/Silver/Gold or equivalent layered data-platform patterns.
• Experience implementing data quality, observability, lineage, metadata, and production support practices.
• Proven ability to partner with architects, data modelers, product leaders, and domain stakeholders.
• Experience leading distributed teams and creating effective delivery practices across time zones.
• Strong communication skills and the ability to explain technical choices, risks, and tradeoffs to non-technical stakeholders.
• Experience with Unity Catalog or comparable governance, catalog, and access-control capabilities.
• Experience with data contracts, schema evolution, change data capture, APIs, event streaming, and large-volume data processing.
• Experience supporting multiple data domains, global data products, or regional data-residency requirements.
• Experience with infrastructure-as-code, DataOps, and automated environment provisioning.
• Experience in a regulated, high-scale, or operationally sensitive environment.

Skills

Required

• SQL
• Python
• Spark/PySpark
• ELT/ETL patterns
• Databricks
• Delta Lake
• Cloud storage
• Orchestration
• CI/CD
• Data quality
• Observability
• Lineage

Preferred

• Unity Catalog
• Data contracts
• Schema evolution
• Change data capture
• APIs
• Event streaming
• Large-volume data processing
• Infrastructure-as-code