Senior Data Engineer at Illumina, Inc. in India - Bengaluru - Manyata

Job description

• The Databricks Platform Engineer is the technical owner of our Databricks Lakehouse platform — the shared foundation on which enterprise data products, analytics, reporting and AI/ML workloads are built.
• This is a hands-on individual-contributor role responsible for the reliability, scalability, security, automation, governance and cost efficiency of the Databricks platform. The successful candidate will have production experience operating Databricks as an enterprise platform, not just building on it. They will work AI-natively and set that standard for the team.
• You will own the Data Platforms domain within a cross-skilled Data & Analytics Platforms engineering team. You will establish platform engineering standards, design and build infrastructure automation, create reusable reference architectures and engineering patterns, and ensure the platform can be operated reliably at enterprise scale. You will also mentor engineers and help build platform engineering capability within the India team.

Responsibilities

• Platform ownership
• Own workspace- and account-level architecture: Unity Catalog, compute policies, Lakeflow Jobs and shared compute infrastructure, identity and service principals, secrets management, audit logging, system tables, storage integration, platform upgrades and environment promotion across development, test and production.
• Design, build and review infrastructure-as-code, CI/CD and observability capabilities for the platform, establishing reusable standards and automation for the wider engineering team.
• Own platform reliability: availability, capacity, incident management, root-cause analysis, and DR/BCP; define and meet platform SLOs for shared platform services.
• Run Databricks FinOps: consumption monitoring, cost optimisation, budget and anomaly management, and chargeback/showback with transparent unit costs for consuming teams.
• Implement platform security and governance controls: identity federation/SCIM, service principals, network controls, secrets, RBAC/ABAC, governed tags, row/column security and masking, PII handling, lineage and audit.
• Enable and operate the platform capabilities required for AI/ML workloads — MLflow, Model Serving, vector search and GenAI infrastructure patterns — partnering with analytics and AI teams on workload architecture, governance and cost controls.
• Build and maintain reference implementations and paved-road patterns (PySpark, SQL, Delta Lake, Auto Loader) that show engineering teams how to use the platform; enable consuming teams rather than owning delivery of their data products.
• Treat the platform as an internal product: understand consuming-team needs, define paved-road capabilities and self-service patterns, track adoption and reliability, and continuously reduce developer friction.
• Own the platform roadmap: upgrades, new capabilities, deprecations.
• Engineering standards
• Contribute to the team engineering handbook and architecture decisions; lead code reviews on platform and foundation work.
• Take part in architecture and engineering forums, communicating trade-offs clearly to peers and global stakeholders.
• Maintain reusable patterns and automation used across the team.
• Establish AI-native engineering practice for the team: how AI assistants are used for infrastructure code, test generation, runbooks, code review and troubleshooting, and where human review is mandatory.
• Mentoring and collaboration
• Mentor engineers through onboarding, pairing and code review.
• Provide secondary coverage for an adjacent platform area as part of the team's cross-skilling model.

Requirements

• 3–5 years of professional data platform or data engineering experience, including at least 2 years administering and operating Databricks in production as an enterprise platform (account and workspace administration, Unity Catalog, compute policies, environment promotion), not only developing pipelines on it.
• Demonstrated ability to design, build and review infrastructure-as-code (Terraform or equivalent) and CI/CD for data platforms (GitHub Actions, AWS CodePipeline or similar).
• Strong Python and PySpark; advanced SQL.
• Experience running platform operations: monitoring and alerting, incident management, runbooks, capacity and performance management.
• Experience with Databricks cost management and FinOps practices.
• Platform security and compliance experience: identity, service principals, secrets, network controls, audit logging, RBAC and PII handling, with an understanding of controls such as SOX.
• Solid understanding of distributed systems and system design for large-scale data processing.
• Strong AWS fundamentals: S3, IAM, VPC and networking, KMS, and how Databricks integrates with them.
• Demonstrated effective use of AI-assisted engineering tools, or a strong ability and willingness to adopt them, across coding, IaC, testing, documentation and troubleshooting, with sound judgement around verification, security and human review. Specific tools are not a screening criterion.
• Strong written and verbal communication skills; able to work effectively with global stakeholders and lead technical discussion.
• Bachelor's degree in Computer Science, Information Systems, Engineering or a related field, or equivalent demonstrable experience.
• Databricks Certified Data Engineer Professional and/or AWS Certified Solutions Architect.
• Experience operating platforms in compliance-driven environments (SOX, GxP / 21 CFR Part 11 or similar): change control, audit evidence, access reviews, segregation of duties.
• Experience supporting, onboarding or mentoring other engineers.
• Experience taking over or migrating an existing platform estate.
• Experience working in a global delivery model with distributed stakeholders.

Skills

Required

• Databricks administration and operation
• Unity Catalog
• Compute policies
• Python
• PySpark
• SQL
• Monitoring and alerting
• Incident management
• Runbooks
• Capacity and performance management
• Databricks cost management and FinOps
• Distributed systems and system design

Preferred

• AWS Certified Solutions Architect
• Mentoring and onboarding
• Platform migration
• Global delivery model