Data Engineer at Scalence L.L.C. in Irvine, CA

Job description

• Request ID: 107735-1
• Title: Data Engineer
• Locations: - Irvine, CA (ONSITE)
• Duration: 06 Months
• Pay Range: $50 - $54/Hour on W2/C2C (All inclusive)
• We are looking for a highly skilled Data Engineer with 10+ years of experience in designing, building, and supporting large-scale data platforms and pipelines. The ideal candidate will have deep expertise in Python, PySpark, Apache Airflow, SQL, and AWS, along with hands-on experience in troubleshooting production data pipeline issues and optimizing distributed data processing workloads.
• The candidate will be responsible for developing scalable data solutions, orchestrating workflows, managing cloud-based datasets, and ensuring reliable delivery of business-critical data across enterprise systems.

Responsibilities

• Design, develop, and maintain robust data pipelines using Python and PySpark.
• Build containerized data processing applications that ingest, transform, and publish data across enterprise platforms.
• Develop and support Apache Airflow DAGs, custom operators, scheduling frameworks, and workflow orchestration processes.
• Implement complex business rules, data transformations, reconciliation processes, and effective-dating logic.
• Work with AWS S3, Delta Lake, Parquet datasets, and PostgreSQL databases for scalable data storage and processing.
• Optimize Spark applications by identifying and resolving memory, performance, and scalability bottlenecks.
• Manage schema changes, database migrations, and data quality validation processes.
• Develop and maintain Docker containers and Kubernetes-based deployments.
• Support CI/CD and multi-environment deployment pipelines.
• Troubleshoot production incidents, perform root cause analysis, and implement preventive solutions.
• Integrate with external platforms and APIs for data import/export and enterprise data exchange.
• Create runbooks, technical documentation, monitoring processes, and data lineage artifacts.
• Collaborate closely with business stakeholders, architects, and cross-functional engineering teams.

Requirements

• 10+ years of experience in Data Engineering and software development.
• Strong hands-on expertise in Python development, including modular, configuration-driven, and unit-tested applications.
• Extensive experience with Apache Spark / PySpark for building and tuning distributed data processing workloads.
• Proven experience developing and maintaining Apache Airflow workflows and DAGs.
• Strong SQL skills with expertise in relational database design and data modeling.
• Experience with PostgreSQL, schema migrations, and data reconciliation processes.
• Hands-on experience with AWS S3 and cloud-based data platforms.
• Experience troubleshooting production data pipeline failures and performing root cause analysis.
• Strong understanding of ETL/ELT frameworks and large-scale data processing architectures.
• Experience with Git, CI/CD processes, and software development best practices.
• Experience with Docker and Kubernetes.
• Knowledge of Delta Lake and Parquet-based data architectures.
• Experience integrating enterprise applications through APIs.
• Familiarity with data governance, lineage, and monitoring frameworks.
• Exposure to Anaplan integrations and bulk data import/export processes.
• Experience working within Agile development environments.

Skills

Required

• Python
• PySpark
• Apache Airflow
• SQL
• AWS
• PostgreSQL
• ETL/ELT frameworks
• Git
• CI/CD

Preferred

• Docker
• Kubernetes
• Delta Lake
• Parquet
• API integration
• Data governance
• Data lineage
• Monitoring frameworks