Remote | Data Engineer — $140,000–$180,000/year at 24-MAG in New York, New York, United States
- Company: 24-MAG
- Location: New York, New York, United States
- Posted: Sep 20, 2026
- Type: Full-time
- Salary: $140,000–$180,000/year
- Experience: 6+ years
Overview
We are sharing a full-time opportunity for an experienced Data Engineer with strong expertise in Python, SQL, Apache Spark, AWS, distributed data processing, and scalable data architecture to build and operate infrastructure supporting AI-driven products and research initiatives.
Job description
- We are sharing a full-time opportunity for an experienced Data Engineer with strong expertise in Python, SQL, Apache Spark, AWS, distributed data processing, and scalable data architecture to build and operate infrastructure supporting AI-driven products and research initiatives.
- The role will focus on designing and scaling distributed data pipelines, managing large datasets across cloud environments, and building reliable systems for analytics, experimentation, and model development.
Responsibilities
- Data Pipelines & Distributed Processing
- Design, build, and maintain large-scale pipelines for structured and unstructured data
- Develop distributed processing workflows using Apache Spark or comparable frameworks
- Optimise transformations, partitioning strategies, and computational workloads
- Identify and resolve performance bottlenecks across high-volume data systems
- Support downstream analytics, experimentation, and model-development requirements
- Cloud Architecture & Data Engineering
- Design scalable AWS-based data architectures across SQL and NoSQL systems
- Build reliable ingestion, transformation, storage, and distribution workflows
- Write efficient Python and SQL for production data processing
- Evaluate storage and database technologies against workload requirements
- Improve scalability, maintainability, accessibility, and operational efficiency
- Data Quality, Reliability & AI Support
- Implement monitoring, validation, and automation across data workflows
- Identify failures, anomalies, and data-quality issues
- Maintain integrity and reliability throughout pipelines and storage layers
- Collaborate with AI researchers, data scientists, and engineering teams
- Support data infrastructure for AI/ML training, evaluation, and experimentation
Requirements
- Strong professional experience in data engineering or distributed data systems
- Advanced proficiency in Python and SQL
- Hands-on experience with Apache Spark or comparable distributed-processing frameworks
- Strong experience with AWS data services and cloud-native architecture
- Experience with SQL and NoSQL databases
- Demonstrated experience processing large-scale datasets
- Strong understanding of partitioning, performance optimisation, and scalable architecture
- Familiarity with orchestration, automation, monitoring, and data-quality workflows
- Exposure to AI/ML or research environments is advantageous
- Familiarity with LLM training, evaluation, or experimentation datasets is beneficial
- Experience with data-visualisation tools such as Matplotlib, Seaborn, or Plotly is a plus
Skills
Required
- Python
- SQL
- Apache Spark
- AWS
- Distributed data processing
- Scalable data architecture
- SQL and NoSQL databases
- Partitioning
- Performance optimisation
- Orchestration
- Automation
- Monitoring
Preferred
- AI/ML
About 24-MAG
This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams. By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy