Staff Research Engineer, Autonomy VLM at Rivian in US-CA-Palo Alto

Job description

• Vision-Language Models (VLMs) are a foundational pillar of our Autonomy stack. In this Staff
• Research Engineer role, you will play a key role in delivering the overarching VLM strategy,
• especially training, shipping, optimizing the VLM models, as well as extending to
• multi-modalities and enabling new use cases, among others. In this role, you will also be
• responsible to define and deliver VLM-driven solutions to solve some of autonomy's hardest
• challenges, including automated data mining, handling long-tail distributions, rare edge-case
• detection, and scene anomaly reasoning. As part of the model delivery, you will also own the
• whole end-to-end lifecycle of VLM model delivery: data acquisition, metrics definition,
• benchmarking, model performance optimization, deployment, feedback loop.

Responsibilities

• ● Drive and deliver the VLM model strategy: Define, drive and execute the roadmap of
• VLM model delivery, including training and delivering VLM models, optimization,
• deployment, as well as the extension to other multi-modalities.
• ● Accelerate data mining: Design and deliver VLM/LLM related models and strategies
• that power automated data mining, long-tail distributions, rare/edge case detection, and
• anomaly detection at scale, across multiple modalities (vision, lidar, text, etc).
• ● Iterate and optimize performance: Establish rigorous evaluation and monitoring
• benchmarks. Identify and root-cause top-tier system anomalies, prioritizing high-impact
• optimizations to continuously push the needle on performance.
• ● Cross-functional collaboration: Partner closely with core Autonomy teams
• (Perception, Planning, Calibration, Systems, etc) to translate vehicle feature
• requirements into concrete ML deliverables.
• ● Influence trade-offs & requirements: Define system requirements and guide
• cross-functional efforts through technical trade-off decisions.

Requirements

• Education: BS, MS, or PhD in Computer Science, Robotics, Electrical Engineering, or a
• highly related quantitative field.
• ● Experience: 5+ years of professional experience scaling ML solutions, with a strong
• ○ VLM model training: Hands-on experience training or fine-tuning VLMs using
• modern parameter-efficient techniques (LoRA, QLoRA) and RL alignment.
• ○ Large-scale data mining: Proven track record developing VLM/LLM-related
• techniques for data mining, long-tail distributions, rare cases, safety-critical
• events.
• ○ Zero/few-shot capabilities: Experience with open-vocabulary, zero-shot, or
• few-shot classification models, particularly in long-tail scenarios.
• ○ System engineering: Strong proficiency in Python alongside a solid
• understanding of modern Perception pipelines, benchmarking tools, and
• infrastructure.
• ○ Execution: Demonstrated ability to root-cause complex issues across a
• distributed, cross-functional stack in a fast-paced environment.
• ● Experience applying VLMs within the Autonomous Vehicle domain.
• ● Experience with Auto Prompt Optimization (APO) and automated prompt engineering
• techniques.
• ● Experience with spatial grounding in 2D and/or 3D.
• ● Experience extending foundational models to extra modalities (e.g., LiDAR, Radar, IMU,
• ego-motion).
• ● Experience utilizing VLMs or Foundation Models for complex behavior reasoning and
• planning.
• ● Experience with onboard edge deployment, cloud inference architectures, and balancing
• compute/efficiency trade-offs.
• ● Experience with quantization techniques (PTQ, QAT) and high-performance inference
• engines like TensorRT.

Skills

Required

• VLM model training
• Fine-tuning VLMs
• RL alignment
• Data mining
• Long-tail distributions
• Rare/edge case detection
• Anomaly detection
• Open-vocabulary classification
• Zero-shot classification
• Few-shot classification
• Python
• Perception pipelines

Preferred

• Autonomous Vehicle domain
• Auto Prompt Optimization (APO)
• Automated prompt engineering
• Spatial grounding in 2D/3D
• Behavior reasoning and planning
• Onboard edge deployment
• Cloud inference architectures
• Compute/efficiency trade-offs

Benefits

• The listed base salary range for this role is $228,00 - $285,000 for San Francisco Bay Area based applicants. This is the lowest to highest salary we in good faith believe we would pay for this role at the time of this posting. An employee’s position within the salary range will be based on several factors including, but not limited to, specific competencies, relevant education, qualifications, certifications, experience, skills, geographic location, shift, and organizational needs.
• We offer a comprehensive package of benefits for full-time and part-time employees, their spouse or domestic partner, and children up to age 26, including but not limited to paid vacation, paid sick leave, and a competitive portfolio of insurance benefits including life, medical, dental, vision, short-term disability insurance, and long-term disability insurance to eligible employees. You may also have the opportunity to participate in Rivian’s 401(k) Plan and Employee Stock Purchase Program if you meet certain eligibility requirements. Full-time employee coverage is effective on their first day of employment. Part-time employee coverage is effective the first of the month following 90 days of employment. More information about benefits is available at rivianbenefits.com.
• Pay Disclosure