Senior Machine Learning Engineer at Pantheon Data in Reston, VA, USA

Job description

• Pantheon Data is seeking a Senior Machine Learning Engineer to design, build, and operate production AI systems for federal clients - including hybrid retrieval-augmented generation (RAG) applications, Intelligent Document Processing (IDP) pipelines, and LLM-backed decision-support tools running in AWS GovCloud.
• This is a hands-on, production-focused engineering role. You will work with GovCloud-hosted foundation models on Amazon Bedrock, build hybrid retrieval over relational and vector data stores, design rigorous evaluation harnesses, and ship secure, traceable, evidence-grounded systems in FedRAMP High / DoD IL4–IL5 environments. This is not a notebook-only, prompt-only, or research-only role: successful candidates can walk through real systems they have built - the data flow, retrieval and inference architecture, deployment approach, evaluation strategy, failure modes, and what they personally implemented.

Responsibilities

• Design and build hybrid RAG applications combining semantic (vector) search, lexical/full-text search, and structured SQL retrieval, with rank fusion, evidence grounding, and source-cited answer generation.
• Integrate and operate GovCloud-hosted LLMs and embedding models via Amazon Bedrock (e.g., Anthropic Claude, Amazon Titan embeddings), including prompt/version management, guardrails, streaming inference, and cost/latency tuning.
• Build Intelligent Document Processing capabilities: OCR post-processing, layout-aware parsing, table and form extraction, NLP/LLM structured extraction, chunking and embedding strategies, and human-in-the-loop review workflows.
• Develop serverless, event-driven ingestion pipelines (AWS Step Functions, Lambda, S3, Aurora PostgreSQL with pgvector) that turn messy source documents into reliable, traceable, queryable information.
• Design and run evaluation programs for retrieval quality, extraction accuracy, answer groundedness, hallucination risk, and end-to-end system performance - including golden datasets, automated eval harnesses in CI, and regression testing across model and prompt versions.
• Engineer for security and compliance from the start: least-privilege IAM, private VPC endpoints and restricted-egress network boundaries, guardrails and prompt-injection defenses, output validation, audit logging, and CUI-aware data handling in support of ATO requirements.
• Make practical engineering decisions about when to use deterministic logic, classical NLP, OCR, embeddings, LLMs, fine-tuned or distilled models, or human review - and defend those tradeoffs to technical and non-technical stakeholders.
• Design, implement, and maintain ML/AI software components for RAG, IDP, and generative AI systems serving federal customers.
• Build and operate data pipelines for unstructured and semi-structured data: ingestion, extraction, cleaning, enrichment, validation, quality scoring, quarantine/review, and storage.
• Develop and evaluate NLP, OCR, computer vision, embedding, retrieval, and LLM-based approaches for document understanding and question answering.
• Contribute production-quality Python with clear structure, tests, logging, error handling, and documentation; participate in code review and team-based delivery.
• Deploy and support AI services in cloud and containerized environments, including Bedrock model integration, batch processing, workflow orchestration, and observability (CloudWatch, tracing, structured logs).
• Define evaluation methodology and metrics; build eval datasets and harnesses; analyze errors; and drive iterative improvement of retrieval and generation quality.
• Implement responsible-AI and security controls: Bedrock Guardrails or equivalent, PII/sensitive-data redaction, content filtering, prompt-injection mitigation, and output traceability to source evidence.
• Troubleshoot system behavior across model output, data quality, retrieval, schema design, infrastructure, latency, cost, and user workflow.
• Mentor other engineers, help establish standards for evaluation, reproducibility, and responsible AI use, and raise the technical quality of the team.
• Communicate clearly with technical and non-technical stakeholders, including project managers, customers, and executive leadership.

Requirements

• Bachelor's degree in Computer Science, Engineering, Mathematics, Data Science, or a related technical field, or equivalent professional experience.
• 8+ years of professional software engineering experience, including 5+ years hands-on in machine learning / AI engineering or a closely related role.
• Demonstrated experience shipping AI/ML systems beyond notebooks and demos: production pipelines, services, APIs, inference endpoints, evaluation harnesses, or customer-facing tools you built and supported.
• Hands-on experience building LLM-backed applications with managed foundation-model services - Amazon Bedrock strongly preferred (Claude, Titan, or similar models), ideally in AWS GovCloud or another regulated/isolated environment.
• Deep experience with RAG architectures: embeddings, vector search (pgvector or comparable), hybrid retrieval (semantic + lexical + structured), chunking strategies, re-ranking or rank fusion, and citation/evidence grounding.
• Experience designing and running LLM/RAG evaluations: retrieval and answer-quality metrics, groundedness and hallucination measurement, golden datasets, automated regression evals, and human-in-the-loop validation.
• Strong Python engineering skills: readable, maintainable, tested code; debugging; packaging; and integration with other systems.
• Experience with AWS services relevant to this stack: Lambda, Step Functions, S3, IAM, CloudWatch, and relational databases (PostgreSQL preferred).
• Experience working with unstructured or semi-structured data: PDFs, scanned documents, forms, tables, technical manuals, drawings, or engineering documentation.
• Working knowledge of AI security practices: guardrails, prompt-injection defenses, output validation, data redaction, and least-privilege access patterns.
• Ability to design and reason about end-to-end data flow: source data, preprocessing, retrieval, model/inference step, persistence, API/service boundary, evaluation, and user-facing output.
• Experience with Git-based workflows, code review, CI/CD, and team-based software delivery.
• Clear written and verbal communication, including explaining tradeoffs, limitations, and failure modes.
• Ability to work effectively remotely in cross-functional teams.
• Ability to meet deadlines and produce quality work.
• Proficient in Microsoft Suite software including Outlook, Word, Excel, SharePoint, and PowerPoint.
• Applicants selected will be subject to a security investigation and may need to meet eligibility requirements. Secret Clearance is required for continued employment.
• Work Location: Reston, VA - Remote
• Our company prioritizes the benefits of flexibility and collaboration, whether that happens in person or remotely.
• If the position is remote or hybrid, you may periodically work from a Pantheon Data office location or client site.
• If this position is assigned to a Pantheon Data office location or client site, you'll work with colleagues and clients in person, as needed for specific client requirements.
• Interview Requirement: Candidates who are local to the area should be prepared to participate in an in-person interview as part of the selection process. Candidates outside the local area may be considered for a virtual interview.
• Direct experience operating LLM workloads in AWS GovCloud under FedRAMP High or DoD IL4/IL5, including private VPC endpoints, zero- or restricted-egress boundaries, and ATO-supporting documentation.
• Experience with Bedrock features beyond basic inference: Guardrails, Knowledge Bases, Agents, Model Evaluation, provisioned throughput, and cross-model routing.
• Intelligent Document Processing depth: OCR pipelines (e.g., Textract), layout-aware models, table/form extraction, and document quality scoring.
• Familiarity with federal compliance frameworks and their engineering implications: NIST 800-53, CMMC, CUI handling, SBOM/supply-chain controls, and STIG-hardened environments.
• Experience with infrastructure as code (Terraform), containerized deployment (Docker/ECS), and CI/CD pipelines (GitLab CI or GitHub Actions) in regulated environments.
• Experience with observability and LLMOps tooling: structured logging, tracing, model/prompt version management, drift detection, and cost/latency dashboards.
• Experience building internal validation tools, review interfaces, dashboards, or lightweight full-stack applications (e.g., Next.js/React front ends over Python services).
• Familiarity with common ML frameworks and tooling: PyTorch, Hugging Face, scikit-learn, MLflow, or similar.
• Advanced degree in a technical discipline.
• Demonstrated mentorship of junior engineers or contribution to team technical direction.

Skills

Required

• Python
• Amazon Bedrock
• RAG architectures
• Vector search (pgvector)
• Hybrid retrieval
• LLM/RAG evaluations
• AI security practices
• Git-based workflows
• CI/CD
• Microsoft Suite

Preferred

• AWS GovCloud
• FedRAMP High / DoD IL4/IL5
• OCR pipelines (Textract)
• Terraform
• Docker/ECS
• GitLab CI or GitHub Actions
• Observability and LLMOps tooling
• Next.js/React

Benefits

• We are always looking for good people! Pantheon Data is committed to providing its employees with competitive salaries and benefits in order to increase employee satisfaction and productivity. In addition to our benefits, we also offer SmartBenefits through the Washington Metro Area Transportation Authority, where you specify an amount of your pre-tax wages be paid directly to your SmarTrip account. In some cases, tuition assistance may be available for continuing education expenses and certifications related to their position. Additional details may be found at https://pantheon-data.com/careers/
• The salary range for this position is $140,000 - $200,000. This is not, however, a guarantee of compensation or salary. Rather, salary will be set based on experience, geographic location and possibly contractual requirements and could fall outside of this range.