Remote | AI Software Engineering Domain Expert — $100–$200/hour at 24-MAG in New York, New York, United States

Job description

• We are sharing a specialised part-time consulting opportunity for experienced software engineering professionals with strong expertise in technical analysis, architecture, documentation, quality assurance, prompt development, and AI-output evaluation to contribute to an advanced AI training and software-engineering evaluation project.
• Selected professionals will review AI-generated technical content, design and refine prompts, evaluate outputs against structured rubrics, and contribute high-quality engineering analysis and documentation. The work is suited to Software Engineers, Staff Engineers, Technical Leads, Engineering Managers, Solutions Architects, and similar professionals with strong technical judgement and communication skills. No prior experience in AI is required.

Responsibilities

• Technical & Architecture Evaluation
• Review AI-generated software-engineering content for accuracy, completeness, and practical relevance
• Evaluate architecture, system-design, and implementation decisions
• Identify incorrect assumptions, weak reasoning, or maintainability, scalability, reliability, and performance concerns
• Review RFCs, design specifications, engineering proposals, and related technical materials
• Provide structured feedback grounded in professional engineering practice
• Prompt Development & AI Evaluation
• Design and refine prompts for complex software-engineering scenarios
• Create tasks testing technical reasoning and engineering judgement
• Evaluate model outputs using detailed rubrics for correctness, relevance, and technical quality
• Identify ambiguous or edge-case outputs requiring deeper analysis
• Provide clear written rationales supporting evaluation decisions
• Technical Writing & Quality Assurance
• Draft and edit technical documentation, reports, and analytical summaries
• Conduct research and fact checking to validate technical claims
• Review and annotate engineering content according to project standards
• Support consistency and calibration across evaluation workflows
• Translate complex engineering concepts for technical and non-technical audiences

Requirements

• 3+ years of professional software-engineering experience
• Background as a Software Engineer, Senior Software Engineer, Staff Engineer, Technical Lead, Engineering Manager, Solutions Architect, or comparable professional
• Strong analytical reasoning and technical problem-solving skills
• Experience authoring or reviewing technical documentation
• Familiarity with architecture documents, RFCs, design specifications, engineering proposals, postmortems, or code reviews
• Strong understanding of software architecture and engineering trade-offs
• Excellent written communication and technical-editing skills
• Strong attention to detail and commitment to quality
• Experience with prompt authoring, content evaluation, data annotation, or rubric-based assessment is advantageous
• Familiarity with AI coding tools, model evaluation, RLHF, or prompt engineering is beneficial but not required
• Advanced academic qualifications are valued but not required
• No prior AI-training experience is required

Skills

Required

• Technical analysis
• Architecture evaluation
• Documentation
• Quality assurance
• Prompt development
• AI-output evaluation
• Analytical reasoning
• Technical problem-solving
• Technical writing
• Technical editing
• Attention to detail

Preferred

• Prompt authoring
• Content evaluation
• Data annotation
• Rubric-based assessment
• Familiarity with AI coding tools
• Model evaluation
• RLHF
• Prompt engineering