← All Positions
Member of Technical Staff, AI Evaluation
Job Summary
As a Member of Technical Staff focused on AI Evaluation, you will play a critical role in defining and building the evaluation frameworks for autonomous systems interacting with the physical world. You will be responsible for establishing benchmarks that measure engineering reasoning and performance, ensuring the transition from junior to senior capabilities in complex physical environments.
Essential Functions
- Build and own the evaluation framework from scratch to define high-quality engineering design standards.
- Collaborate with engineering subject matter experts, customers, and internal teams to identify success metrics.
- Recreate real-world engineering tasks and failure modes within controlled evaluation environments.
- Design and refine automated judges and human evaluation processes to ensure alignment with expert judgment.
- Translate evaluation results into actionable priorities for improving AI reasoning and capabilities.
Required Qualifications
- 5 to 15 years of experience as a Member of Technical Staff, AI Engineer, or AI Research Scientist.
- Extensive experience in AI evaluations, including building tasks, designing judges, and human-in-the-loop evaluation.
- Strong background in physical engineering (mechanical, electrical, aerospace, robotics, or power systems).
- Proven ability to lead cross-functional teams and work directly with customers and partners.
- Experience in early-stage startup environments performing zero-to-one development.
- Understanding of agentic AI systems and continuous learning models.
Preferred Qualifications
- PhD preferred with experience in startups, research labs, or large technology firms.
- Experience in physical AI or robotics industries.
- Background in academia with a track record of productizing research ideas.
Technical Skills
- Agentic AI Systems
- AI Evaluation Frameworks
- Automated Judges (LLM-as-a-Judge)
- Human-in-the-loop Evaluation
- Evaluation Environments
- SFT and RLVR
- Engineering Design Tools
Education & Certifications
Technical background in a physical engineering domain such as mechanical engineering, aerospace, robotics, or power systems. PhD preferred.
Compensation & Benefits
Salary range: $200,000 - $300,000 per year.