Model Test & Measurement Engineer

Overview

Talon Defense seeks a Model Test & Measurement Engineer to assess the performance, reliability, and quality of AI and analytics capabilities. Working alongside a multidisciplinary engineering team, this role will develop independent evaluation methods, automated testing tools, and clear reporting to support the delivery and ongoing improvement of enterprise AI solutions.

Key Responsibilities

• Design independent evaluation approaches for machine learning and AI-enabled applications
• Develop labeled datasets, annotation guidance, and sampling strategies for meaningful performance measurement
• Build automated evaluation tools and reproducible reports across model and application versions
• Define metrics and acceptance criteria with engineering teams and technical stakeholders
• Analyze errors, uncertainty, robustness, and performance differences across relevant datasets and conditions
• Assess fairness and human oversight where applicable and document findings and limitations
• Provide objective evaluation evidence to support release decisions and ongoing improvement
• Maintain appropriate independence from the development and tuning of the models being evaluated

Required Qualifications

• Bachelor's degree in Computer Science, Data Science, Software Engineering, Mathematics, Statistics, or a related discipline, or equivalent experience, certifications, or military technical training
• Experience with model validation or independent model evaluation as a primary responsibility
• Experience constructing ground truth, developing sampling plans, and applying statistical evaluation methods
• Proficiency in Python and tools for data analysis, automated testing, and reproducible reporting
• Experience defining performance metrics and acceptance criteria for operational analytics or ML systems
• Ability to assess false positives, false negatives, population differences, and uncertainty in evaluation results
• Ability to document findings clearly and maintain independence in evaluation activities
• Strong communication skills and ability to collaborate within a multidisciplinary technical team

Preferred Qualifications

• Experience with government or regulated-industry model validation, model risk management, or independent assurance
• Familiarity with the NIST AI Risk Management Framework or comparable AI assurance practices
• Experience evaluating fairness, disparate effects, and human review processes
• Experience evaluating entity resolution, NLP, retrieval-augmented generation, or calibrated scoring systems