Artificial Intelligence Researcher

AI Technology Evaluation Engineer

Greater Philadelphia area - Hybrid

Key Responsibilities

• Research and evaluate emerging AI platforms, LLMs, coding assistants, agents, MCP-enabled solutions, and related enterprise technologies
• Design and execute structured evaluations, benchmarks, test scenarios, and scorecards to assess capabilities, reliability, security, interoperability, scalability, and value
• Experiment, prototype, and build proofs-of-concept to technical capabilities and limitations
• Evaluate agent configuration, integrations, data flows, authentication, permissions, security risks, and governance considerations
• Analyze token usage, context windows, rate limits, licensing, consumption patterns, and other cost drivers
• Compare competing technologies, document findings, and present clear recommendations to technical and business stakeholders
• Develop reusable evaluation methods, standards, guidance, and decision criteria for future IT assessments
• Build scripts, test harnesses, integrations, or sample applications as needed to support evaluations

Required Skills & Experience

• Bachelor's degree in Computer Science, Engineering, Information Systems, Data Science, or related discipline, or equivalent experience
• Working proficiency with Generative AI concepts and technologies, including LLMs, prompting, context, RAG, tool use, agents, orchestration, and evaluation
• Experience evaluating M365 Copilot, GitHub Copilot, Claude Code, AI agents, MCP servers, or comparable technologies
• Familiarity with prompt/agent evaluation, automated/adversarial/security/AI safety testing
• Strong technical foundation in software engineering, enterprise platforms, cloud technologies, integrations, or solution architecture
• Experience with Microsoft 365, Entra ID, Azure AI, AWS AI services, or hybrid enterprise environments
• Experience creating technical scorecards, comparison matrices, architecture assessments, security reviews, or technology standards
• Knowledge of software architecture, CI/CD, source control, observability, or enterprise data integration
• Ability to independently research unfamiliar technologies, conduct hands-on testing, and reach evidence-based conclusions
• Develop Python or another modern language to build prototypes, integrations, and test assets
• Experience with APIs, authentication, cloud services, developer tooling, or technical prototyping
• Understanding of enterprise security, identity/access management, permissions, data protection, and risks
• Strong written and verbal communication skills and a self-directed approach to working in an agile innovation environment