Agent Evaluation Design Jobs

1 job found

Today

Product Designer

Meta San Francisco, CA

San Francisco, CA

Today

$201k - $278k/yr Full time Agent Evaluation Ai Design Design

Apply Now

Job Description

Meta is building a multimodal AI agent experience through Meta Business Agent, and this Product Design role helps shape it end to end. You will work across prompting, evaluation, and interaction systems spanning chat, voice, and visual surfaces for businesses ranging from small business to enterprise.

What you’ll do

Write and revise the prompts that power Meta’s agentic tools, features, and behaviors on a product surface.
Test the surface, convert findings into prompt fixes, ship updates, and verify that the prompt behavior delivered to users matches the intended design.
Package prompt limitations for training purposes, including evaluation artifacts such as graders for crisp behaviors and human-feedback questions plus good/bad pairs for softer signals like writing quality.
Partner closely with engineering and data science to define what strong agent output looks like across chat, voice, and visual surfaces, and to drive the evaluation strategy that measures it.
Expand evaluation coverage beyond text into voice and visual outputs, including how quality is judged when the surface is conversational or visual.
Define quality bars that work across a long tail of businesses so that a single agent specification can support small business through enterprise without bespoke rework.
Rapidly prototype new experiences and concepts, helping the team prioritize new workstreams by considering both impact and feasibility.
Define, evolve, and unify visual and interaction systems that scale across products and platforms.
Define and operationalize evaluation strategy, including building evals, golden sets, judges, and related tooling alone and with data science and engineering.
Apply knowledge of how code works in software: update prompts and language through internal tools, review changes to confirm the shipped experience, and build logic that adapts agent behavior to context.
Work with product and cross-functional partners to define strategy for multimodal agent communication.
Communicate design strategy and rationale to leadership and cross-functional executives, supporting organizational alignment across product, engineering, research, and content strategy.

What you bring

11+ years designing software products with an emphasis on UX, interaction, and visual design aligned to business objectives.
Experience building and maintaining evaluation pipelines for LLM products, including graders, rubrics, comparison sets, regression suites, and the infrastructure to run them across models.
Experience designing and building agentic experiences.
B2C or B2B experience.
Deep expertise with prototyping tools, plus the ability to evaluate and advance design craft across different fidelity levels.

Benefits

$201,000/year to $278,000/year plus bonus, equity, and benefits.

Preferred qualifications

Experience applying responsible, ethical AI practices such as risk assessment, bias mitigation, and quality and accuracy reviews.
Track record integrating AI tools to optimize or redesign workflows with measurable impact (for example, efficiency or quality improvements).
Experience turning product rubrics into training signals such as graders, human-feedback questions, or preference pairs.
Experience shaping agent behavior on voice or other multimodal surfaces.
Ongoing AI skill development, including prompt/context engineering and agent orchestration, with staying current on emerging AI technologies.
AI-native design fluency, including model design and generative AI logic, and defining assistant behavior and tone at the systems layer rather than only surface-level copy.
Front-end or notebook-to-app experience, plus opinions about what makes an eval result easy to understand at a glance.
Experience with both B2C and B2B, preferred.

Location: San Francisco, CA (onsite)

Salary: USD 201,000 - 278,000 per yearly

View full job page →

Get Agent Evaluation Design Jobs alerts

New Agent Evaluation Design Jobs, daily.