Verified Job
IT / Software / Data Analyst
Product Quality Engineer (AI & Agentic Systems)- job post Collective OS Remote
IT / Software / Data Analyst
#781570
Remote / WFH
Collective OS
Job Description
Full job description
Product Quality Engineer (AI & Agentic Systems)
About This Role
Collective OS needs a product-minded engineer who can turn business intent into executable evidence. You will define what good behavior means, create realistic personas and scenarios, build automation, and evaluate agent outputs where exact-text assertions are not sufficient. This is not a downstream manual-testing function. It is a hands-on engineering role that reports into the CTO organization and works as an embedded partner to Product, Design, and Engineering from discovery through release and production learning.
What You Will Own
- Translate product decisions, business rules, and agent specifications into clear behavioral contracts and acceptance criteria.
- Create representative firm and operator personas, together with their contacts, relationship networks, target organizations, integrations, consent states, and data conditions.
- Build and maintain critical-path automation across browser, API, worker, integration, and data layers, including Playwright-based journeys.
- Design AI evaluation suites that combine deterministic checks, structured rubrics, repeated trials, calibrated graders, and periodic human review.
- Test signal detection, relationship-sensitive routing, scoring, recommendations, grounding, memory, permissions, and graceful failure across realistic and adversarial conditions.
- Define release evidence and production quality signals; turn operator corrections and escaped failures into durable regression cases.
- Improve the shared simulation, scenario, trace, and evaluation tooling used by Product and Engineering.
What Success Looks Like
Quality becomes an observable product discipline rather than a final approval gate.
- High-value journeys and high-risk behavioral boundaries are covered by reusable personas, scenarios, and gold datasets.
- Model, prompt, routing, scoring, and data-source changes are evaluated through measurable evidence rather than isolated demos or intuition.
- Failures are diagnosable across ingestion, detection, routing, agent decision, persistence, and presentation.
Product Quality Engineer (AI & Agentic Systems)
About This Role
Collective OS needs a product-minded engineer who can turn business intent into executable evidence. You will define what good behavior means, create realistic personas and scenarios, build automation, and evaluate agent outputs where exact-text assertions are not sufficient. This is not a downstream manual-testing function. It is a hands-on engineering role that reports into the CTO organization and works as an embedded partner to Product, Design, and Engineering from discovery through release and production learning.
What You Will Own
- Translate product decisions, business rules, and agent specifications into clear behavioral contracts and acceptance criteria.
- Create representative firm and operator personas, together with their contacts, relationship networks, target organizations, integrations, consent states, and data conditions.
- Build and maintain critical-path automation across browser, API, worker, integration, and data layers, including Playwright-based journeys.
- Design AI evaluation suites that combine deterministic checks, structured rubrics, repeated trials, calibrated graders, and periodic human review.
- Test signal detection, relationship-sensitive routing, scoring, recommendations, grounding, memory, permissions, and graceful failure across realistic and adversarial conditions.
- Define release evidence and production quality signals; turn operator corrections and escaped failures into durable regression cases.
- Improve the shared simulation, scenario, trace, and evaluation tooling used by Product and Engineering.
What Success Looks Like
Quality becomes an observable product discipline rather than a final approval gate.
- High-value journeys and high-risk behavioral boundaries are covered by reusable personas, scenarios, and gold datasets.
- Model, prompt, routing, scoring, and data-source changes are evaluated through measurable evidence rather than isolated demos or intuition.
- Failures are diagnosable across ingestion, detection, routing, agent decision, persistence, and presentation.
