IT Support jobs in Halifax, NS

Job description:

Data Skill Source is hiring an AI Agent Quality Assurance (QA) Team Lead to stand up and run a quality assurance workstream for a fast-growing B2B SaaS client in the intake-to-procurement space. This is a North America–based role leading a small, distributed QA team that tests and evaluates the client's customer-facing AI agents and multi-step workflows.

You'll own the QA workstream end to end: defining how the team tests, building the evaluation standards, managing day-to-day execution, and reporting quality signals back to the client's AI Operations team. This is a player-coach role—you'll set the standard by doing the work, then scale it through your team.

If you have strong QA or AI evaluation instincts and want to build a function rather than just execute tickets, this is for you.

What You'll Do

Workstream Ownership

• Stand up and own the Agent QA workstream: testing approach, evaluation frameworks, coverage priorities, and quality standards.
• Define what "good" looks like for the client's AI agents and translate loosely defined quality goals into reproducible test suites and metrics.
• Report quality signals, risks, and trends clearly to the client's AI Operations team.

Team Leadership

• Lead a distributed team of QA Engineers (Manila-based), coordinating across time zones with required ET overlap.
• Assign and prioritize work, review output, unblock the team, and maintain consistent quality across contributors.
• Onboard, coach, and level up team members; surface performance issues early.

Hands-On QA & Evaluation

• Design and run evaluation pipelines, adversarial/security testing, and workflow audits alongside the team.
• Validate agent accuracy, schema/API compliance, grounding, and edge-case handling across multi-step workflows.
• Drive root-cause analysis on failures and translate them into actionable feedback for the client's engineers.

What We're Looking ForCore Traits (Non-Negotiable)

• Ownership: You take a loosely defined mandate and turn it into a functioning, well-run workstream.
• Player-Coach Balance: You can do the technical work yourself and lead others doing it.
• Strong Communication: Direct, clear written and spoken English; you translate technical detail into signal for non-QA stakeholders.
• Judgment Under Ambiguity: You make reasonable calls when standards aren't fully defined, and know when to escalate.

Technical Skills

• Professional QA, software engineering, or AI/system evaluation experience, with hands-on backend testing, API validation, or eval-pipeline work.
• Experience evaluating LLM-based systems or AI agents (prompt behavior, grounding, adversarial testing) strongly preferred.
• Comfort with automated test suites, structured data validation, and schema/API testing.
• Prior experience leading, mentoring, or coordinating a team—formal or informal.

Nice to Have

• Experience working with distributed/offshore teams.
• Background in B2B SaaS, enterprise software, or procurement/fintech domains.

Note: Formal titles matter less than demonstrated ability. We care far more about what you've built and led than where you've worked.

What We Offer

• Consistent, long-term contract work with a stable, high-growth B2B SaaS client.
• 30+ hours/week, fully remote (Canada or US).
• MacBook Pro provided.
• The opportunity to build a QA function from the ground up and grow with a scaling managed-service program.

About Data Skill Source

Data Skill Source partners with fast-growing SaaS companies to deliver managed services, systems support, and applied automation. We focus on practical, production-ready technology—not demos or experiments that never ship.

How to Apply

Please submit your resume and a short note describing:

• QA or AI evaluation workstreams you've owned or built from scratch.
• Your experience leading or coordinating a team.
• Any hands-on work evaluating AI agents, LLM systems, or complex backend workflows.
• Links to GitHub repos, portfolios, or write-ups (if available).

Job Types: Full-time, Contract

Work Location: Remote (Canada or US)

Pay: From $30.00 per hour

Benefits:

• Flexible schedule

Work Location: Remote