What we test for
Typical work
- Golden sets built from your real traffic rather than from invented examples
- Eval suites that run in CI and block a release that got worse
- Red teaming: injection, jailbreaks, and the answers you could not ship
- Production monitoring, so the drift is found before a customer finds it
The coding test
We hand over an AI feature and tell you it works. Build the eval that proves or disproves it: the case set, the scoring, and the threshold you would gate a release on. Then show us a change that lifts the average while making one group of users worse off. Scored on whether the eval would survive someone trying to game it.
Five interview questions you can reuse
We use questions like these in our soft-skills and technical rounds. Steal them for your own interviews.
How do you build a golden set that is worth anything?
A strong answer covers: From real traffic, stratified so the hard and rare cases are not drowned out, labelled by someone who knows the domain, and then frozen. A set written by the same person who wrote the prompt only tests the prompt against itself.
When can you trust an LLM as a judge, and when can you not?
A strong answer covers: For graded qualities at scale, once you have measured the judge against human labels on a sample and know its agreement rate. Not for anything the judge model is itself bad at, and never letting a model family grade its own output unchecked.
An average score went up. Why is that not enough?
A strong answer covers: Averages hide the groups that got worse. Read the per-case diff, segment by the dimension the business cares about, and watch the tail. A change that lifts the mean and breaks your largest customer is a regression.
What does red teaming an AI feature actually involve?
A strong answer covers: Trying to make it do the thing you promised it would not: injection through any field that reaches the model, extraction of the system prompt or of data belonging to other users, and outputs you could not defend. Written down as cases and rerun, not done once before launch.
How do you catch a model getting worse in production without labels?
A strong answer covers: Proxy signals: refusal and fallback rates, retrieval scores, latency and length distributions, user corrections and thumbs-down, plus a sampled slice sent for human review on a schedule. Alert on the distribution changing shape, not on a single bad answer.
One monthly price per developer. Everything in it.
No deposit, no minimum term, no placement fee. You pay one invoice a month for each developer on your team.
Per developer, all-in. Your quote arrives by email with your exact number after you send the brief.
- Developer pay, local payroll and taxes
- Vetting: coding test, soft skills, English
- Free replacement within 90 days
- A direct line to our CEO
What changes the price
- Seniority and role. Senior engineer, tech lead or architect.
- Stack. Some stacks and niche tools cost more to staff.
- Team size. Larger teams get a better monthly rate per seat.
- White-label. Agencies reselling under their brand get partner pricing.
Terms in plain words
- First 2 weeks risk-free: if it isn't working, you don't pay for them
- Month to month, 30 days' notice to end
- No deposit and no minimum number of seats
- Want to hire the developer full-time? Buyout of 3 months' fee in year one, 2 months' after
Questions tech leads ask us
How senior are your AI QA engineers?
Six or more years in testing or engineering, and at least one eval suite that actually gated a release. We ask what it caught, and what it missed.
Why would a staffing company be good at this?
TopDevCrew is part of qa-bit, a QA company. Measuring software nobody is sure how to measure is what the group has been doing for years, and AI is a new subject for the same discipline.
Can I see the coding test results?
Yes. Every profile card carries the score and the reviewer notes.
What does it cost?
$6,900 per developer per month, all-in, and flat: the same rate whatever the stack. Your exact quote arrives by email after you send the brief.
Skip the sales rep. Talk to the CEO.
Tell us what you're building, your stack and when you need people. Three matched profiles within 48 hours, and a straight answer on whether we're the right fit.