Live opening · Posted 3 days ago
LLM Evaluation Engineer
Nebulai · United States (Remote)
You are 3 days behind. JobBeeper subscribers saw this role while it was still new.
At a glance
The key details from the original listing.
CompanyNebulai
LocationUnited States (Remote)
Work modeYes
Source
Linkedin
Listed3 days ago
Your early-applicant advantage
Live timing from JobBeeper.
15 min
from Linkedin publishing this role to us finding it
11 min
median time from a role going live to a subscriber being told
6 hours
subscribers had this role before this page existed
61,072
roles found in the last 24 hours — the newest are not on this site yet
About the role
Description supplied by the original job listing.
Treat LLM quality as an engineering problem—not a vibe check.
Through Nebulai, you embed with client teams building eval harnesses for RAG, agents, and copilots: golden sets, rubric-based judges, regression suites, online monitoring, and dashboards that tell product when quality slipped. Fewer silent failures. Clearer ship/no-ship calls. Governed delivery that escapes the pilot trap.
You’ll be a strong fit if you:
- Have built offline and/or online evals for LLM apps (faithfulness, task success, safety)
- Are solid in Python and comfortable with experiment tracking and CI hooks
- Can partner with PMs and domain experts to define what “good” actually means
- Know common failure modes: hallucination, retrieval miss, tool misuse, prompt drift
Bonus if you’ve:
- Used frameworks like RAGAS, DeepEval, Braintrust, LangSmith, or custom judges
- Instrumented production tracing for LLM calls
- Worked in regulated settings where auditability matters
Apply at https://nebulai.app
Work arrangement
Yes
More openings worth a look
Recently tracked roles with full details and direct application links.
Good roles move before most people even see them.
Tell JobBeeper what you want and get fresh matches delivered in minutes.
Start your free trial →