Jacob Weiss
AI researcher
I work on post-training for AI agents. An agent in production fails in the same few ways every day, and a prompt edit is a guess. While catches the failure in traffic, trains the model on it, and proves the gain on a held-out set before anything ships. I co-founded it and I write most of the code.
As Head of AI at Silvia, I built its evals and judges, post-trained its specialist models, and wrote the character spec that shapes how it talks about money. People ask it hard questions about their taxes, mortgages and portfolios, and a wrong answer costs them. That is the pressure the method came out of.
I came to this from mathematics. I have three master’s degrees, from Johns Hopkins and Georgia Tech, and I spent the early agent years contributing to open-source frameworks. The part that was missing was the measurement: whether an agent actually got better, and by how much. That is the problem I work on now.
Everything below was learned by running the experiment. Each result has a recipe you can rerun, a held-out set, and an interval.
Research
- Sep 20, 2026Distill without a reward
On-policy distillation and self-distillation, run as a three-arm experiment with paired held-out intervals. Distillation matched GRPO. Self-distillation was flat on a small model, and the per-token teacher KL says why.
- Sep 19, 2026Which judge can you trust
Five judges grade 300 real tool-agent transcripts against a deterministic conduct rule. Agreement with intervals, Cohen's kappa, leak rate, speed, and the one failure class every judge misses.
- Sep 19, 2026Build a reasoning model
A base model, GRPO with a verifiable reward on MATH, one H100, 46 minutes. A strict format reward costs nine points of pass@1 against a correctness-only reward, replicating the SimpleRL-Zoo finding.
- Sep 19, 2026Agent evals on frontier models
Situations simulated from the agent's own tools, pass@1 with a bootstrap interval, pass^4 as the saturation signal, and a paired delta that fails CI when a guarded behavior regresses.
- Sep 18, 2026A calibrated probability as the judge
Why a calibrated probability beats a parsed sentence as an LLM judge for training data: confidence per row, an unsure band, failure classes, and the agreement check against human labels.
- Sep 17, 2026RL for text to SQL
Seed a database, author tasks, verify by running the SQL, train with GRPO. An 8B model went from 26% to 35% in one 44-minute round, replicated three times. A later round took a 4B model from 53% to 74% on 459 held-out tasks.
- Sep 17, 2026Evals that get harder
Agent evals saturate as the model improves. How to steer the situation writer toward what the agent fails, re-probe after training, and the four ways an eval silently lies.
- Sep 9, 2026Simulated data for support agents
On the held-out telecom split of tau2-bench, an 8B model went from 5% to 30% after fine-tuning on 1,057 simulated conversations, ahead of the same count of expert-written rows. Covering design, grading, p-value, public dataset.
Code
- HF datasetSilvia Tax Bench
Two hundred expert-level U.S. tax questions with rubrics, open under CC BY 4.0. Seven AI products answered the ten hardest, graded blind for accuracy and grounding. Silvia scored highest.
- uv add whileaiwhileai-sdk
Scientific RL and SFT post-training for AI agents. Build evals that can fail, simulate situations, judge with a check against people, train with GRPO, SFT or DPO, and prove every gain on a held-out set.
- while-ai datasets
Eleven public datasets on Hugging Face: simulated agent conversations, judged tool-agent transcripts, text-to-SQL tasks with verifiers, and character-training rows from a model spec.
- daisy
A coding agent in the VS Code sidebar. Open-source model, workspace file tools, shell access.
- ai-scout
Agentic object detection: fast, and accurate on the things you name.
- aws-ai-templates
CDK templates for deploying agents on AWS.
- contributorpixeltable
50 merged pull requests, 2024 to 2025. Model integrations, tool calling, docs.
- contributoragno
47 merged pull requests, 2024. Model providers, tools, storage backends.
Education
- M.S. Financial EngineeringJohns Hopkins University
- M.S. Applied and Computational MathematicsJohns Hopkins University
- M.S. AnalyticsGeorgia Tech