About Yuhan Chi

Question. Can shallow token-level signals such as entropy, log-probability, and confidence trajectories separate correct from incorrect math reasoning traces without extra model calls?

Method. I carried out the full pipeline: dataset selection, model deployment, trace generation, evaluation design, analysis, figure refinement, related-work organization, and writing. The experiments compare token statistics across MATH and BigMath traces from Qwen and Llama models, with controls for within-problem evaluation, fixed-direction scoring, and permutation-null calibration.

Finding. Protocol choices such as global pooling, in-sample scoring, and direction-agnostic AUROC can shift reported AUROC by up to about 0.18. Final-token entropy reaches 0.72–0.75 direction-agnostic AUROC, but drops to 0.47–0.48 under fixed-direction evaluation.

Takeaway. Shallow token statistics are useful diagnostics, not stable standalone verifiers unless the evaluation protocol is controlled.

Links: paper · code · workshop

I test one deliberately narrow feature family on DeepSeek-R1-0528-Qwen3-8B: token-confidence summaries, token-trajectory statistics, continuity, novelty, reflection count, and a small activation-derived descriptor. The study covers 7,680 math runs, 12,672 science runs, and 10,688 coding runs, with problem-grouped splits and best-of-64 reranking.

Result. The same feature family is highly diagnostic for math reasoning, partly useful for GPQA-style science questions, and weak on LiveCodeBench-v5 coding tasks: AoA moves from 0.958 in math to 0.799 in science and 0.434 in coding; best-of-64 reranking changes from +10.0 pp to +8.0 pp and then -0.6 pp.

Takeaway. The result is not “text cannot verify code.” It is narrower: these cheap CoT-surface features are domain-specific measurement instruments. They can track convergence-like behavior in math, but they do not reliably track executable correctness in code. Robustness checks include an 83-scalar coding sweep, grouped ablations, a CoT-only judge, MLPs, SSL pretraining, semantic-knot annotation, and token-level de-knotting.

Links: code · demo · technical note

An independent reimplementation and adaptation inspired by Learning to Reason in 13 Parameters, moved from math reasoning toward verifiable competitive-programming code generation.

The project uses Qwen2.5-Coder-3B, a tiny shared-parameter adaptation mechanism, and compile-and-run rewards rather than static heuristics. Its main value is the full research loop: data processing, training, multi-GPU setup, reward design, evaluation, and validation.

Motivation. Textbooks often begin with a polished definition, but a learner usually needs the problem first: what broke, why older tools were not enough, and what the new idea repairs. IntuitMath is built around that path from curiosity to rigor.

How it works. The skill guides an agent to explain concepts through motivation, examples, failed attempts, counterexamples, proof repair, and then the rigorous version. A definition should feel like the answer to a real problem, not the first line of a textbook.

How to use it. Install it as an agent skill and use prompts or slash-style commands such as /intuit-explain, /intuit-solve, /intuit-proof, /intuit-study, and /intuit-note. It can help with concept explanations, problem solving, proof repair, counterexample hunting, study planning, and polished Markdown or HTML/KaTeX notes.

Links: code · README

A compact C++ implementation built to understand transformer internals without relying on high-level deep-learning frameworks. The goal is educational: make the data flow, tensor operations, and model components explicit enough to inspect and modify.