Reinforcement Learning in LLMs - Why and How
From imitation to optimization: when LLMs need RL, how verifiable rewards unlock reasoning, and a minimal GRPO playbook.
•9 min read
•
Blog
28 posts
From Transformers to ChatGPT
This note provides a high-level summary of the progress in large language models (LLMs) covering major milestones from Transformers to ChatGPT. The note serves as a fast-paced recap for readers to catch up on this field quickly.
•36 min read
•
Exponential-Min and Gumbel-Max
Exponential-min and Gumbel-max tricks for reformulating sampling from a discrete distribution as argmin and argmax, making the sampling operation differentiable.
•2 min read
•
Why High-Dimensional Gaussians Feel Like Soap Bubbles
Concentration of measure pushes Gaussian samples onto a thin shell—here's the intuition, the math, and why typicality matters for generative models.
•6 min read
•
The Physics–Music Dilemma
Music notation gives us a tidy grid of notes, but physics delivers a messy spectrum of vibrations. Here's why tuning is always a compromise.
•3 min read
•
From PPO to DPO (and GRPO)
PPO made RLHF work; DPO made it simple. This post derives DPO from PPO, explains why it’s a supervised alternative (not RL), where it shines, and where RL/GRPO still helps.
•5 min read
•
Reparameterization vs REINFORCE
You know how to differentiate through a function—but how do you differentiate through a sampling step? Two estimators: score‑function (REINFORCE) and pathwise (reparameterization); pathwise backpropagates through the sampling transform with lower variance.
•4 min read
•
Speculative Decoding: Exact Speedups with Draft Models
Can we speed up generation without changing the distribution? A small draft model proposes, the big model accepts/rejects—yielding exact samples, faster.
•3 min read
•
Maximum Entropy and Maximum Likelihood
Why that particular sigmoid in logistic regression? This short post shows how simple moment constraints lead to exponential families (MaxEnt chooses the model) and how MLE fits them.
•6 min read
•
List of Advice
A living collection of advice from mentors, friends, and books.
•9 min read
•
Expectation-Maximization Algorithm in 10 Minutes
A quick walk-through of Expectation-Maximization (EM) algorithm and its cousins.
•11 min read
•
Diffusion Models
This is the first post of hopefully a series of post walking through diffusion models. This post will introduce the foundations, focusing on two foundational papers, that many other papers built upon.
•5 min read
•
Life Is Short, a Reading List
A living list of books, essays, and videos that helps me keep perspective on life.
•1 min read
•
TIL: React Server Components
•4 min read
•
Inner Peace Requires External Validation
•1 min read
•
Hume’s Law
A quick riff on Hume's is–ought gap—why facts don't dictate values, and how the leap from 'is' to 'ought' rests on sentiment.
•1 min read
•
Dopamine
There has been a lot of confusing information about dopamine. I finally found a literature review-style article, and here is what I learned.
•5 min read
•
Hiragana and Katakana for Chinese Speakers
A quick reference tying each hiragana and katakana character back to its Chinese origin, plus the patterns and mnemonics I lean on while studying.
•2 min read
•
London
A January dash to London to finally see Jay Chou live, with museums, parks, and good meals along the way.
•1 min read
•
Building LLM-Powered Products
This is a quick note to discuss a few topics below related to building LLM-powered products and applications, such as how to let LLM use tools and become autonomous agents, how to incorporate domain adaptation, and the production hurdles.
•8 min read
•
How Does Auto-GPT Work?
In this note, we'll take a look at how Auto-GPT work and discuss LLM's ability to do explicit reasoning and to become an autonomous agent. We'll touch upon a few related works such as WebGPT, Toolformer, and Langchain.
•5 min read
•
Italy
In October 2021, we spent two weeks traveling to various cities in Italy, including Rome, Cinque Terre, Florence, Tuscany, and Venice. This was our first trip to Italy, and we have documented our journey with a report and photos.
•9 min read
•
Utah
Trip report (itinerary and photos) from our recent trip to southern Utah (Zion, Arches, Canyonlands and Bryce).
•7 min read
•
Recent Progress in Language Modeling
This page is a high-level summary / notes of various recent results in language modeling with little explanations
•3 min read
•
NLP Starter Resources
A list of starter resources for Natural Language Processing (NLP), mostly with deep learning.
•1 min read
•
Quantitative Tech Interview Preparation Guide
A short list of interview preparation resources for Data Scientists, Machine Learning Engineers, Machine Learning Scientists, Quant Developers and Quant Researchers.
•6 min read
•
Recent Progress in Neural Variational Inference
A literature survey of recent papers on Neural Variational Inference (NVI) and its application in topic modeling.
•1 min read
•
A Brief Survey of Generative Models
A high-level summary of various generative models including Variational Autoencoders (VAE), Generative Adverserial Networks (GAN), and their notable extentions and generalizations, such as f-GAN, Adversarial Variational Bayes (AVB), Wasserstein GAN, Wasserstein Auto-Encoder (WAE), Cramer GAN and etc
•7 min read
•
End of posts • 28 posts