Highlights
What I built
- Typed graph nodes with values, gradients, parents, and operations.
- Reverse-mode engine validated against finite-difference gradchecks.
- Training loops show consistent loss decrease on toy problems.
← Back to Projects
Machine LearningIteratingFeatured
MiniTorch-OCaml
Reverse-mode autodiff engine with custom computation graphs.
I wanted to understand backprop beyond framework APIs, so I implemented graph nodes, reverse traversal, and gradient checks in OCaml.
- Date
- Spring 2026
- Signal
- ML Systems
- Build stage
- Extending toward richer tensor operations
- Stack
- OCaml, Autodiff
ml-systemsautodiffocamlbackprop
Autodiff visual
Project notes
Architecture
How the system works
- Tensor ops construct a directed graph during forward pass.
- Backward traversal propagates gradients through parent references.
- Optimizer updates parameters from accumulated gradients.
Challenges
What made it hard
- Gradient bugs were subtle and rarely obvious from forward outputs.
- Type-safe abstractions had to stay flexible enough for new operations.
- Keeping the API small while preserving useful experimentation loops.
Lessons
What I learned
- Math correctness needs numerical checks, not confidence.
- Strong types catch structure issues, not calculus mistakes.
- Loss curves are the practical proof that autodiff is working.
Stack / materials
OCamlAutodiffComputation GraphsGradient Checking
- Adding new operators now always includes forward + backward + gradcheck tests.
- Vectorized backend is a future step once core operator set is stable.
Media timeline
Build photos, clips, and process visuals. The goal is to show how the project evolved, not just the final screenshot.
Backprop flow
Build snapshot
Iteration snapshot
Gallery 4