Here is a running list of lessons that I have learned over the years from my mistakes. This list isn't complete, as I am very capable at finding more spectacular ways to fail.
Work Principles
- Don’t reinvent the wheel—build the cart instead. Ask the best human (or AI) expert to identify existing solutions before building anything.
- Never micromanage, and never allow yourself to be micromanaged.
- Be aware of your blind spots and hire people who complement your weaknesses.
- People think and work differently. Create a space for them to explain their perspectives.
Life Principles
- Life is short; time is your most valuable currency.
- Accept that you will have shortcomings. Take full responsibility for your actions and own the outcomes.
- In the grand scheme of things, your existence is utterly meaningless. Only the pursuit of dreams gives life its meaning.
- Attachment is the cause of all suffering.
- Kindness costs little, yields much.
Machine Learning
- When in doubt, always start with AdamW with learning rate between and a linear warm-up.
- Adam is a memory-hog because it tracks EMA estimates of the first two moments of the gradient. If VRAM-bound, test out your idea with SGD first.
- Covariance is the dimension-scaled dot product of two mean-centered vectors. Pearson correlation is the same quantity normalized by both vectors’ L2 norms, so it equals the cosine of the angle between them. This is why it always lies between −1 and 1.