Nando de Freitas
@NandoDF

I seek to understand intelligence & agency and build AI aligned with compassion, freedom & universal human empowerment through progress in science & engineering

 London, England
scholar.google.com/citations…
Joined April 2009
446 Photos and videos
Article

Distillation, RL, DAgger and causality: when imitation is or isn't SFT

DAgger (Dataset Aggregation) is a powerful algorithm in imitation learning that addresses one of the key shortcomings of standard behavioural cloning (ie supervised learning, aka SFT in LLM jargon):

4
24
186
23,808
Article

June 1: The Flow Matching Loss of Gen AI

Everyone is using flow matching to generate images and videos, and even protein structures, text, and any data. There are many papers on this, with many mathematical derivations, but all those

1
17
129
26,716
Article

May 30: Flow matching

Diffusion allows us to map samples from Gaussian distributions (noise) to actual data over a few diffusion iterations, resulting in images, protein structures, audio, text, video, code, etc. We did

1
15
167
33,466
Article

May 29: Classifier (free) guidance and DDPM

To wrap up our discussion on diffusion, and before we embark on flow matching, I would like to present a very important trick: classifier free guidance. This is a must know for anyone wanting to get
7
34
8,554
Article

May 27th: Diffusion generative models

In the last few days, we witnessed some amazing results from @GoogleDeepMind on text generation and video generation (Veo 3) and several teams, including my current one @MicrosoftAI , on audio

3
22
144
25,054
Article

May 16: The diffusion loss to train Gen AI models

We started the explanation of generative AI by stating the fundamental intuition: if what you imagine does not match reality, then you must update the imagination mechanism. The mechanism here is a

3
6
38
17,695
Article

May 14: The forward process of diffusion

Today we look at how to implement the forward diffusion process, i.e. how to add noise to the image in a smart way. Then, tomorrow we will derive a loss that enables us to train a model to undo the
7
17
5,546
Article

May 13: Score matching intro

The model distribution can be expressed in a very general mathematical form, the so called exponential representation: Z(theta) is known as the normalising constant or partition function. It ensures

2
3
23
6,197
Article

May 12: Matching imagination to reality

The data (images, proteins, videos, songs) is represented with the generic vector x. The real data is assumed to come from an unknown distribution p_d(x). Since we don't have access to this
2
8
4,266
Article

May 9: Diffusion

Diffusion is one of the most successful ways of generating images, video, music and even protein structures. Example applications include Lyria, Veo, Imagen, Sora, Dalle, and newer versions of models

4
7
79
16,458
Article

May 8: Homework, CoT to help primary school children

I hope you've enjoyed the posts about RL for LLMs. Now it's time for homework. I can't escape being a professor 😅 The homework is use any good coding app like Gemini, ChatGPT, Copilot, Claude etc to
3
6
6,079
Article

May 7: Agentic AI, tool-use and multi-turn dialogue

In tool-use (or Agentic AI as people call it these days), e.g. where an AI agent uses an API to get information or requests a lab test, the API returns data, which the agent uses as observations to

1
4
30
7,205
Load more