2026-03-02 11:24 UTC
Classically pre-training was done for neural networks to train the domain symmetries to them. It is possible to handcraft neural architectures to have an inductive bias for certain kinds of symmetries, like CNN layers with pooling for translation invariance.
Remember that ANN type neural networks are just differentiable computation graphs.
Handcrafted invariant operations are however typically very clumsy and inefficient, as you can see from classical machine vision using things like SIFT and LBP. It is also impossible in practice to handcraft operators which are invariant to more complex symmetries like perspective or time of day in photos.
So, people used a neural backbone with fewer inductive biases at the outset, but used a lot of representative data signal to pre-train these networks to be able to tease and decompose the different hidden explaining variables out of them, to produce representations which are component-wise invariant to different domain symmetries. In plain language, the internal embeddings of these networks can have a specific activation pattern which encodes a cat, no matter what the time of day or the perspective is.
So, nowadays we have encoder-side Transformer-type models with very few strict inductive biases, except that the signal makes a causal sequence and needs to be represented as tokens. We also have way more data than we used to have.
So, what happens with representation learning? We don't learn simple symmetries anymore, but we start learning transferrable knowledge and transferrable cognitive skills as well. Some of this is because of the causal representation of the signal.
Is there more? Is intelligence anything more than transferrable knowledge and cognitive skills? I don't think so.
If a machine learns the decomposed representations of the symmetries and hidden explaining factors of the signal modalities, and furthermore the knowledge and cognitive skills represented in the data, we already have the holy grail of #AI in our hands.
Then the question becomes to be about scaling it up and applying it to everything which is bottlenecked by knowledge and skills. And here we are.
If you need help navigating the changing world under the AI driven transformation, I am #OpenToWork. I am an AI generalist with over 25 years of experience.
Replies (0)
No replies.