Turing Award winner Richard Sutton, author of the 2019 essay "The Bitter Lesson", writes in a new post that raw computing power alone is not sufficient for intelligence. Clever new methods are needed; AI science is "still far from the goal". The Decoder reports this based on Sutton's post on X, which drew wide response in the AI community.
Background
In his 2019 essay, Sutton argued that in the long run, general methods that scale with computing power prevail over specialised human engineering. According to The Decoder, the new statement is not entirely new: Sutton had previously criticised the limits of pure scaling of large language models, called synthetic data a "big mistake", and called for agents with world models that learn continuously from their own experience. Today's language models do not keep learning after training; what is needed is continual learning without destroying earlier knowledge.
Counterpositions
The Decoder also reports differing views among researchers. AI researcher Jerry Tworek, who left OpenAI, believes the bitter lesson still holds, but that the more computing power flows into model training, the more important the algorithm becomes. AI researcher Jayden Teoh criticises that the bitter lesson has repeatedly slowed academic research, as promising ideas were dismissed too quickly.
OpenAI CEO Sam Altman considers the scaling of large language models far from exhausted, according to the report. Anthropic CEO Dario Amodei describes scaling progress as a "rainbow without end". The Decoder notes that both, as heads of companies that bet on this approach, may be biased.
Assessment
For companies relying on further improvements of today's language models through greater computing power, the debate could mean that planning rests on uncertain assumptions about future scaling. Whether new learning methods will bring short-term progress is, according to the cited sources, open.
Sources
Editorially owned by Ideal Syka. Sources and method: Newsroom & method. Tips and corrections: ai@i6eal.de.
