Image source: generated by Unbounded AI

It seems that the next leap in AI cannot be reported on time.

On the 20th local time, according to the "Wall Street Journal" report, OpenAI's new generation artificial intelligence project GPT-5 (codenamed Orion) is facing many difficulties. The project has been in development for more than 18 months and has incurred huge costs, but has yet to achieve the expected results.

Informed sources revealed that Microsoft, OpenAI’s largest “funder”, originally expected to see the new model around mid-2024. OpenAI has conducted at least two large-scale trainings, each time taking several months and consuming massive amounts of data. However, new problems emerged each time and the software was unable to achieve the results expected by the researchers.

Analytics believe that there may not be enough data in the world to make it smart enough.

01. The huge cost is staggering, and the GPT-5 project is not progressing smoothly

Analysts have previously predicted that technology giants may invest US$1 trillion in artificial intelligence projects in the next few years. Other estimates show that the computational cost of a six-month training session for GPT-5 alone is as high as approximately $500 million. OpenAI CEO Sam Altman said that future AI model costs are expected to exceed $1 billion. But people familiar with the project said:

"Although Orion's performance has improved compared to OpenAI's current products, it is not enough to justify its huge operating costs."

In October this year, investors gave OpenAI's $157 billion valuation is largely based on predictions from Altman, who previously said GPT-5 would be a "big leap" and said GPT-4 behaved like a smart high school student, but ultimately GPT-5 is actually more like having a PhD on some tasks.

GPT-5 should be able to unlock new scientific discoveries and complete everyday human tasks such as making appointments or flights, the report said. The researchers hope it will make fewer mistakes than existing AI, or at least admit "suspiciousness," as current models can hallucinate.

However, there is no fixed standard for "when can it become a smart enough AI", and it is more based on feeling.

So far, GPT-5, which is under development, still feels not strong enough. Altman said in November that "no product named GPT-5 will be released in 2024."

02. Data shortage has become the main bottleneck

In order not to make huge The researchers tried to minimize the chance of such failure through small-scale trial runs.

However, GPT-5’s planThere seemed to be problems with planning from the start. In mid-2023, OpenAI began a training run that also served as a test of Orion’s proposed new design. But the process has been slow, suggesting that training on a larger scale could take a very long time, which in turn could make it prohibitively expensive.

OpenAI researchers decided to make some technical adjustments to enhance Orion. They also found that to make Orion smarter, more high-quality, diverse data is needed. Model testing is an ongoing process, and large-scale training runs can take months, with trillions of tokens being "fed" to the model.

However, data such as news articles, social media posts, scientific papers, etc. on the public Internet are no longer sufficient to meet the demand. Ari Morcos, CEO of DatologyAI, said: "It became very expensive, and it was difficult to find more data of the same high quality." To solve this problem, OpenAI chose to create the data from scratch. They hire professionals such as software engineers and mathematicians to write new code or solve mathematical problems to use as training data.

The company is also working with experts in fields such as theoretical physics to explain how they will tackle the field's toughest problems, but the process is very slow, with GPT-4's training using approximately 13 trillion tokens. . Even if 1,000 people write 5,000 words a day, only 1 billion tokens can be generated within a few months.

OpenAI has also begun developing "synthetic data", using AI-generated data to train Orion, and believes that failures can be avoided by using data generated by another of its AI models, o1.

03. Google is catching up, and OpenAI is panicking

< p>This year, as Google launched NotebookLM, its most popular new AI application, OpenAI is even more panicked.

As Orion stalled, the company began developing other projects and applications, including a stripped-down version of GPT-4 and Sora, which can make AI-generated videos. But people familiar with the matter said this has resulted in competition for limited computing resources between teams developing new products and Orion researchers.

In addition, OpenAI is also developing more advanced reasoning models, believing that by letting AI "think" for a longer time, it can solve complex problems that have not been encountered in training.

However, these new strategies also face challenges. Apple researchers found that inference models, including OpenAI’s o1, were likely simply imitating training data rather than actually solving new problems. In addition, o1's method of generating multiple answers also greatly increases the chance of success.operating costs.

Despite this, OpenAI is still relentlessly advancing the development of GPT-5. On Friday, Altman announced plans for a new inference model that is smarter than anything before, but did not say when or if it would launch a model worthy of GPT-5.