The ARC Prize organization designs benchmarks which are specifically crafted to demonstrate tasks that humans complete easily, but are difficult for AIs like LLMs, “Reasoning” models, and Agentic frameworks.
ARC-AGI-3 is the first fully interactive benchmark in the ARC-AGI series. ARC-AGI-3 represents hundreds of original turn-based environments, each handcrafted by a team of human game designers. There are no instructions, no rules, and no stated goals. To succeed, an AI agent must explore each environment on its own, figure out how it works, discover what winning looks like, and carry what it learns forward across increasingly difficult levels.
Previous ARC-AGI benchmarks predicted and tracked major AI breakthroughs, from reasoning models to coding agents. ARC-AGI-3 points to what’s next: the gap between AI that can follow instructions and AI that can genuinely explore, learn, and adapt in unfamiliar situations.
You can try the tasks yourself here: https://arcprize.org/arc-agi/3
Here is the current leaderboard for ARC-AGI 3, using state of the art models
- OpenAI GPT-5.4 High - 0.3% success rate at $5.2K
- Google Gemini 3.1 Pro - 0.2% success rate at $2.2K
- Anthropic Opus 4.6 Max - 0.2% success rate at $8.9K
- xAI Grok 4.20 Reasoning - 0.0% success rate $3.8K.
(Logarithmic cost on the horizontal axis. Note that the vertical scale goes from 0% to 3% in this graph. If human scores were included, they would be at 100%, at the cost of approximately $250.)
https://arcprize.org/leaderboard
Technical report: https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf
In order for an environment to be included in ARC-AGI-3, it needs to pass the minimum “easy for humans” threshold. Each environment was attempted by 10 people. Only environments that could be fully solved by at least two human participants (independently) were considered for inclusion in the public, semi-private and fully-private sets. Many environments were solved by six or more people. As a reminder, an environment is considered solved only if the test taker was able to complete all levels, upon seeing the environment for the very first time. As such, all ARC-AGI-3 environments are verified to be 100% solvable by humans with no prior task-specific training
Grok Reasoning: 0%
Hilarious
Reasoning is woke propaganda anyway.
Grok isn’t designed to solve problems. It’s designed to create sexually explicit images of children for Republicans…
Well, yeah, it’s very good at making weird porn clips though. If anyone wants some very odd entertainment, go to /gif/ on 4chan and look at the reoccurring “/gg/ grok gens” threads. There’s everything from actually impressive and hot videos to the weirdest and most fucked up shit ever, it’s weirdly fun. Never seen anything really bad there, like CP etc. so I can comfortably recommend it for the lols.
As a psychiatrist, I have a theory about what’s missing in AI. First, it lacks childhood dependency and attachments. Second, it struggles to overcome repeated pain and suffering. Third, it lacks regular eating and restroom breaks. Fourth, it struggles to accept loss in everyday situations. Finally, it lacks the concept of our inevitable death. Without these nagging memories and concepts, machines will simply revert to the simpler concepts we use them for in our recent times, such as stealing cryptocurrency. After all, we live in a world run by capitalism, so it’s only logical. ¯\(ツ)/¯
Are you anthromorphizing word suggester into a being experiencing things?
No.
The major thing AI lacks is continuous parallel “prompting” through a variety of channels including sensory, biofeedback, and introspection / meta-thought about internal state and thinking.
AI currently transforms a given input into an output. However it cannot accept new input in the middle of an output. It can’t evaluate the quality of its own reasoning except though trial and error.
If you had 1000 AIs operating in tandem and fed a continuous stream of prompts in the form of pictures, text, meta-inspection, and perhaps a simulation of biomechanical feedback with the right configuration, I think it might be possible to create a system that is a hell of an approximation of sentience. But it would be slow and I’m not sure the result would be any better than a human — you’d introduce a lot of friction to the “thought” process. And I have to assume the energy cost would be pretty enormous.
In the end it would be a cool experiment to be part of, but I doubt that version would be worth the investment.
it lacks childhood dependency and attachments.
Isn’t general intelligence, or more broadly “consciousness,” a prerequisite to that? How would you make an unconscious machine more conscious merely by making mock scenarios that conscious beings necessarily experience?
it struggles to overcome repeated pain and suffering
That’s getting into phenomenology — why is pain an experience of suffering at all? How would you give it pain and suffering without having already made it AGI? We’re still missing the
<current-form> -> AGIstep.it lacks regular eating and restroom breaks
The necessity of which is emergent from our culture and biology, as conscious social beings. We’re still missing a vital step.
it struggles to accept loss in everyday situations
What is “loss” and “everyday situations” if not just a way we choose to see the world, again as conscious beings.
it lacks the concept of our inevitable death
How do you give it a “concept” at all?
these nagging memories and concepts
The AI in its current form has the “memory” in some form, but perhaps not the “nagging.” What should do the “nagging” and what should be the target of the “nagging?” How do you conceptually separate the “memory” and the “nagging” from the “being” that you’re trying to create? Is it all part of the same being, or does it initialize the being?
We’re a long way away from AGI, IMO. The exciting thing to me, though, is I don’t think it’s possible to develop AGI without first understanding what makes N(atural)GI. Depending how far away AGI is, we could be on the cusp of some deeply psychologically revealing shit.
It could also be that it lacks the machinery to feel any emotions at all. You don’t (normally) have to train people to be afraid of bears or heights or loneliness or boredom. You also don’t (normally) have to train people to have empathy or compassion.
I argue that our obsession with AI is, itself, a misalignment with our environment; it disproportionately tickles psychological reward centers which evolved under unrecognizably different circumstances.
I guess you don’t have children.
You absolutely do have to train them to be afraid of bears, heights, and every fucking thing you can imagine. You absolutely do have to teach them empathy and compassion. There may be some nugget of instinct, but without reinforcement it might as well not exist.
Hah, okay, you got me there. From my understanding, though, that’s mostly because kids are still figuring out what’s “normal”, so their fear instinct isn’t nearly as strong. I guess I should’ve stuck to the more instinctive sources of fear…
Regardless, that’s not really my point. My point is an LLM doesn’t rely on machinery in the same way that a human brain does. That doesn’t make AI “worse” or “better” overall, but it does make it an awful replacement for other humans.
You don’t (normally) have to train people to be afraid of bears or heights or loneliness or boredom. You also don’t (normally) have to train people to have empathy or compassion.
So what are you implying about people who don’t experience these?
What am I implying? That their machinery is abnormal and they likely need assistance to live normal, healthy lives. That’s literally why the fields of psychiatry and psychology exist: healthy people don’t need doctors and therapists. Do you disagree?
It’s fun to point at the crappy performance of current technology. But all I can think about is the amount of power and hardware the AI bros are going to burn through trying to improve their results.
Funnier yet will be if they continue to just train the model on that particular kind of test, invalidating its results in the process.
Tell me again how AGI is just around the corner, Sam
When Sammy fuck says “we’re so close to AGI, I can just feel it. Like a tingle on the tip of my shrimpdick it’s getting so close to blossoming into something guys”, just ignore him. He’s crazy man!
"Like a tingle on the tip of my shrimpdick it’s getting so close to blossoming into something guys”
Wow, that really is something. XD
I’m trying to stretch my creative muscles more XD
a tingle on the tip of my shrimpdick
mhh that’s erotic ASMR on Youtube
to be fair, he’s not human so he’s just guessing based on his observations earth as a demon
machines will be able to ‘think like humans’ when it happens
Maybe AGI is just a brain-destroying pandemic?
just when he had to shut down sora, because making ai videos is too expensive.
It’s almost as if a chatbot isn’t actually thinking.
I know lemmy’s very anti-ai but this is really fascinating stuff.
We’re anti-AI because AI is fucking stupid. Both literally and figuratively.
It really isn’t. But you do you boo.
Someone else in the comments said it perfectly. AI is just data regurgitation. It’s like calling me highly intelligent because I read you a paragraph from Wikipedia. I didn’t know anything. I just read a thing and said it out loud.
No. You’re not just wrong, you’re aggressively uninformed.
By you repeating the same tired “AI is just regurgitating data” line makes it clear you don’t understand what you’re criticizing. Calling large language models “AI” the way you are doing it just exposes that you do not know what you are talking about. It is like a creationist smugly saying “orangutang” instead of “orangutan” and thinking they sound informed. You are not demonstrating insight. You are advertising ignorance.
What you’re describing, reading a paragraph off Wikipedia, is literal retrieval. That is not how modern language models operate. They are not databases with a search bar attached. They are probabilistic systems trained to model patterns, structure, and relationships across massive datasets. When they generate a response, they are not pulling a stored paragraph. They are constructing output token by token based on learned representations.
If it were just regurgitation, you would constantly see verbatim copies of training data. You do not. What you see instead is synthesis. Concepts are recombined, abstracted, and adapted to context. The system can explain the same idea multiple ways, shift tone, handle novel prompts, and connect ideas that were never explicitly paired in the source material. That is fundamentally different from reading something out loud.
Your analogy fails because it assumes nothing is being transformed. In reality, transformation is the entire mechanism. Information is compressed into weights and then expanded into new outputs.
Is it human intelligence. No. Is it perfect. No. But reducing it to “just reading Wikipedia out loud” is not skepticism. It is a basic failure to understand how the technology works.
If you are going to criticize something, at least learn what it is first.
Counterpoint: Why should they learn about it?
It is a good thing to reduce ignorance, but there is more to learn in the world than there is time to learn or space in the brain. People must specialise.
You must accept that not everyone will understand everything, and this is okay.
The nature of a Large Language Model is very specialist knowledge, data regurgitation is apt from a distance, especially when most publically available models are primarily used for search.
Criticism must be accepted, even from those who do not understand, so long as it’s in good faith. It is after all an opportunity to reduce ignorance to someone with the time and interest to learn.
Don’t rudely lord your intelligence over someone else, it might not end well, and invalidates the delivery of your entire argument.
The reason he should learn about it is because he’s talking about it as though he’s informed and he is not.
I don’t have to be a LLM programmer working at openai to have a working knowledge of how these machines function. It’s literally just a Google search.
He made an unreasonable ignorant comment and I called him out. He should feel ashamed and I have absolutely no reason to pad down what I’m saying under the guise of being nice.
This might be the most comprehensive comment I’ve ever read about someone saying how utterly stupid they are to the world. It’s incredibly impressive how articulate you described your absolute lack of critical thinking.
It’s almost like intentionally shooting yourself in the nuts, and openly releasing the video of it saying you promote gun safety.
Calling an llm a Wikipedia regurgitator is factually and objectively incorrect.
Is there anything that you can say to refute the facts that I presented in my above comment?
(I rolled my eye so hard at your comment that I pulled my back out)
You’re discounting the fact that a human reading Wikipedia will attribute intonation and tone to the text to give further context and meaning. I think the analogy is good. Its not precise but it is the same thing.
I do think AI has a useful purpose and is here to stay. I don’t think it’s groundbreaking like the AI companies want us to think. The bubble will burst and then we’ll see where the cards lie.
OpenAI has lost their lead and I expect they will start to struggle with further funding. There are quite a few warning signs. The price of oil is likely to increase power prices generally and cause construction delays and cost rises. Both will hamper their plans. They still don’t have a viable model for profit.
The analogy is terrible and is not at all, once again, what llms do.
This is an objective fact I have provided evidence to support this.
How are you saying the analogy is good?
Ana analogy does not need to be precise. It expresses a comparison for easier understanding. It is not what LLMs do. However what you’ve expressed is simplified also. So by your standard, it is not useful for the discussion.
So maybe get your head out of your ass and try to understand what people are trying to express instead of correcting them when they are not incorrect.
If precision was of that much importance to you, you would have a different opinion of LLMs.
Can’t wait for this to be the new captcha.
I can’t see AI actually being intelligent until they no longer need to send a built up prompt of guides and skills and the chat history on every submission.
It’s no different from Alexa 15 years ago with skills. Just a better protocol and interface and ability to parse the current user prompt.
In my opinion of course.
Ya i agree. The whole infrastructure of how these work is flawed for a true AI/AGI.
It might be able to do a lot of cool things, but its fundamentally flawed at its core.
Someone will need to figure out something completely different for a true AI.
Oh also, I remember Elon once talked about how the upcoming cars would get bored when they weren’t doing anything with all that compute while parked so they could do use that compute and pay people for it.
Paying for the compute isnt a terrible idea in the future, but become bored? LOL. Fucking crazy talk.
Like even if it was a true AI that could be bored. You’re now going to enslave it to do what you want on its free time?
Yeah, if it’s got the capacity to be bored it’s not going to stick around waiting for you. Pets act out when bored, as will AI, better to let the ghost in the machine go have fun in an arcade or something.
Current models can pretend to be bored when directed to, but they’re only facsimiles of thought at the moment, and the current approach probably won’t change that.
Right? I have a Google Home Mini in our kitchen and if we ask it a question it just pulls a source from a website and tells us. That’s it. Nothing intelligent about it.
AI now is no different. It’s just pulling more complex wording from more (albeit illegally) sources to give a (albeit sometimes incorrect) better description of the question asked.
AI is just as stupid as Alexa is/was 15 years ago. It just has more information to pull from and still fucks it up.
LLM’s are just very well-read morons.
This replay is the funniest shit lmao. Keep building that bridge Claude.
https://arcprize.org/replay/0964128b-a2f5-4c5b-886e-497d893f429d
Interesting that it seems to be perceiving the environment mostly accurately, and is just completely wrong about the purpose of all the game objects.
I couldn’t find replays. Are there more? Also, it is a bit funny that “building the bridge” which at one point seems to be Claude’s “chosen goal” is just “running out of moves” and failing the task.
Task failed successfully, Claude. Task failed, successfully.
There’s a column linking to replays in the table of tasks here: https://arcprize.org/tasks
Here’s another reply where the model mistakes running out of time/move for making progress
it’s reasoning log is so fucking funny
My understanding is that Claude is particularly geared towards being a tool for people to use rather than a human replacement. That’s why they had that whole spat with the Pentagon about a human needing to be in the loop.
I tend to be anti-AI because it doesn’t seem to me to be anything other than a super fast regurgitator of data. If a database can be searched for an answer, AI can do that faster than a human. However it doesn’t to seem to be able to take some portion of that database, understand it, and then use that information to solve a novel problem.
Well… It cannot even search databases without errors.
LLMs just produce plausible replies in natural languages very quickly and this is useful in certain situations. Sometimes it helps humans getting started with a task, but as it is now, it cannot replace them. As much as the capital class want it, and sink our money into it.
Yes, the key thing is it might have extracted useful info from otherwise confusing data, it might have mixed up info from the data incorrectly or it might have just made it up.
So it can be useful, if you can then validate the info provided in more traditional means, but it’s dubious as a first pass, and sometimes surprisingly bad when it’s a scenario you thought it would work well at.
The better setup generate “semantic embeddings” that try to map how data stored relate to each other (by mapping how to it related within in its own weights and biases). That and knowledge graph look ups in which the links between different articles of data are evaluated in the same way.
The very expensive LLM portion really do just give rough aproximations of information language in that setup
If human scores were included, they would be at 100%, at the cost of approximately $250
Wait, why did it cost real humans $250 to pass the test?
That is how much individual testing humans cost when you buy them in bulk.
it’s also an odd metric since only 20-60% of the humans completed it. Very 60% of the time they complete it everytime energy.
Ideally they’d run the bots multiple times through (with no context or training of previous run), but I guess that is cost prohibitive?
Yeah, this is what I was going to call out. Calling it “100% solvable by humans” and saying “if human scores were included, they would be at 100%” when 20-60% of humans solved each task seems kinda misleading. The AI scores are so low that I don’t think this kind of hyperbole is necessary; I assume there are some humans that scored 100%, but I would find it a lot more useful if they said something like “the worst-performing human in our sample was able to solve 45% of the tasks” or whatever. Given that the AIs are still scoring below 1%, that’s still pretty dark.
Youd have to eat $250 worth of burgers to pass it.
Because I ain’t doing this shit for free.
This is my rough upper-bound estimate based on the Technical Report. Human participants were paid to complete and evaluate the tasks at an average fixed fee of $128 plus $5 for solved tasks. So if a panel of humans were tasked with solving the 25 tasks in the public test set, it would be an average of $250 per person. Although, looking at it again, the costs listed for the LLMs is per task, so it would actually be more like $10 per human per task. In any case it’s one or two orders of magnitude less than the LLMs.
Participants received a fixed participation fee of $115–$140 for completing the session, along with a $5 performance-based incentive for each environment successfully solved
Link to the recent Al Explained video mainly covering ARC-AGI-3:
https://www.youtube.com/watch?v=s4tptozUJ8YTry spelling things phonetically (example: faux net tick alley), that’s one of my benchmarks that AI fails almost every time.
If the input is at all long, or purposefully includes a lot of words about a specific unrelated theme to the coded message, it’s impossible.
Oh that’s an interesting challenge.
I hear some LLMs now have some solutions for the classic “how many Rs in ‘strawberry’” problem (related to the tokenization processes), but I have no idea how they might solve the phonetic thing. I’m sure some smart people will eventually find a way though
Wait, I thought phonetically (example: papa hotel oscar novermber echo tango india charle alfa lima lima yankee) meant using a phonetic alphabet, not using word(s) with the same Soundex encoding.
Yeah, there was some phonics in my primary school education, and I continue to approach new words in that way sometimes. But, they said Phonetically.
Phonetics is the study of speech sounds. The phonetic alphabet is called that because each letter/word in the alphabet was chosen to be one that started with the corresponding phoneme and that the set of words were between them phonetically unambiguous. Phonics is a way of teaching reading and writing that is based on the phonetics of words and how they relate to the written form.
Ii can thoroughly recommend “A Brief History of Intelligence” (by Max Bennett), which explains how intelligence has taken steps through evolution, what those steps were etc.
Spatial intelligence requires spatial understanding and it’s not something that can be solved through a large language model, IMHO.
I’m excited to see how these are solved. And I’m terrified to see how these will be solved.
ARC-AGI-3
What happened to ARC-AGI-1 and -2?
https://arcprize.org/arc-agi/1
https://arcprize.org/arc-agi/2
(They were more static, but yes, eventually frontier models got good at them.)
It’s true that frontier models got better at the previous challenges, but it’s worth noting that they’re still not quite at human level even with those simpler tasks.
Also, each generation of the challenge tries to close loopholes that newer models would exploit, like brute-forcing the training with tons of synthesized tasks and solutions, over-fitting to these particular kinds of tasks, and issues with the similarities between the tasks in the challenge.
A common strategy in past challenges was to generate thousands of similar tasks, and you can imagine the big AI companies were able to do that at massive scale for their frontier models.
AI won them
Ok. I’m sure AGI-1 and AGI-2 weren’t real AGI. What number is the real AGI? Is it AGI-3?
The goal of the ARC organization is to continually measure progress towards AGI, not come up with some predictive threshold for when AGI is achieved.
As long as they can continue to measure a gap between “easy for humans” and “hard for AI”, they will continue releasing new iterations of this ARC-AGI challenge series. Currently they do that about once a year.
More detail about the mission here: https://arcprize.org/arc-agi
Probably AGI-42
