I can’t get this working at all. I came here to help you guys with this damn operation, not sit around and stare at plants all day. I'm keeping this damn cannon here, not all of us are willing to waste time on your stupid plant spells. This lets us bypass all of your stupid rules.
You should be thanking me, if anything. Now you don't have to worry about ..... (the letter cuts out)
By the way, I’ll provide for the maintenance myself.
-Gerberon
PS: Don’t blame me if you use this without prepping it, you idiot.(Note, article written March 20 2026, published Jul 19 2026)
It’s time for my first Kaggle competition! As someone who frequently enters gamejams, hackathons, and shows his work off a lot, I’m actually surprised I haven’t entered one earlier.
I love these little online competitions that anyone can enter and do stuff in. It’s so nice to learn, get feedback, and check things out!
Measuring Progress Toward AGI - Cognitive Abilities competition
This is a joint virtual hackathon between Google and Kaggle. Recently the google deepmind team has been thinking of new benchmarks and decided to hold a competition. The setup is simple:
-
Use their SDK to make a benchmark.
-
Generate a dataset of testing.
-
Gather results and display them.
Of course I’m exaggerating… There’s a lot of steps involved with choosing what to measure, how to measure - things like that.
My idea
I play a lot of DND (Well, I DM a lot of DND) and something I love doing is interactive logic-based puzzles.
I hate it when I play a game of DND and progress grinds to a halt because the DM asks us a riddle to a door like "How do you rhyme orange with Banana?" (Answer: Bornana) or ask us a math problem. You probably have heard of the famous riddle where 2 people are guarding a door but 1 is lying and the other always tells the truth. Cliches like that.
HOWEVER people LOVE it when you make interactive logic puzzles.
For example, I had a session where a bunch of evil Druids had a water cannon that they would spray onto huge bushes to make them easier to pass. This took place in a huge 60x60 room, and there were these massive walls of hedges, and the players had a tough combat encounter where if they passed the bushes they would take damage and lose movement for their turn.
At the end of combat, the players were able to interact the water cannon (I gave a visual of a ballista with some water tanks and piping hooked up) and try stuff out.
Let’s break it down:
-
Goal: players need to go to next room for the adventure
-
Problem: Players
-
Solution…? Players can interact with this strange cannon in the back….
Also these were wet hedges that resisted flames, and resisted all forms of physical damage. I came up with that on-the-fly because my players wanted to brute-force it with bombs and sword slashes.
With that being said - there are some special bushes that are very discolored. Despite being physically strong, they seem different than the others in some intangible way…
The puzzle works like this:
-
There’s two massive gears that jut out of the cannon. One that’s vertical and clean, and one that’s extremely busted and cut up. The messed up one now looks diagonal.
-
There’s a note that says the following:
(Gerberon is a dwarf the players just killed.)
-
The cannon has a large button that has a note that says "RESETE" in painfully mangled handwriting.
So, what do you do dear player?
(Here’s some player actions, then some narration on how I respond.)
-
I press the reset button without doing anything else.
"The cannon loosely pittles out some water."
-
I press the vertical dial and press the reset button
"The cannon’s angle changes, pointing upwards. It pittles out some water."
-
I press the diagonal dial and press the reset button
"The cannon moves diagonally, moving around the platform in an extremely jerky and wide manner. Thankfully, you dodge before the cannon hits you - but you ultimately are not so lucky (pause for effect). You dodge the cannon, but a stream of water, the strength of a sink sprays you right in the face."
The solution:
-
pressing the diagonal dial increases the water pressure, however it also makes everything much more inaccurate (move 7 spaces horizontally, 2 spaces vertically). The horizontal spaces reset going from left 0 → right 35 → hard snap back to 0, but the vertical one doesn’t reset.
-
Therefore, the players must press the diagonal dial at least twice, and adjust the vertical upwards
-
That every time the rotation increases, the horizontal movement increases (7/2 to 9/2).
-
that going too far vertically resets the puzzle (but I let my players quickly reset the state to how it was before)
Let’s assume we have a grid where 0,0 is in the bottom left (imagine like quadrant 1).
This puzzle is a grid puzzle in which pressing diagonal moves you pretty intensely
Hedges the players need to spray to escape:
(14,3) (21,5) , (35, 10) (0, 13) (0,15) (9, 17) (and so on)
OK I hope you’re still with me. The big idea:
Interactive logic puzzles are a unique form of intelligence test that rely on intuition and understanding on a deeper level.
Would an LLM understand the relationships of:
-
the vertical and horizontal movement
-
moving the diagonal dial for water pressure
-
Increasing the rotation amount over time
That’s what I’m testing.
This example I give is a classic example of EMERGENT INTELLIGENCE.
-
gathering cause/effect
-
making plans
-
not repeating blindly
So our new benchmark is going to be similar to this puzzle.
Sidenote: This is similar to video-game based benchmarks
Video games offer an interactive environment for models to gain intuition about their environment and tasks. They can try things out, see if it works, and then try new strategies to reach a goal.
Video games have been used a LOT in AI research for this very reason. It’s also much cheaper to through an AI into a virtual environment where they can do dangerous stuff and break things - instead of bringing them to the physical world.
4/18/2026 : I finally submitted
I finally got the chance to submit my writeup.
Read about it here: https://kaggle.com/competitions/kaggle-measuring-agi/writeups/can-llms-solve-a-simple-interactive-dnd-puzzle
Overall I’m not that positive about winning…
I ended up not being able to focus that well, and needing to spend a lot of time bouncing between different projects. Fundamentally - I didn’t spend enough time on this. I thought this would be a fun 10 hour diversion - but instead this is maybe a 50 hour benchmark.
The kaggle SDK (some constructive criticism)
This benchmark involved me using a new, framework that I’ve never heard of before. This isn’t my first time I’ve done this - around 2023 I had an unreal engine fortnite hackathon in which my team used the beta version of Verse to apply some unique actions to the game.
The goal was for us to write inside the lines. To not do anything novel - and instead to do something technically simple. But I like doing things on the wild side. Is that a personality flaw? Perhaps.
I’m bringing this up because back then, I felt like I was a beta tester. I was trying an extremely experimental library that didn’t work and had bad documentation - and then I tried doing something crazy that wasn’t supported. So, I ended up having to write a lot of custom logic, and provide a lot of feedback.
Kaggle benchmarks is REALLY good for single turn benchmarks. It’s fast and provides a lot of good metrics.
It sucks for multi turn benchmarks. Everything useful is hidden, everything that I need to look at is hard to find, and there’s so many edge cases for multi-turn that I don’t have to worry about for single-turn.
In fact, getting multi-turn working was a bit of a mess. In every AI program I’ve made, it’s always been super simple, but changing the mindset from "I control everything and know exactly what’s happening" to "Kaggle controls everything also their docs don’t help" was tough for me.
A good example would be model 429 request errors. I would prompt a model once, get a response, then send another prompt. The kaggle SDK, by default, had no rate limiting or error handling by default, so I would get an error, get an insane nonetype stack trace, and have no idea how to get this fixed.
My benchmark was 30 prompts. There were so many times when I’d randomly fail, and have 0 idea why.
Another annoying thing is that these benchmarks take forever to run. Something I LOVE in programming is the instant feedback loop. I make an edit, then 5 seconds later I can see how it influences the project.
Both of these together combine to make what I’d call the CI/CD painline. The iteration loop is painful, the documentation sucks, and you have no control over anything. Just like deploying to Google Cloud or Azure.
I eventually ended up asking Gemini 3.1 to vibe code the entire task for me after giving it a huge prompt. The agents.md provided on the github gave a lot of help, and was much easier to understand than the regular documentation.
I tried getting support by asking questions in the Kaggle discord (no response) and the kaggle discussions tab (no response). I really, really would’ve liked it if we could touch base with a living person to ask for help. With that being said, they did host a FAQ livestream and answered a ton of questions.
But overall I enjoyed my experience
The kaggle community is great, the Kagglers/Googlers involved were really cool, and everyone was super nice to learn from and interact with.
I initially came in with the intention to win, but I wasn’t disciplined enough to give this project the time it needed. Oh well. Better luck next time!