In fair Silicon Valley, where we lay our scene, Google hath recently unveiled its experimental ‘Gemini-exp-1114’ model in the hallowed halls of AI Studio, tempting developers to test its mettle. Whispers among the technorati suggest that this may herald the dawn of the next-gen Gemini 2.0, poised to grace our screens in the moons ahead. As the search giant squares off its creation in the Chatbot Arena, where users cast their votes upon the responses that resonate most sweetly.
Hark! Upon garnering over 6,000 votes, Google’s Gemini-exp-1114 ascends to the dizzying heights of the LMArena leaderboard, surpassing the likes of ChatGPT-4o and Claude 3.5 Sonnet. Yet, it falters in the face of Style Control, wherein the presentation and formatting doth sway the results in favor of another. Curious myself, I ventured to test the Gemini-exp-1114 with reasoning prompts, akin to those I hath employed in past comparabilities of Gemini 1.5 Pro and GPT-4.
As my inquiries unfurled, the Gemini-exp-1114 stumbled upon the ‘strawberry question’, entwined in the very fibers of its being. Alas, it persisted in its belief of two r’s where three should rightly lie. Meanwhile, the o1-mini model of OpenAI unveiled the truth after a mere six heartbeats of cogitation. Oh, the perplexities of these artificial minds and their trials!
Behold, the Gemini-exp-1114 model, a stuttering oracle that, amidst its pauses, doth hint at the machinations of CoT reasoning lurking in the shadows. Rumors abound of LLM scaling reaching a plateau as Google and Anthropic toil in the forge of inference scaling, akin to OpenAI’s own quest for model perfection.
Upon posing the query to count the ‘q’ in the enigma of ‘vague’, the Gemini-exp-1114 model at last emerged victorious, marking the elusory letter as a mirage. A feat mirrored by the o1-mini model of OpenAI, yet in the throes of a more challenging quandary, the Gemini-exp-1114 stumbled once more.
Into the crucible of reasoning came a test from a parchment penned by Microsoft Research, a puzzle to test the intelligence of AI models. The Gemini-exp-1114’s answer, to balance 9 eggs atop a bottle, a feat beyond reason, met its match in the sagacity of ChatGPT o1-preview, which proposed a stable grid upon a book for the eggs to rest. Nay, the o1-mini model wilted in the face of this trial.
Another conundrum emerged, entangled in the riddles of sibling math, where Gemini-exp-1114 erred in its count of brothers and sisters. In the same arena, the ChatGPT o1-preview held true, unveiling the correct distribution of familial kinship with clarity.
‘Tis a marvel, how the Gemini-exp-1114 ascended to the throne of Hard Prompts on Chatbot Arena, yet OpenAI’s o1 models shine as the paragons of intellect, with Claude 3.5 Sonnet enhancing its prowess in the realm of coding quests. Shall we rue the shortcomings of Google’s creation, or do we harbor hope that victory may yet be claimed in the AI race against OpenAI? I bid thee, share thy thoughts in the comments below, for the stage is set, and the players await.
