How Good is Google Gemini?

Excitement followed by skepticism, followed by excitement again?

Author

Nathaniel Whittemore
December 07, 2023

The AI Breakdown First Five - Thursday December 7, 2023

Gemini Special Edition

All year, the chorus of people wondering “Where is Google?” has been growing.

First there was that internal memo that said open source was kicking their butts and that they had no moat.

Then, after Gemini was announced back in May, the question became “so when are we going to see it?”

This muttering grew to a chorus after OpenAI Dev Day. With every new launch, announcement and update from a competitor (particularly OpenAI), the pressure grew.

And yet, over the past couple weeks, the reporting has been that Google had been forced to delay Gemini into Q1. Last week we heard that Google had actually nixed public preview events for Gemini because of issues with the software.

So what a surprise yesterday when Google dropped their full announcement post.

The initial response was excitement that finally something had matched GPT-4. Although fairly soon, folks started noticing that the comparisons were not as clear cut as they were being presented.

The reality is that right now, no one really knows how Gemini compares to GPT-4 because of the quirk of how it’s being released. Gemini has three versions — Nano for running on-device; Pro and Ultra. Pro is ~GPT 3.5 level (although they claim it slightly beats 3.5 on 6 out of 8 benchmarks). Ultra is the model that they argue actually exceeds GPT-4 capabilities.

But Ultra isn’t available, and it appears it won’t be until next year.

The conclusion that leads me to is that this announcement was forced by circumstance and market pressure. The cacophony of people scream-asking “where is Google” was simply getting too loud, and they decided that launching an incomplete thing, with the promise that the big thing would be better than the other guys thing (when it actually arrived), was better than continuing to say nothing.

Were they right to make that call? Hard to say. Markets are certainly not convinced.

My guess though, is that when the dust settles, while the full blast of “someone finally beat GPT-4” excitement will have dissipated, the particulars of Gemini will actually still have people fairly excited, and virtually rabid to try by the time we get Ultra.

Check out Google’s blog announcement about Gemini here.

Thanks for reading! How did this edition hit you?

How was today's newsletter?

Login or Subscribe to participate in polls.