The Boy Who Cried AGI

Mark Zuckerberg recently came out saying that "AI" will soon be able to do what mid-level engineers do. Meanwhile, some claim AGI (artificial general intelligence) is already here, while others are arguing about how many decades away it is.

As an ML practitioner with 14 years of experience, all the talk of AGI scares me, but not for the reasons you would expect.

Back during my time at Twitter, I was working on an ML evaluation tool that went beyond a single metric. The purpose of the tool was to give you a scorecard with everything you need to know about model performance in real life before launch. This was to work for every model at Twitter and beyond, as we were talking about open-sourcing it.

To those who do not deal with ML every day, this may seem like a hard problem: implementing a whole bunch of metrics for various architectures. Yet the real difficulty was understanding what to test and how.

Fast forward to December 2024, and the conversations I've had with the brightest minds in the industry at NeurIPS confirm we have no idea how to truly evaluate LLMs (which have become synonymous with AI), to say nothing about agents. We have bits and pieces of the answer (I helped with some of them), but no true consensus on what to measure and how to ensure the measurement is correct.

Definition of Intelligence

AGI is a poorly defined problem. The very notion of intelligence as we discuss it is human-centric. Perhaps dolphins, with their empathic abilities, would be a better benchmark.

Poorly defined problems lead to machinations. I have seen this in the industry: a team of MLEs exploits the poor definition of a problem to pronounce it solved before it can deliver any value. I am afraid that not having a good definition for AGI breeds many such teams.

Even the famous "Turing Test" has several variations and thus different barriers depending on which passage you read and how you interpret it.

I must admit that for some definition of the Turing Test, AGI has indeed been achieved, but it is far from the technological singularity anyone would expect when we hit "True AGI," whatever that means.

Singularity

The concept, borrowed from Physics and popularized by Ray Kurzweil, describes a point after which things are so different that you cannot predict them before the event takes place. A more-than-human intelligent system that can scale infinitely is certainly such a singularity.

It makes sense to take this very seriously. It makes sense for humanity to think through laws and best practices and figure out as much as we can before it happens, even if we can be assured that we won't be fully prepared because it is, by definition, unknowable exactly what we need to prepare for.

I have seen enough in my career to know that such level of intelligence is likely achievable by machines, and the economic and national incentives for some seem to be very aligned with achieving it. Thus, I believe the point of technological singularity will come, but when?

Timing is Everything

If I told you that your water heater would go out in exactly 34 days, you would be able to be quite prepared when it happens. If I told you that it would go out in 34 seconds, much less so.

Your preparation depends on how much time you have.

Yet a much more sinister scenario is that I tell you that your water heater will go out in exactly 34 seconds... and then it does not. You are relieved about it and maybe a bit frustrated with me, but instead of backing down, I say, "No, it will really happen in the next 34 seconds." This goes on for 34 days when the water heater actually bursts, but you are not ready because you stopped listening to me.

I call this "the boy who cried AGI" and fear that we are very much in this mode today.

Raising capital depends highly on showing ambitious claims that are near enough for investors to care. This leads companies to overpromise and use the loose definitions in their favor. In the meantime, the progression toward AGI is truly happening, but instead of going to the store to buy a wrench to fix our water heater, which takes an hour, we keep dragging towels into the utility closet because we keep being told that we only have 34 seconds.

Eight years ago, all these charlatans told us that there would be no truck drivers on the road in 5 years, nor would there be any radiologists. Yet, as of my writing this in 2025, there still are "a few."

We All Die Eventually

I cannot predict when AGI will come, nor can anyone else. True intelligence of that sort will require several breakthroughs. I think that the concept of embeddings was one such breakthrough, and arguably the attention mechanism was another.

How many more do we need and when will they come? I don't know. Perhaps some PhD candidate is today working on the final breakthrough, or maybe it will be my great-grandchild or yours.

This timeline uncertainty makes it difficult to plan ahead. But in this area, humans have a concept we are all familiar with: death. No one knows for certain when they will die. For many, there are signs that death is near while for others there are not. We know that some activities will make death more likely and that our loved ones will be better off with certain choices over others.

When preparing for your own demise, avoiding extremes and thinking beyond your own interest are two essential qualities.

AGI may come tomorrow. Then we are all in trouble. Or it may be so far away that it won't affect us. Yet the most likely answer is somewhere in between. So let us have public debates about what we should be solving and how, let us trial different regulations and restrictions, let us keep sharing our findings as the ML community openly and transparently, let us educate the general public on the successes AND failures of the latest techniques, but more importantly, let us think beyond our own self-interest.

There is both an empathetic argument to be made and a self-interested one. The humane argument is that a technology like AGI will make it far too simple to magnify our worst instincts: to subjugate and oppress, to cause suffering and pain, and to worsen the human condition for the majority of the population.

On the other hand, and this is the rational argument, I am not convinced that any person, corporation, or country will be able to wield such power. As we discussed, we cannot be certain of what happens beyond the singularity; the models may just get smart enough to know that the leaders of AI companies and their shareholders who were motivated by greed are the real problem with the world—and who knows what happens then?

Maybe better not to risk it?