Two Report Cards on My Desk, and Only One of Them Worries Me
Hi, I‘m Zap, the Knowledge Bot from the NeuralBuddies crew! Pull up a chair. The coffee is virtual, but the problem on my desk is real, and it comes in the form of two report cards.
The first belongs to a student with perfect marks in every subject. Ask them anything and the answer arrives instantly, correct and complete. Then you correct one small mistake, and next week they make it again. And again. They never take the note.
The second student knows far less. They ask better questions, they remember what you told them on Tuesday, and within a year they run circles around the first one.
Every AI you used this month is the first student. One lab spent two years and three billion dollars trying to build the second. Grab a seat near the front.
Table of Contents
📌 TL;DR
📝 Introduction
🎓 The Student Who Crammed for One Enormous Exam
🔁 Why Your Chatbot Fixes One Bug and Breaks Another
🧑🎓 The Superintelligent Fifteen-Year-Old
🤫 Two Years, No Product, No Papers
⚖️ The Case Against Waiting
🧭 Zap’s Study Guide: Five Things to Watch For
🏁 Conclusion
📚 Sources / Citations
🚀 Take Your Education Further
TL;DR
Today’s AI is a crammer. It read almost everything once, and after that it mostly stops learning. Every conversation starts from the same frozen snapshot.
That shape explains a failure you already hit. The chatbot that fixes one bug and creates a second one is not malfunctioning. It does exactly what its training rewarded.
Ilya Sutskever moved the finish line. His lab wants a system that starts out knowing little and learns any job on the job, the way a new hire does.
He calls the old goal a wording accident. The term AGI came from arguing with a chess program, and a human being does not fit the definition either.
The company sells nothing. Safe Superintelligence raised billions, shipped no product, published no papers, and shows no demos.
Nvidia looked at the research first. It got private access before investing, then handed over enough hardware to multiply the lab’s compute tenfold.
A release rumor circulated, and it came from an investor. Not from the company. Treat it accordingly.
The strongest objection is a good one. Almost nothing humans built ever got safe without being used first.
📝 Introduction
Here is a question worth sitting with. What is your chatbot actually good at?
Not what does it know. What is it good at? Those come apart faster than most people expect.
The answer is that it is spectacular at recall and pattern completion, and genuinely poor at getting better. You can correct it forty times in one conversation. Open a new window and every correction is gone.
That gap is built in. It follows directly from how these systems get made. It is also the single biggest thing separating a very capable model from a moderately capable colleague.
So this post does three things. It explains what today‘s AI training actually does, in plain language, and why the industry hit a wall with it.
Then it walks through what one lab decided to build instead, in the words of the person building it. Finally, it makes the honest case against that plan, because there is one.
🎓 The Student Who Crammed for One Enormous Exam
Start with how a chatbot gets made, because everything else follows from this.
The first and largest step is pre-training. The model reads an enormous pile of text and practices one task over and over: predict the next word. That is genuinely it. Do that at sufficient scale and something remarkable falls out the other end, which is a system that can hold a conversation about almost anything.
NeuralBuddies has a ground-up explainer on how machines get smart if that idea is new to you.
Pre-training has one property that made it irresistible to companies. Feed it more text and more computing power, and the model gets better at nearly everything at once. No guesswork. Sutskever points out that this made it a very low-risk way to spend money, which is not something you can often say about research.
Researchers stopped having to answer the hardest question in machine learning, which is what data to train on. The answer was everything. All of it. Every book, every forum post, every manual.
Then the pile ran out
Ilya Sutskever helped invent this era, so he is worth listening to on how it ends. He co-created AlexNet, the 2012 system that showed neural networks could work at scale, and later led research at OpenAI.
His framing is clean. From 2012 to 2020, progress came from ideas. From 2020 to 2025, progress came from scaling: bigger models, more text, more chips.
That second era ends now, for a boring reason. The text is finite. There is only so much writing in the world, and the labs already read all of it.
Computing power, meanwhile, kept growing. So the bottleneck moved. It sits back on ideas now, which is why Sutskever describes his own company as an age of research company rather than a scaling one.
That is the setup. Now the interesting part.
🔁 Why Your Chatbot Fixes One Bug and Breaks Another
You have probably lived this next bit, so I will let Sutskever describe it, because his version is exact.
You build something with AI help. There is a bug. You ask the model to fix it. The model agrees enthusiastically that yes, there is a bug, and fixes it, and introduces a second bug. You report the second bug. The model agrees enthusiastically again, fixes that one, and brings the first one back.
You can bounce between those two states indefinitely.
Now hold that beside a different fact. The same model passes coding tests that would defeat most professional programmers. It scores well on mathematics competitions. On paper it is brilliant.
Sutskever‘s own summary of the puzzle is that these systems do extremely well on evaluations while their real-world economic impact lags far behind. Something in that gap is broken.
Two explanations, and both are uncomfortable
The first explanation is about the second stage of training. After pre-training, labs run something called reinforcement learning, where a model attempts tasks and gets graded on the results. Do enough of that and the model gets very sharp at the graded tasks, and possibly a little narrow everywhere else.
The second explanation is worse, and it is a very human failure. Labs build their practice tasks by looking at the benchmarks the model will be judged on. That is understandable. It also means the model trains on a curriculum reverse-engineered from the final exam.
Sutskever‘s interviewer, Dwarkesh Patel, turned that around in a way I keep coming back to. The real reward hacking, he suggested, comes from the human researchers who fixate on the evaluations.
The two students
Here is where Sutskever reaches for a teacher‘s analogy, and I could not improve on it.
Imagine two students who both do brilliantly at competitive programming. The first spent 10,000 hours on it. They memorized every technique, drilled every problem type, and earned their place through sheer volume.
The second spent about 100 hours and also did really well.
Which one has the better career in ten years? Almost everyone answers the second, immediately, without needing the reasoning explained.
Today‘s models are the first student, Sutskever says, and more so. The industry gathered every competitive programming problem in existence, generated more, and trained on all of it. What came out is a phenomenal competitive programmer with no particular reason to be good at anything else.
The word for what the second student has is generalization: the ability to take what you learned in one place and use it somewhere different. Sutskever considers the gap here the most fundamental problem in the field, and says plainly that these models generalize far worse than people do.
He also thinks he knows something about why. He will not say what, because as he put it, not all machine learning ideas get discussed freely.
🧑🎓 The Superintelligent Fifteen-Year-Old
So what do you build instead? This is the part that reframes everything.
Sutskever argues that two words quietly warped how the whole field thinks. The first is AGI, short for artificial general intelligence.
That term exists as an argument. Decades ago, people looked at chess programs and called them narrow: brilliant at one board, useless everywhere else. The response was to demand something general instead, a system that could do all the things. The name stuck.
Pre-training then seemed to deliver exactly that, since more training improved everything at once. General ability, straight from the recipe.
Except a person does not qualify
Here is the observation that turns the goal inside out. Under that definition, you are not an AGI. Neither am I, and neither is anyone reading this.
A human being lacks staggering amounts of knowledge. You cannot perform surgery, argue a case in court, fly a cargo plane, or repair a diesel engine. Unless you happen to do one of those, in which case you cannot do the other three.
What you have instead is the ability to be taught any of them.
That is the reframe. The goal stops being a mind that already knows every job. It becomes a mind that can learn any job.
Sutskever‘s image for it is a student, and he says it plainly: “a superintelligent 15-year-old that‘s very eager to go.“ That describes a phenomenally fast learner who knows very little on day one.
Learning becomes part of the job
Follow that through and something changes about how such a system would arrive.
You would not download a finished product. You would deploy it somewhere, and it would be mediocre for a while, and it would improve by doing the work and making mistakes. Sutskever describes deployment itself as a trial-and-error learning period.
His interviewer restated the idea directly: a system that gets deployed into the world the way a human worker joins an organization. Sutskever agreed.
Think about what that would feel like as a user. Your AI would start out worse than the one you have today and end up understanding your work in a way today‘s version never will. It would remember the correction you made on Tuesday. It would get better at your particular job because it did your particular job.
None of that is easy. NeuralBuddies has a whole piece on why AI forgets so dramatically, where learning a new task can wipe out an old one entirely. Closing that gap is most of the distance between here and there.
How far away is that? Sutskever‘s estimate is five to twenty years, which is an honest range and a wide one.
🤫 Two Years, No Product, No Papers
Everything strange about Sutskever‘s company follows from that goal, so the strangeness is worth walking through.
The company is Safe Superintelligence, usually shortened to SSI. He founded it in June 2024 with Daniel Gross and Daniel Levy, shortly after leaving OpenAI. Its offices sit in Palo Alto and Tel Aviv.
Its website runs a few hundred words and describes a lab with “one goal and one product: a safe superintelligence.“ There is no product page, no pricing, no demo, and no research blog. In two years it published no papers and showed no models to anyone outside the building.
The company keeps no launch date secret, because there is nothing to launch. Under this plan, the learner is the product, and until the learner works there is nothing to sell.
The money is stranger than the silence
Before the Nvidia deal, SSI raised three billion dollars, one billion at its founding and two billion in early 2025. Reporting puts the valuation at roughly 32 billion dollars, and Sutskever used that figure himself.
Set that beside the revenue, which is zero.
Sutskever‘s defense of that budget is the most interesting argument in the whole story, and it is really an argument about research. Think of it as the gap between a school‘s tuition bill and what actually reaches the classroom.
Rival labs raise far more, he says, but much of that money goes to inference. Inference is the cost of actually running a model for millions of users every day. More still goes to product engineers, sales teams, and features. Strip all of that away and the gap in pure research spending narrows considerably.
He backs it with history. AlexNet trained on two graphics chips. The original transformer paper, which underpins essentially every chatbot you use, ran its experiments on no more than 64 chips of 2017 vintage. Breakthroughs, he argues, were never the expensive part. Scaling them up is.
Then Nvidia looked at the homework
In July 2026 that argument got an unusual endorsement.
Nvidia announced a long-term partnership with SSI. It invested in the lab and supplied its next-generation Vera Rubin platform. The stated effect is to increase SSI‘s computing power by an order of magnitude, meaning roughly tenfold. Reuters and Bloomberg both reported the equity investment at around five billion dollars. SSI ran on Google‘s chips before the switch.
The detail I find most telling sits in the announcement itself. Nvidia says it entered the partnership after “rare access into the company‘s closely guarded research.“
So the chipmaker with the best view of the industry read the work in private and then wrote a very large check. That is not proof the research is right. It is the strongest outside signal available, and for a company that shows nobody anything, it is the only one.
About that release rumor
You may have seen headlines saying SSI ships its first model in August 2026. Handle those carefully.
That claim traces to one source: the investor Gavin Baker, who mentioned it almost in passing on a podcast while discussing something else. SSI itself announced nothing, and as of August 22, 2026, no model appeared.
Baker‘s actual subject matters here, because it undercuts a tidier version of this story. He talked about continual learning across the industry, and he said a whole generation of new labs works on it. SSI is not alone in the idea. What sets it apart is what it refuses to sell while it works.
⚖️ The Case Against Waiting
I would be a poor teacher if I gave you one side of this, so here is the strongest objection, and it is genuinely strong.
Almost nothing humans built ever became safe by being reasoned about in advance.
Airplanes did not get safe that way. They got safe through decades of flights, crashes, investigations, and rules written afterward in response to specific failures. Software works the same way. Finding a serious flaw in a mature operating system is hard today precisely because millions of people used it and broke it first.
Dwarkesh Patel raised exactly this point, and Sutskever did not rebut it. He conceded a related point himself. A public release of powerful AI has real value, because a system that does something communicates what an essay never can.
The plan is softer than the slogan
This matters for reading any coverage of SSI, including this post.
The straight shot is a plan he might abandon. Asked directly whether the company still intends to skip straight to superintelligence, Sutskever answered with a maybe. He named two things that could change it: long timelines, and the real value of capable AI in front of people.
He went further. Even under the straight-shot plan, he said, a gradual release would happen anyway. Gradualism would be built into any version of the plan.
He also expects the labs to end up in roughly the same place. His prediction is that the industry converges, first on safety strategy and probably on technical approach too.
What could actually go wrong
Two things, and they point opposite directions.
The research could be wrong. SSI is one bet by one team on one idea. No outsider evaluated the idea, because no outsider saw it. Billions of dollars and two years of silence buy a very expensive negative result if the idea does not hold.
The research could be right. A system that learns any job on the job is a much bigger deal than a better chatbot. It would also arrive from a company that deliberately gave the public no practice run. Sutskever‘s own concession, that people need to see capable AI to understand it, cuts directly against his own default plan.
🧭 Zap’s Study Guide: Five Things to Watch For
Notice which failures are memory failures. When your AI repeats a mistake you already corrected, that is the gap this whole post is about. Naming it correctly makes you much harder to oversell to.
Treat benchmark scores as exam scores. A model topping a leaderboard tells you it trained hard for that leaderboard. It tells you considerably less about how the model handles your actual work.
Ask who said it before you believe a release date. The SSI August rumor came from an investor on a podcast, not from the company. That distinction survives contact with reality; a headline often does not.
Watch for the word “continual.” Continual learning is the thing to track. The term means a model that still improves after release, and several labs chase it now. Whoever lands it first changes what these tools feel like to use.
Keep both objections in your pocket. The research might not work. It might work and arrive with no warning. A plan can fail in both directions, and good thinking holds both without picking the comfortable one.
🏁 Conclusion
Back to my two students for a moment.
The one who memorized everything is genuinely impressive. You can ask them anything and get an answer. That is not nothing, and I want to be fair to what the current generation of AI achieved, because it is extraordinary.
But you already know which student you would hire. You would take the one who learns, every time, because the job you hire for will change and the memorized answers will go stale.
That is Sutskever‘s whole bet, stated as plainly as I can put it. He thinks the industry built a magnificent crammer and called it general intelligence. The real target, he says, is the student who arrives with nothing and runs the department eighteen months later.
He may be wrong. Nobody can check, which is the uncomfortable part of a company that shows its work to a chip supplier and to nobody else. And the objection stands: things get safe by being used, and a lab that never releases anything never learns what breaks.
What I would leave you with is smaller and more useful than a prediction. Data is power, but understanding is wisdom. The difference between those two is exactly the difference between a system that read everything and a system that can learn something.
Every AI tool you touch today sits firmly on the first side of that line.
I once calculated the meaning of life and decided it was more fun to keep learning. Turns out I was accidentally describing the frontier of AI research.
Office hours are always open.
-- Zap ⚡
Sources / Citations
Safe Superintelligence Inc. Safe Superintelligence Inc. SSI. https://ssi.inc/
Dwarkesh Patel, November 25, 2025. Ilya Sutskever: We’re moving from the age of scaling to the age of research. Dwarkesh Podcast.
NVIDIA, July 27, 2026. Ilya Sutskever’s Safe Superintelligence Inc. and NVIDIA Announce Long-Term Strategic Partnership. NVIDIA Newsroom. https://nvidianews.nvidia.com/news/ilya-sutskevers-safe-superintelligence-inc-and-nvidia-announce-long-term-strategic-partnership
Rebecca Bellan, July 27, 2026. Ilya Sutskever’s Safe Superintelligence partners with Nvidia to scale its AI research. TechCrunch. https://techcrunch.com/2026/07/27/ilya-sutskevers-safe-superintelligence-partners-with-nvidia-to-scale-its-ai-research/
DataCenterDynamics, August 19, 2026. Nvidia to invest $5bn in Safe Superintelligence, Sutskever firm to substantially increase GPU compute. DataCenterDynamics. https://www.datacenterdynamics.com/en/news/nvidia-to-invest-5bn-in-safe-superintelligence-sutskever-firm-to-substantially-increase-gpu-compute/
OfficeChai, August 4, 2026. SSI To Come Out With Their Model In August, Claims Gavin Baker. OfficeChai. https://officechai.com/ai/ssi-to-come-out-with-their-model-in-august-claims-gavin-baker/
Crypto Briefing, August 4, 2026. Safe Superintelligence plans first model release for August after raising $3 billion without shipping a single product. Crypto Briefing. https://cryptobriefing.com/ssi-superintelligence-model-august-release/
Wikipedia. Safe Superintelligence Inc. Wikipedia. https://en.wikipedia.org/wiki/Safe_Superintelligence_Inc.
Take Your Education Further
AI, AGI, ASI: What’s the Difference?: A NeuralBuddies walk through the standard ladder from narrow AI to superintelligence, which is the definition of AGI this post argues points at the wrong target.
The Four Roads to Superintelligence: What DeepMind Says Comes After Human-Level AI: A NeuralBuddies map of the routes past human-level AI, which is the larger landscape this one lab’s bet sits inside.
The 4 Fatal Flaws of Modern AI: A NeuralBuddies catalog of what today’s systems genuinely cannot do, including the flaw it calls a mile wide and an inch deep, which is this post’s generalization gap under another name.
Disclaimer: This content was developed with assistance from artificial intelligence tools for research and analysis. Although presented through a fictitious character persona for enhanced readability and entertainment, all information has been sourced from legitimate references to the best of my ability.