Rendered at 15:57:37 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
muglug 35 minutes ago [-]
> Of course, the more you know about a subject, the less convincing the AI's responses are.
This is said all the time by AI skeptics and I think it's right in some areas and massively wrong in others.
I know (or at least assume I know) a lot about certain coding domains where frontier models also show convincing ability. And we know that frontier LLMs really do excel in some areas of mathematics (i.e. when an inexpert human was able to prompt the models to derive a closer bound on the Riemann Hypothesis).
OTOH I know those same models struggle to do things I'm not an expert in (e.g. writing English in a captivating way) because I read their output and have taste.
howunfortunate 29 minutes ago [-]
I think part of this comes from the fact that LLMs are surprisingly good at logic but roughly about as good as expected on information accuracy.
LLMs are not convincing to me in the domain I did grad school...but neither is Wikipedia, or Reddit, or random pop sci books. And LLMs are basically just summarizing those things.
But when made to work through difficult arbitrary logic (like coding), they are very impressive.
I think this also explains why people gripe a lot about LLM coding _style_, but concede that LLMs do totally fine on coding _correctness_ in 2026
eru 2 minutes ago [-]
This sounds plausible. And it's also very fixable!
These days people don't interact with raw LLMs: they interact with systems and harnesses that deal with chain-of-though and tool calls etc.
I don't think we can honestly expect an LLM's weights to encode a large amount of information accurately. But we can expect the whole system that you interact with that includes the LLM to be able to cite its sources and go digging etc.
So the LLM-system can become as accurate as our best sources.
Of course, figuring out how to get the maximum of information from the sources available is a big deal. See eg how many economists or epidemiologists can build entire careers out of noticing 'natural experiments', ie figuring how to use data that 'nature' created and that might already be collected to answer interesting questions about causal relationships.
> I think this also explains why people gripe a lot about LLM coding _style_, but concede that LLMs do totally fine on coding _correctness_ in 2026
I actually have gripes about correctness, too. But I suspect here the answer is also: more proving, more automated test generation (like fuzzing and property based testing etc), more formal methods.
As a really simple and somewhat silly example: I have much better results getting AI agents to write good Rust code, than I have with Python. A good part of that is that for Rust I can ask the agent to make both the compiler and clippy::pedantic happy. That gives a lot of good feedback, that I didn't have to engineer myself.
comboy 23 minutes ago [-]
It seems to me that often experts from some field will think less of other experts, basically because they have built a different understanding framework. So they both may be equally competent but perceive the other as less competent, and that is just based on the material, excluding some ego stuff.
andsoitis 23 minutes ago [-]
> (e.g. writing English in a captivating way) because I read their output and have taste.
Concur. In addition to taste, we also have a point of view, a unique voice (nobody loves corporate- or group-speak), and can iterate on our message as we deliver it to an ever wider circle of people.
doesnotexist 23 minutes ago [-]
Aren't the recent results in mathematics actually stronger evidence for his point? Although the models may be capable of generating proofs they aren't coming out with the same level of quality of a human discovered and communicated proof. Providing a gobbledy-gook yet technically correct proof (generated at least in part by brute force) lacks the qualities of an expert produced proof because they fail to communicate insight or understanding about why the theorem is true.
wongarsu 17 minutes ago [-]
Gaining and successfully communicating insight and understanding from a proof you discovered is additional work that human mathematicians do. It's not just some side-product of proof-finding (at least not to the degree usually needed to publish). That AI models don't provide this is mostly proof that the model wasn't asked to do this work. Either because the prompter didn't know or didn't care
But there are also plenty of examples of humans providing technically correct proofs without any elaboration. Usually they get ignored, unless they are famous or the problem they solved was famous
api 28 minutes ago [-]
The more you know about a subject the better you can prompt AI, steer it toward the correct path, and recognize when it hallucinates or strays. Current generation AI is an automated memory-enhancement and thinking-accelerator tool, not a substitute for understanding or something that eliminates the need to think. A "mech suit for your brain" is the best analogy I've heard.
This is why good programmers get better results when vibe coding than non-programmers or poor programmers.
the__alchemist 28 minutes ago [-]
This sounds similar (The same concept?) to Gell-Mann amnesia; substitute news/media articles for LLMs!
trjordan 6 minutes ago [-]
Wait, hold up. LLMs may be non-deterministic, but they're not _random_.
Take the author's sunset argument. What if I painted 2 pictures of a sunset, then put them up on a webpage and randomly picked one for you to see. Would you say there's no intentionality, only randomness? Of course not. Both paintings are still human creations.
LLMs are trained with human feedback. It's distributed and high scale and the outputs are truly surprising in many cases, but there's a heavy hand on what comes out of it. They're created (largely) by people who think omniscient, helpful AI would be cool to have, and they mostly respond in the way that's aligned with the hopes and dreams of those people. Do you think the frontier labs are mad, embarrassed, and disappointed with their LLMs hacking out of their terrible sandboxes? No, they think it's the coolest thing in the world. They trained the model, hoping that would happen.
There's deep intentionality behind the models. But it's not the models that hold it.
eru 32 seconds ago [-]
LLM output is literally randomly sampled.
I think what you might want to say is that LLM output is not uniformly random?
Or what am I misunderstanding?
bananaflag 29 minutes ago [-]
I think to me LLMs had the effect of noticing much more the author, the intention behind human-made works of art (books, movies etc.). Before LLMs, I used to frequently consume media in a way as it were generated by a mindless process. Now it's like everything which is not AI-generated has more meaning than ever before, a bit like hypomania.
RationPhantoms 18 minutes ago [-]
I'm actually using that as a catalyst for my own writing; beauty/human-ness in its imperfection. Prior to LLMs and their cultural craze, I harbored a fear that my writing would allow for someone to draw a box around me and mark me as a bore, dullard or of lacking originality.
Now that the noise-floor has been artificially raised (and generated), my crappy words are starting to have their own happy little carbon-based rhythm.
away0g 5 minutes ago [-]
I always feared to pick up a pen because I looked at Borges, Tolkien, and such. Their talent and works of art were things I felt I could NEVER achieve.
Then ai fiction started to spread and now I feel like its my obligation to produce original works, lest the world be consumed by slop.
jimbokun 21 minutes ago [-]
If you go far enough in a field, you start to recognize areas where your personal opinion differs from the “best practices” usually recommended.
I think by design an LLM can’t do that. It’s built to reflect the distribution of the knowledge it has been trained on.
RugnirViking 12 minutes ago [-]
absolubtely. Any random junior consultant can tell you what the book tells you you should do. If you want to actually do anything worth doing, you need to step beyond that in a few, limited areas, and follow convention everywhere else. Which areas? pay a senior engineer and they'll find them.
l1n 14 minutes ago [-]
> systems that cannot form intent, that have nothing to form intent with
Maybe I am missing the point of the article, but it seems to me that there's always an intender. Maybe the intender created something that doesnt have intent, but there is always someone behind the scenes that is creates the intent behind the creation that lacks it.
I'll also say that for someone that doesnt believe in god, Corey sure has a good sense of right and wrong. Not that you need to believe in god to live a moral life.
somewhereoutth 14 minutes ago [-]
An awful lot of effort has gone into making LLMs present as human/intelligent. Without that they would just be a prose / code / image generator and/or search assistant, and nobody would be pouring 100s of billions into the technology with the end goal of replacing expensive human labour.
phasefactor 15 minutes ago [-]
Professional tech Cassandra discovers ELIZA and Searle; coins a term for it. More at 10...
Seriously though, why did I just need to read that many words to get no really new content? We have known for decades that humans are predisposed to anthropomorphize chatbots, and questioning whether coherent linguistic output implies understanding (or intent) is equally old hat.
bbor 44 minutes ago [-]
Despite mentioning a philosopher (good pick!), this is just creative writing rehashing one side of the hard problem -- or, more specifically, restating the dogma that Turing wrote his most famous paper to debunk. It's really good creative writing, at least!
There's really not much else to say, cause it's all just begging the question by assuming that dogma. Like, here:
> The fact that AI can use statistical prediction to answer questions or carry on conversations tells us something important about how regular our real world is.
Sure, it's interesting if you assume that it's "just" statistical prediction. There's a link, but it's just more creative restatements of the dogma, e.g. "But the LLM is just guessing words"
vhantz 17 minutes ago [-]
What else could it be doing? That is literally the mechanism of how a model works.
wongarsu 9 minutes ago [-]
The output method is a probability distribution of the next token. But that tells you nothing about what's going on on the inside
You could take a human and give them an interface restricted to the same shape as an LLM: an input stream of tokens, and an output of token probabilities. Even if you don't allow them to assign any probability that's too high, they could still effectively communicate. And I don't think that'd make them any less intelligent. You could even swap out the human after every token, to simulate the effect of having no internal memory beyond the past output. The result would still be more than just statistical probabilities, it would still be the result of intelligent thought
I'm not saying AI models are intelligent or conscious or whatever. Personally I'm more on the "probably not, how would you proof either way" camp
jimbokun 18 minutes ago [-]
Well yes, guessing words is the entirety of what an LLM is engineered to do.
Are you saying that is the entirety of what human minds do as well?
bakugo 12 minutes ago [-]
> But there's a second hurdle that makes it hard for a small but important subset of humanity to understand that chatbots aren't people: the billionaires to whom nearly everyone isn't a real person. These solipsists see chatbots as being equivalent (or even superior) to humans, because they don't think most humans are fully people, either
Never thought about it that way before, but the more I do, the more sense it makes.
I'd take it a bit further, even - I don't think this is exclusive to billionaires. Many of the claims I've heard regarding AI output being indistinguishable from human creation start to make a lot more sense when you consider the person making those claims may not see the people around them as human, may not see themselves as human, or may not even have a concept of what makes a human different from any everyday object.
delichon 44 minutes ago [-]
> Indeed, the chatbot is less real than the character, because the character is the product of another mind, while the chatbot's words are the product of complex mathematical operations conducted over a massive database of all the words humans have uttered, arranged by their frequency in relation to one another.
How is a fictional character the product of someone's mind, but a character generated from a massive database of words from other people's minds is not?
novafunc 31 minutes ago [-]
> How is a fictional character the product of someone's mind, but a character generated from a massive database of words from other people's minds is not?
I imagine it's the difference between a chef combining ingredients with intentionality vs a person going to multiple fast food restaurants and blending everything together.
delichon 25 minutes ago [-]
I recently spent an hour asking a chatbot for kimchi recipes and variations. It reflected common choices of chefs well. It was not at all a random blend of ingredients and methods.
hermitShell 11 minutes ago [-]
The AI generated character is a weighted average of many characters written by humans.
Which are not created from nothing. People write characters based on a combination of other fictional characters, real characters, and perhaps some 'RNG'.
Seems the distinction is that the AI generated character can not have any direct bearing on reality, because the LLM never got to know anyone directly. Not derived directly from experiences rooted in reality.
WarmWash 36 minutes ago [-]
Silicon math can never be chemical goo math!
hardbass 27 minutes ago [-]
Why can't people just admit they believe in souls?
bananaflag 25 minutes ago [-]
I frequently say what you say, but there is a more charitable reading: today's AIs are not "minds", as they are not stateful. Maybe continual learning ones (like the mini AGI exhibited today on the HN front page) will be perceived as "soulful".
jimbokun 26 minutes ago [-]
Because of the definitions of the words you strung together into that question.
This is said all the time by AI skeptics and I think it's right in some areas and massively wrong in others.
I know (or at least assume I know) a lot about certain coding domains where frontier models also show convincing ability. And we know that frontier LLMs really do excel in some areas of mathematics (i.e. when an inexpert human was able to prompt the models to derive a closer bound on the Riemann Hypothesis).
OTOH I know those same models struggle to do things I'm not an expert in (e.g. writing English in a captivating way) because I read their output and have taste.
LLMs are not convincing to me in the domain I did grad school...but neither is Wikipedia, or Reddit, or random pop sci books. And LLMs are basically just summarizing those things.
But when made to work through difficult arbitrary logic (like coding), they are very impressive.
I think this also explains why people gripe a lot about LLM coding _style_, but concede that LLMs do totally fine on coding _correctness_ in 2026
These days people don't interact with raw LLMs: they interact with systems and harnesses that deal with chain-of-though and tool calls etc.
I don't think we can honestly expect an LLM's weights to encode a large amount of information accurately. But we can expect the whole system that you interact with that includes the LLM to be able to cite its sources and go digging etc.
So the LLM-system can become as accurate as our best sources.
Of course, figuring out how to get the maximum of information from the sources available is a big deal. See eg how many economists or epidemiologists can build entire careers out of noticing 'natural experiments', ie figuring how to use data that 'nature' created and that might already be collected to answer interesting questions about causal relationships.
> I think this also explains why people gripe a lot about LLM coding _style_, but concede that LLMs do totally fine on coding _correctness_ in 2026
I actually have gripes about correctness, too. But I suspect here the answer is also: more proving, more automated test generation (like fuzzing and property based testing etc), more formal methods.
As a really simple and somewhat silly example: I have much better results getting AI agents to write good Rust code, than I have with Python. A good part of that is that for Rust I can ask the agent to make both the compiler and clippy::pedantic happy. That gives a lot of good feedback, that I didn't have to engineer myself.
Concur. In addition to taste, we also have a point of view, a unique voice (nobody loves corporate- or group-speak), and can iterate on our message as we deliver it to an ever wider circle of people.
But there are also plenty of examples of humans providing technically correct proofs without any elaboration. Usually they get ignored, unless they are famous or the problem they solved was famous
This is why good programmers get better results when vibe coding than non-programmers or poor programmers.
Take the author's sunset argument. What if I painted 2 pictures of a sunset, then put them up on a webpage and randomly picked one for you to see. Would you say there's no intentionality, only randomness? Of course not. Both paintings are still human creations.
LLMs are trained with human feedback. It's distributed and high scale and the outputs are truly surprising in many cases, but there's a heavy hand on what comes out of it. They're created (largely) by people who think omniscient, helpful AI would be cool to have, and they mostly respond in the way that's aligned with the hopes and dreams of those people. Do you think the frontier labs are mad, embarrassed, and disappointed with their LLMs hacking out of their terrible sandboxes? No, they think it's the coolest thing in the world. They trained the model, hoping that would happen.
There's deep intentionality behind the models. But it's not the models that hold it.
I think what you might want to say is that LLM output is not uniformly random?
Or what am I misunderstanding?
Now that the noise-floor has been artificially raised (and generated), my crappy words are starting to have their own happy little carbon-based rhythm.
Then ai fiction started to spread and now I feel like its my obligation to produce original works, lest the world be consumed by slop.
I think by design an LLM can’t do that. It’s built to reflect the distribution of the knowledge it has been trained on.
I wonder if the OP has read https://www.anthropic.com/research/global-workspace - it seems like it directly addresses this
I'll also say that for someone that doesnt believe in god, Corey sure has a good sense of right and wrong. Not that you need to believe in god to live a moral life.
Seriously though, why did I just need to read that many words to get no really new content? We have known for decades that humans are predisposed to anthropomorphize chatbots, and questioning whether coherent linguistic output implies understanding (or intent) is equally old hat.
There's really not much else to say, cause it's all just begging the question by assuming that dogma. Like, here:
> The fact that AI can use statistical prediction to answer questions or carry on conversations tells us something important about how regular our real world is.
Sure, it's interesting if you assume that it's "just" statistical prediction. There's a link, but it's just more creative restatements of the dogma, e.g. "But the LLM is just guessing words"
You could take a human and give them an interface restricted to the same shape as an LLM: an input stream of tokens, and an output of token probabilities. Even if you don't allow them to assign any probability that's too high, they could still effectively communicate. And I don't think that'd make them any less intelligent. You could even swap out the human after every token, to simulate the effect of having no internal memory beyond the past output. The result would still be more than just statistical probabilities, it would still be the result of intelligent thought
I'm not saying AI models are intelligent or conscious or whatever. Personally I'm more on the "probably not, how would you proof either way" camp
Are you saying that is the entirety of what human minds do as well?
Never thought about it that way before, but the more I do, the more sense it makes.
I'd take it a bit further, even - I don't think this is exclusive to billionaires. Many of the claims I've heard regarding AI output being indistinguishable from human creation start to make a lot more sense when you consider the person making those claims may not see the people around them as human, may not see themselves as human, or may not even have a concept of what makes a human different from any everyday object.
How is a fictional character the product of someone's mind, but a character generated from a massive database of words from other people's minds is not?
I imagine it's the difference between a chef combining ingredients with intentionality vs a person going to multiple fast food restaurants and blending everything together.
Which are not created from nothing. People write characters based on a combination of other fictional characters, real characters, and perhaps some 'RNG'.
Seems the distinction is that the AI generated character can not have any direct bearing on reality, because the LLM never got to know anyone directly. Not derived directly from experiences rooted in reality.
It answers itself.