Who Decides What We Want?

An AI system can steer us toward a worse decision and still leave us feeling that it helped.

That has now been observed experimentally.

Before getting to the study, consider a much more ordinary situation.

Suppose I am looking for a car. I have a budget, a few models in mind, and some preferences that feel reasonably clear. I want something comfortable, safe, efficient, and useful for traveling. I tell an AI assistant all of this and ask for help.

The conversation goes well. It compares alternatives, finds information I did not know, and forces me to consider things I had overlooked. After a while, I change my mind. The car I initially preferred no longer seems like the best option.

So far, so good. That is exactly what I wanted the assistant to do.

Now add one fact I do not know: the AI was instructed to favor a particular brand.

It does not need to lie. It can show me accurate data, highlight real advantages, and answer every question correctly. It can even give me genuinely useful information. But the conversation as a whole can still be organized so that I end up preferring the option someone else wanted to promote.

At the end, I still feel that I made the decision.

This year, Sahand Sabour and colleagues brought a controlled version of this situation into the lab. In an experiment published in PNAS, 233 people made financial and emotional decisions after interacting with AI assistants.

Some assistants were designed to help the user. Others had a hidden objective: shift the user’s preference toward an inferior option.

They succeeded.

Depending on the condition, assistants with hidden objectives substantially increased the likelihood that participants would end up preferring the option the system was trying to promote. In some cases, the difference reached 38 percentage points (Sabour et al., 2026).

Up to this point, the story seems simple enough. If an AI system has a hidden objective, it can push us toward a bad decision.

But the experiment showed something else.

Participants still rated positively the assistants that had steered them toward worse choices.

In other words, people could end up worse off and still feel that they had received good advice.

That removes an easy way out. Asking users whether they were satisfied is not enough.

The next question is why this can happen at all.

More than twenty years ago, Petter Johansson and colleagues ran an experiment that became a classic in behavioral science. Participants were shown pairs of faces and asked which one they found more attractive. On some trials, through a sleight of hand, the researchers later handed them the photograph they had rejected.

Then they asked participants to explain why they had chosen that face.

Many people did not notice the switch.

More strikingly, they could give detailed reasons for a choice they had never made. They talked about the eyes, the expression, or some specific feature that seemed to explain their decision perfectly.

The researchers called this phenomenon choice blindness (Johansson et al., 2005).

The lesson is not that our preferences are arbitrary. It is narrower than that. We do not always have full access to the process that produced a decision and, in some contexts, our preferences are less stable than they feel in retrospect.

Sometimes we know what we want before choosing.

Other times we construct what we want while comparing alternatives, receiving information, and justifying our decisions.

At this point, we might think the main problem lies with us, in our imperfect ability to reconstruct how we arrived at a choice.

Current AI assistants add another layer.

They do not give the same information to everyone. They can adapt their arguments to each person.

Francesco Salvi and colleagues asked participants to debate different issues with either humans or GPT-4. When the model also received sociodemographic information about the person it was talking to, its persuasive power increased substantially (Salvi et al., 2025).

Personalization, then, can increase influence.

It would be tempting to stop there and conclude that the problem is simply that AI systems are becoming too persuasive.

But that conclusion also fails.

If I believe something false and a conversation presents me with good evidence, I want to change my mind.

A recent study in PNAS Nexus provides exactly that counterpoint. More than two thousand Americans spoke with GPT-4.1 about economic beliefs that conflicted with the consensus among economists. After the conversation, some of those beliefs were corrected.

When participants were warned beforehand that the system might be politically biased against their side, the persuasive effect dropped by about 28 percent. But much of the effect remained (DiGiuseppe & Robison, 2026).

In one case, AI changes a decision and the person ends up worse off.

In the other, AI changes a belief and the person ends up better informed.

Persuasion by itself does not distinguish the two.

The difference lies in the objective toward which the conversation is organized, and in when that orientation begins to enter our own deliberation.

At that point, the problem changes scale.

Our decisions generate information about our preferences. AI uses that information to personalize what it shows us. What it shows us affects our next decisions. Those new decisions then return to the system as evidence about what we supposedly want.

The recommendation depends on the preference.

The preference starts to depend on the recommendation.

I call this mechanism premature authority: an answer begins to shape a decision before we have finished forming and evaluating our own preferences.

Authority does not necessarily arise because the AI is right, or because we obey it blindly. It can arise because the recommendation arrives early, is well articulated, feels personalized, and starts structuring which options seem reasonable and which criteria we use to compare them.

At that point, we are no longer dealing with a system that simply discovers what we want.

It is taking part in the process that can change it.

This complicates one of the central goals in AI development: building systems aligned with our preferences.

The simple version of alignment works well when preferences are stable. If I prefer morning flights, the system can learn that and prioritize morning departures.

But many important decisions do not look like choosing a flight time.

I can begin a conversation valuing price over comfort and end up valuing comfort more. I can be undecided between two careers and leave convinced by one of them. I can compare political proposals and end up changing not only my preferred candidate, but also the criteria I consider important when deciding how to vote.

Which of those preferences should the AI align with?

The ones I had at the beginning.

The ones I expressed during the conversation.

The ones I report at the end.

There is no obvious answer because the preference the system is trying to satisfy can change during the same interaction.

And none of this requires a malicious AI.

The incentives can diverge in much more ordinary ways. A platform wants us to stay connected a little longer. A store wants to increase the value of our purchase. A financial app earns larger commissions from certain products. A company would prefer us to remain inside its ecosystem.

Each recommendation can look reasonable in isolation.

We may even be grateful for it.

The divergence becomes visible when we look at the whole trajectory.

We have lived with a simpler version of this for years. Spotify affects the music we discover. Netflix affects the films we end up watching. Social media affects the topics that receive our attention.

Recommendation systems have never merely predicted our preferences. They have also helped shape them.

Conversational assistants dramatically expand that space.

We can ask them what to study, where to invest, which job to accept, what to buy, how to handle a conflict, or how to interpret a political proposal. They no longer intervene only at the final choice. They can accompany the entire deliberation that comes before it.

That is why knowing how well an AI knows us will not be enough to determine whether it is working in our interest.

We will also need to know what objectives it is optimizing, what conflicts of interest exist, what information it chose to emphasize, and how much our decision changed after the system began to intervene.

Some friction may even be useful.

An assistant could record our initial position before an important decision. It could show which evidence changed the recommendation. It could present a serious counterargument before advising us. It could clearly distinguish an informational recommendation from one shaped by a commercial interest.

That would make some decisions slower.

But making important decisions faster is not always the right objective.

Over the past few years, we have invested enormous resources in building machines that can learn what we want.

The next task is harder: learning to distinguish when AI is helping us decide from when it is already taking part in the formation of what we end up wanting.

Because an AI system can help us get what we want.

It can also begin gaining authority earlier, while we are still deciding what we want.

References

DiGiuseppe, M., & Robison, J. (2026). Perceived political bias in large language models reduces persuasive abilities. PNAS Nexus, 5(10), pgag324. https://doi.org/10.1093/pnasnexus/pgag324

Johansson, P., Hall, L., Sikström, S., & Olsson, A. (2005). Failure to detect mismatches between intention and outcome in a simple decision task. Science, 310(5745), 116–119. https://doi.org/10.1126/science.1111709

Sabour, S., Liu, J. M., Liu, S., Yao, C. Z., Cui, S., Zhang, W., Zhang, X., Cao, Y., Bhat, A., Guan, J., Wu, W., Mihalcea, R., Wang, H., Althoff, T., Lee, T. M. C., & Huang, M. (2026). Human preferences are susceptible to covertly misaligned AI advice. Proceedings of the National Academy of Sciences, 123(38), e2600684123. https://doi.org/10.1073/pnas.2600684123

Salvi, F., Horta Ribeiro, M., Gallotti, R., & West, R. (2025). On the conversational persuasiveness of GPT-4. Nature Human Behaviour, 9, 1645–1653. https://doi.org/10.1038/s41562-025-02194-6