---
title: "AI & The Next Layer"
description: "Generative AI is expected to make personalised manipulation dramatically cheaper. What is documented and what is still developing."
url: https://attentionmachine.org/en/attention-machine/ai/
lang: en
updated: 2026-10-25
license: "CC BY-NC-SA 4.0 (https://creativecommons.org/licenses/by-nc-sa/4.0/)"
---

# AI & The Next Layer

The attention machine was already powerful. Generative AI is expected to make its most dangerous applications dramatically cheaper and faster. This page sets out what has happened so far.

This is a rapidly developing area. What follows distinguishes between what is documented (evidence exists), what is plausible (infrastructure exists, early cases documented), and what remains unknown.

## What has changed

The attention machine's manipulation capability was always limited by one thing: the cost and labour of producing personalised content at scale. Creating convincing fake personas, tailored disinformation, or emotionally targeted messaging required significant human effort. Generative AI is removing that constraint[^src-goldstein-2023].

Since 2022[^src-maslej-2023], tools exist that can produce convincing text, realistic images, cloned voices, and synthetic video — and researchers expect them[^src-goldstein-2023] to drive down the cost of influence campaigns and make them easier to scale. When combined with the behavioural targeting infrastructure described on the Advertising page — which can identify specific psychological profiles and serve content to them precisely — the result is a system capable of producing personalised manipulation at a scale and speed that was not previously possible.

> **Synthetic media (deepfakes)**
>
> Images, audio, or video generated or altered by AI to depict people saying or doing things they did not. The technical barrier has collapsed: a convincing voice clone now needs seconds of sample audio. The significance is less any single fake than the "liar's dividend" — once people know anything _could_ be fake, genuine evidence can be dismissed as fabricated too.
>
> **Sources**
>
> - Chesney & Citron (2019), [_Deepfakes and the New Disinformation War_](https://www.foreignaffairs.com/articles/world/2018-12-11/deepfakes-and-new-disinformation-war), Foreign Affairs.
> - Wang, C., Chen, S., Wu, Y. et al. (2023), [_Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers_](https://arxiv.org/abs/2301.02111), Microsoft Research, arXiv preprint — VALL-E, personalised speech "with only a 3-second enrolled recording".
> - US Federal Trade Commission (20 March 2023), [_Scammers use AI to enhance their family emergency schemes_](https://consumer.ftc.gov/consumer-alerts/2023/03/scammers-use-ai-enhance-their-family-emergency-schemes) — "All he needs is a short audio clip of your family member's voice".

## What is documented

**Romania, November 2024.** In an unprecedented step in the European Union, a presidential election was annulled following evidence of coordinated algorithmic amplification on TikTok. Romanian intelligence declassified documents[^src-curtea-constitutionala-2024] stating that the little-known ultranationalist candidate Călin Georgescu, who had been polling behind at least five other candidates[^src-romanias-constitutional-2024], received coordinated amplification and paid promotion on TikTok[^src-agerpres-2024]. He won the first round with 23% of the vote. Romania's Constitutional Court ordered the electoral process to be restarted entirely[^src-romanias-constitutional-2024] — an annulment without precedent in the EU, on grounds of digital interference. What the sources support differs by claim. The coordinated amplification rests on the declassified intelligence note, which the Court accepted and critics dispute; that influencers were paid to post content that ended up promoting him is also documented outside the note, by a news investigation that drew on the tax authority (below); the role of generative AI in the content is a further, thinner claim.

> **Coordinated inauthentic behaviour (CIB)**
>
> Networks of accounts that hide who is really behind them and act together to make a message look more popular or organic than it is — fake personas, paid amplification, bot-assisted sharing. The term comes from Meta's own enforcement vocabulary. What makes it "inauthentic" is the concealed coordination, not the content itself, which may be true, false, or merely divisive.
>
> **Sources**
>
> - [Atlantic Council (New Atlanticist)](https://www.atlanticcouncil.org/blogs/new-atlanticist/romania-annulled-its-presidential-election-results-amid-alleged-russian-interference-what-happens-next/) — analysis of the Romania annulment (2024).
> - Gleicher, N. (6 December 2018), [_Coordinated Inauthentic Behavior Explained_](https://about.fb.com/news/2018/12/inside-feed-coordinated-inauthentic-behavior/), Meta Newsroom.
> - Gleicher, N. (2019), [_Removing More Coordinated Inauthentic Behavior From Iran and Russia_](https://about.fb.com/news/2019/10/removing-more-coordinated-inauthentic-behavior-from-iran-and-russia/), Meta Newsroom.

**The 2024 European Parliament elections** were regarded as a "testing ground"[^src-farooq-2025] for AI-powered disinformation. The same research, published in 2025 (Farooq et al., _The International Journal of Press/Politics_), documented citizens' concerns about and awareness of AI-generated content during the campaign.

**Scale in 2024 US elections.** The Centre for Emerging Technology and Security (CETaS)[^src-cetas-2024] at the Alan Turing Institute found no conclusive evidence that AI-generated content materially altered the 2024 US election results. However, the same research found that AI-generated content did influence election discourse, amplified harmful narratives, and entrenched political polarisation. The absence of a decisive measurable effect in one election cycle is not evidence that the capability is not a structural concern.

## What the infrastructure enables

The concern about AI and the attention machine is not primarily about individual incidents of fake content. It is about what the combination of cheap content generation and behavioural targeting infrastructure makes structurally possible:

Influence operations that previously required significant human labour can now be run far more cheaply[^src-goldstein-2023]. A disinformation campaign targeting specific psychological profiles across multiple countries — served to people when algorithms identify them as maximally susceptible — can be produced and deployed by actors who could not previously afford this kind of precision.

The EU AI Act, the DSA, and various national regulations are all attempting to address this. The DSA is now being enforced — the Commission fined X €120 million under it in December 2025[^src-european-commission-2025], for breaching its transparency rules — and the AI Act's transparency duties for AI-generated content[^src-european-commission-2026] have applied since 2 August 2026. How far either reaches influence operations of this kind is not yet settled. The capability is ahead of the governance, as it has been throughout the history of the attention machine.

## AI companions

A separate category: conversational AI products designed for ongoing personal relationships — AI companions, chatbots with persistent memory, products designed to simulate friendship or romance. The commercial incentive is close to social media's — optimise for engagement and return use — even where the money comes from subscriptions rather than advertising. The evidence on effects — on loneliness, social development, emotional dependency — is thin. This is one of the genuinely open questions documented on the [What We Don't Yet Know](https://attentionmachine.org/en/symptoms/unknown/) page.

**How we know — Romania 2024, and what is documented versus contested**

The confirmed facts: Romania's Constitutional Court[^src-romanias-constitutional-2024] annulled the first round of the November 2024 presidential election; declassified intelligence described[^src-agerpres-2024] a coordinated TikTok amplification and paid-promotion operation around a candidate whom polls had placed behind at least five other candidates; he took 23% of the first-round vote. These are matters of public record and an official court ruling.

**What is contested or unresolved.** The precise role of _generative AI_ in the Romanian operation — as opposed to coordinated human-run accounts and paid promotion — is not fully established. "Algorithmic amplification" (the recommender pushing the content) is what the declassified note asserts and the Court accepted; it has not been shown independently. "AI-generated content" is a separate claim, with thinner evidence still in this specific case. We keep the two apart. The declassified documents were themselves contested: a later investigation by the news site Snoop.ro, drawing on the tax authority,[^src-euronews-2024] found that one influencer campaign which ended up promoting the candidate had been paid for by a rival pro-EU party, and the annulment was criticised by constitutional-law[^src-kuti-2024] scholars[^src-iancu-2025] and by the US administration[^src-rfe-rls-romanian-2025], and defended by others[^src-cornea-2025] as "an expression of militant democracy" (more on the [Elections](https://attentionmachine.org/en/symptoms/elections/) page). Likewise, CETaS's finding[^src-cetas-2024] on the 2024 US elections — no conclusive evidence of a decisive effect on the result — is a statement about one cycle and one kind of measurement, not a verdict that the capability is harmless.

## Sources

[^src-cetas-2024]: [CETaS (Alan Turing Institute), _AI-Enabled Influence Operations: Safeguarding Future Elections_](https://cetas.turing.ac.uk/publications/ai-enabled-influence-operations-safeguarding-future-elections) (November 2024).

[^src-farooq-2025]: Farooq, A., van den Hoogen, E., Tulin, M. & de Vreese, C. (2025), [_Generative AI Generating Concerns: Citizens' Perspectives During the 2024 European Elections_](https://doi.org/10.1177/19401612251376060), The International Journal of Press/Politics — survey of 2,929 citizens in the Netherlands, Germany and Poland.

[^src-romanias-constitutional-2024]: [Romania's Constitutional Court annuls the first round](https://www.pbs.org/newshour/world/top-court-annuls-romanias-first-round-of-presidential-voting) (December 2024) — PBS NewsHour / AP.

[^src-curtea-constitutionala-2024]: Curtea Constituțională a României (6 December 2024), [_Hotărârea nr. 32_](https://www.ccr.ro/wp-content/uploads/2024/12/Hotarare_32_2024.pdf) — the unanimous decision annulling the whole presidential electoral process; European Court of Human Rights (11 February 2025), [_Georgescu v. Romania_](https://hudoc.echr.coe.int/eng?i=001-242417) (no. 37327/24) — the candidate's complaint against the annulment declared inadmissible.

[^src-euronews-2024]: Euronews (23 December 2024), [Romanian centre-right party's TikTok campaign used for far-right win](https://www.euronews.com/next/2024/12/23/romanian-centre-right-partys-tiktok-campaign-used-for-far-right-win-report) — the Snoop.ro / tax-authority finding on the PNL-funded influencer campaign.

[^src-agerpres-2024]: AGERPRES (4 December 2024), [_Declassified documents from CSAT/Romania - target of Russian hybrid actions, Georgescu's campaign - financed_](https://agerpres.ro/2024/12/04/declassified-documents-from-csat-romania---target-of-russian-hybrid-actions-georgescu-s-campaign---f--1398576), Romania's national news agency — as the agency reports the SRI note: a TikTok campaign run "both with the help of recommendation algorithms and through paid promotion"; TikTok confirmed payments of US$381,000 to accounts promoting the candidate.

[^src-atlantic-council]: [Atlantic Council (New Atlanticist) — analysis of the Romania election annulment](https://www.atlanticcouncil.org/blogs/new-atlanticist/romania-annulled-its-presidential-election-results-amid-alleged-russian-interference-what-happens-next/).

[^src-chesney-citron-2019]: Chesney & Citron (2019), [_Deepfakes and the New Disinformation War_](https://www.foreignaffairs.com/articles/world/2018-12-11/deepfakes-and-new-disinformation-war), Foreign Affairs.

[^src-cigi-2025]: [CIGI, _Then and Now: How Does AI Electoral Interference Compare in 2025?_](https://www.cigionline.org/articles/then-and-now-how-does-ai-electoral-interference-compare-in-2025/) (August 2025).

[^src-maslej-2023]: Maslej, N. et al. (April 2023), [_The AI Index 2023 Annual Report_](https://hai.stanford.edu/assets/files/hai_ai-index-report_2023.pdf), Institute for Human-Centered AI, Stanford University — "2022 saw the release of text-to-image models like DALL-E 2 and Stable Diffusion, text-to-video systems like Make-A-Video, and chatbots like ChatGPT."

[^src-goldstein-2023]: Goldstein, J. A., Sastry, G., Musser, M., DiResta, R., Gentzel, M. & Sedova, K. (January 2023), [_Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations_](https://arxiv.org/abs/2301.04246), Georgetown CSET, OpenAI and Stanford Internet Observatory — "Language models drive down the cost of generating propaganda … Influence operations with language models will become easier to scale, and more expensive tactics (e.g., generating personalized content) may become cheaper."

[^src-rfe-rls-romanian-2025]: RFE/RL's Romanian Service (14 February 2025), [_Romanian Opposition Welcomes Vance Comments On Election Annulment_](https://www.rferl.org/a/jd-vance-romanian-comments-election/33315252.html), Radio Free Europe/Radio Liberty — US Vice President JD Vance, at the Munich Security Conference, said Romania cancelled the election on the "flimsy suspicions of an intelligence agency and enormous pressure from its continental neighbors".

[^src-kuti-2024]: Kuti, C. (10 December 2024), [_On Means and Ends: The Annulment of the Romanian Presidential Elections_](https://doi.org/10.59704/c9c08659ed569160), Verfassungsblog — the Court acted ex officio and annulled the whole process although "there is no constitutional or legal provision" for either.

[^src-iancu-2025]: Iancu, B. (13 May 2025), [_Constitutional Death Foretold? The Romanian Elections Saga in a Nutshell_](https://revdem.ceu.edu/2025/05/13/constitutional-death-foretold/), Review of Democracy (CEU Democracy Institute) — Ruling 32 rested on "a canopy provision" of Article 142 and "generally cryptic" intelligence reports.

[^src-cornea-2025]: Cornea, R. (1 April 2025), [_Romanian Militant Democracy in Action_](https://doi.org/10.59704/1a0400f2b9629e46), Verfassungsblog — defends Ruling 32 as "an expression of militant democracy".

[^src-european-commission-2025]: European Commission (December 2025), [_Commission fines X €120 million under the Digital Services Act_](https://ec.europa.eu/commission/presscorner/detail/en/ip_25_2934).

[^src-european-commission-2026]: [European Commission — _Transparency obligations under Article 50 of the AI Act_](https://digital-strategy.ec.europa.eu/en/faqs/transparency-obligations-under-article-50-ai-act) — who the duties fall on, the 2 August 2026 date, the grace period to 2 December 2026, and what counts as human review or editorial control.

[State Actors & Information War →](https://attentionmachine.org/en/attention-machine/state-actors/)\
[What We Don't Yet Know →](https://attentionmachine.org/en/symptoms/unknown/)\
[Yes, but… →](https://attentionmachine.org/en/objections/)

---

Some content on this site was created with the assistance of AI tools and may contain errors. [How we use AI →](https://attentionmachine.org/en/about/how-we-use-ai/)