Just over a week ago, Donald Trump gathered members of the world’s press before him and told them they were liars. “The press, honestly, is out of control,” he said. “The public doesn’t believe you any more.” CNN was described as “very fake news… story after story is bad”. The BBC was “another beauty”.

That night I did two things. First, I typed “Trump” in the search box of Twitter. My feed was reporting that he was crazy, a lunatic, a raving madman. But that wasn’t how it was playing out elsewhere. The results produced a stream of “Go Donald!!!!”, and “You show ’em!!!” There were star-spangled banner emojis and thumbs-up emojis and clips of Trump laying into the “FAKE news MSM liars!”

Trump had spoken, and his audience had heard him. Then I did what I’ve been doing for two and a half months now. I Googled “mainstream media is…” And there it was. Google’s autocomplete suggestions: “mainstream media is… dead, dying, fake news, fake, finished”. Is it dead, I wonder? Has FAKE news won? Are we now the FAKE news? Is the mainstream media – we, us, I – dying?

I click Google’s first suggested link. It leads to a website called CNSnews.com and an article: “The Mainstream media are dead.” They’re dead, I learn, because they – we, I – “cannot be trusted”. How had it, an obscure site I’d never heard of, dominated Google’s search algorithm on the topic? In the “About us” tab, I learn CNSnews is owned by the Media Research Center, which a click later I learn is “America’s media watchdog”, an organisation that claims an “unwavering commitment to neutralising leftwing bias in the news, media and popular culture”.

Another couple of clicks and I discover that it receives a large bulk of its funding – more than $10m in the past decade – from a single source, the hedge fund billionaire Robert Mercer. If you follow US politics you may recognise the name. Robert Mercer is the money behind Donald Trump. But then, I will come to learn, Robert Mercer is the money behind an awful lot of things. He was Trump’s single biggest donor. Mercer started backing Ted Cruz, but when he fell out of the presidential race he threw his money – $13.5m of it – behind the Trump campaign.

It’s money he’s made as a result of his career as a brilliant but reclusive computer scientist. He started his career at IBM, where he made what the Association for Computational Linguistics called “revolutionary” breakthroughs in language processing – a science that went on to be key in developing today’s AI – and later became joint CEO of Renaissance Technologies, a hedge fund that makes its money by using algorithms to model and trade on the financial markets.

One of its funds, Medallion, which manages only its employees’ money, is the most successful in the world – generating $55bn so far. And since 2010, Mercer has donated $45m to different political campaigns – all Republican – and another $50m to non-profits – all rightwing, ultra-conservative. This is a billionaire who is, as billionaires are wont, trying to reshape the world according to his personal beliefs.

Robert Mercer very rarely speaks in public and never to journalists, so to gauge his beliefs you have to look at where he channels his money: a series of yachts, all called Sea Owl; a $2.9m model train set; climate change denial (he funds a climate change denial thinktank, the Heartland Institute); and what is maybe the ultimate rich man’s plaything – the disruption of the mainstream media. In this he is helped by his close associate Steve Bannon, Trump’s campaign manager and now chief strategist. The money he gives to the Media Research Center, with its mission of correcting “liberal bias” is just one of his media plays. There are other bigger, and even more deliberate strategies, and shining brightly, the star at the centre of the Mercer media galaxy, is Breitbart.

It was $10m of Mercer’s money that enabled Bannon to fund Breitbart – a rightwing news site, set up with the express intention of being a Huffington Post for the right. It has launched the careers of Milo Yiannopoulos and his like, regularly hosts antisemitic and Islamophobic views, and is currently being boycotted by more than 1,000 brands after an activist campaign. It has been phenomenally successful: the 29th most popular site in America with 2bn page views a year. It’s bigger than its inspiration, the Huffington Post, bigger, even, than PornHub. It’s the biggest political site on Facebook. The biggest on Twitter.

Prominent rightwing journalist Andrew Breitbart, who founded the site but died in 2012, told Bannon that they had “to take back the culture”. And, arguably, they have, though American culture is only the start of it. In 2014, Bannon launched Breitbart London, telling the New York Times it was specifically timed ahead of the UK’s forthcoming election. It was, he said, the latest front “in our current cultural and political war”. France and Germany are next.

But there was another reason why I recognised Robert Mercer’s name: because of his connection to Cambridge Analytica, a small data analytics company. He is reported to have a $10m stake in the company, which was spun out of a bigger British company called SCL Group. It specialises in “election management strategies” and “messaging and information operations”, refined over 25 years in places like Afghanistan and Pakistan. In military circles this is known as “psyops” – psychological operations. (Mass propaganda that works by acting on people’s emotions.)

Cambridge Analytica worked for the Trump campaign and, so I’d read, the Leave campaign. When Mercer supported Cruz, Cambridge Analytica worked with Cruz. When Robert Mercer started supporting Trump, Cambridge Analytica came too. And where Mercer’s money is, Steve Bannon is usually close by: it was reported that until recently he had a seat on the board.

Last December, I wrote about Cambridge Analytica in a piece about how Google’s search results on certain subjects were being dominated by rightwing and extremist sites. Jonathan Albright, a professor of communications at Elon University, North Carolina, who had mapped the news ecosystem and found millions of links between rightwing sites “strangling” the mainstream media, told me that trackers from sites like Breitbart could also be used by companies like Cambridge Analytica to follow people around the web and then, via Facebook, target them with ads.

On its website, Cambridge Analytica makes the astonishing boast that it has psychological profiles based on 5,000 separate pieces of data on 220 million American voters – its USP is to use this data to understand people’s deepest emotions and then target them accordingly. The system, according to Albright, amounted to a “propaganda machine”.

A few weeks later, the Observer received a letter. Cambridge Analytica was not employed by the Leave campaign, it said. Cambridge Analytica “is a US company based in the US. It hasn’t worked in British politics.”

Which is how, earlier this week, I ended up in a Pret a Manger near Westminster with Andy Wigmore, Leave.EU’s affable communications director, looking at snapshots of Donald Trump on his phone. It was Wigmore who orchestrated Nigel Farage’s trip to Trump Tower – the PR coup that saw him become the first foreign politician to meet the president elect.

Wigmore scrolls through the snaps on his phone. “That’s the one I took,” he says pointing at the now globally famous photo of Farage and Trump in front of his golden elevator door giving the thumbs-up sign. Wigmore was one of the “bad boys of Brexit” – a term coined by Arron Banks, the Bristol-based businessman who was Leave.EU’s co-founder.

Cambridge Analytica had worked for them, he said. It had taught them how to build profiles, how to target people and how to scoop up masses of data from people’s Facebook profiles. A video on YouTube shows one of Cambridge Analytica’s and SCL’s employees, Brittany Kaiser, sitting on the panel at Leave.EU’s launch event.

Facebook was the key to the entire campaign, Wigmore explained. A Facebook ‘like’, he said, was their most “potent weapon”. “Because using artificial intelligence, as we did, tells you all sorts of things about that individual and how to convince them with what sort of advert. And you knew there would also be other people in their network who liked what they liked, so you could spread. And then you follow them. The computer never stops learning and it never stops monitoring.”

It sounds creepy, I say.

“It is creepy! It’s really creepy! It’s why I’m not on Facebook! I tried it on myself to see what information it had on me and I was like, ‘Oh my God!’ What’s scary is that my kids had put things on Instagram and it picked that up. It knew where my kids went to school.”

They hadn’t “employed” Cambridge Analytica, he said. No money changed hands. “They were happy to help.”

Why?

“Because Nigel is a good friend of the Mercers. And Robert Mercer introduced them to us. He said, ‘Here’s this company we think may be useful to you.’ What they were trying to do in the US and what we were trying to do had massive parallels. We shared a lot of information. Why wouldn’t you?” Behind Trump’s campaign and Cambridge Analytica, he said, were “the same people. It’s the same family.”

There were already a lot of questions swirling around Cambridge Analytica, and Andy Wigmore has opened up a whole lot more. Such as: are you supposed to declare services-in-kind as some sort of donation? The Electoral Commission says yes, if it was more than £7,500. And was it declared? The Electoral Commission says no. Does that mean a foreign billionaire had possibly influenced the referendum without that influence being apparent? It’s certainly a question worth asking.

In the last month or so, articles in first the Swiss and the US press have asked exactly what Cambridge Analytica is doing with US voters’ data. In a statement to the Observer, the Information Commissioner’s Office said: “Any business collecting and using personal data in the UK must do so fairly and lawfully. We will be contacting Cambridge Analytica and asking questions to find out how the company is operating in the UK and whether the law is being followed.”

Cambridge Analytica said last Friday they are in touch with the ICO and are completely compliant with UK and EU data laws. It did not answer other questions the Observer put to it this week about how it built its psychometric model, which owes its origins to original research carried out by scientists at Cambridge University’s Psychometric Centre, research based on a personality quiz on Facebook that went viral. More than 6 million people ended up doing it, producing an astonishing treasure trove of data.

These Facebook profiles – especially people’s “likes” – could be correlated across millions of others to produce uncannily accurate results. Michal Kosinski, the centre’s lead scientist, found that with knowledge of 150 likes, their model could predict someone’s personality better than their spouse. With 300, it understood you better than yourself. “Computers see us in a more robust way than we see ourselves,” says Kosinski.

But there are strict ethical regulations regarding what you can do with this data. Did SCL Group have access to the university’s model or data, I ask Professor Jonathan Rust, the centre’s director? “Certainly not from us,” he says. “We have very strict rules around this.”

A scientist, Aleksandr Kogan, from the centre was contracted to build a model for SCL, and says he collected his own data. Professor Rust says he doesn’t know where Kogan’s data came from. “The evidence was contrary. I reported it.” An independent adjudicator was appointed by the university. “But then Kogan said he’d signed a non-disclosure agreement with SCL and he couldn’t continue [answering questions].”

Kogan disputes this and says SCL satisfied the university’s inquiries. But perhaps more than anyone, Professor Rust understands how the kind of information people freely give up to social media sites could be used.

“The danger of not having regulation around the sort of data you can get from Facebook and elsewhere is clear. With this, a computer can actually do psychology, it can predict and potentially control human behaviour. It’s what the scientologists try to do but much more powerful. It’s how you brainwash someone. It’s incredibly dangerous.

“It’s no exaggeration to say that minds can be changed. Behaviour can be predicted and controlled. I find it incredibly scary. I really do. Because nobody has really followed through on the possible consequences of all this. People don’t know it’s happening to them. Their attitudes are being changed behind their backs.”

Mercer invested in Cambridge Analytica, the Washington Post reported, “driven in part by an assessment that the right was lacking sophisticated technology capabilities”. But in many ways, it’s what Cambridge Analytica’s parent company does that raises even more questions.

Emma Briant, a propaganda specialist at the University of Sheffield, wrote about SCL Group in her 2015 book, Propaganda and Counter-Terrorism: Strategies for Global Change. Cambridge Analytica has the technological tools to effect behavioural and psychological change, she said, but it’s SCL that strategises it. It has specialised, at the highest level – for Nato, the MoD, the US state department and others – in changing the behaviour of large groups. It models mass populations and then it changes their beliefs.

SCL was founded by someone called Nigel Oakes, who worked for Saatchi & Saatchi on Margaret Thatcher’s image, says Briant, and the company had been “making money out of the propaganda side of the war on terrorism over a long period of time. There are different arms of SCL but it’s all about reach and the ability to shape the discourse. They are trying to amplify particular political narratives. And they are selective in who they go for: they are not doing this for the left.”

In the course of the US election, Cambridge Analytica amassed a database, as it claims on its website, of almost the entire US voting population – 220 million people – and the Washington Post reported last week that SCL was increasing staffing at its Washington office and competing for lucrative new contracts with Trump’s administration. “It seems significant that a company involved in engineering a political outcome profits from what follows. Particularly if it’s the manipulation, and then resolution, of fear,” says Briant.

It’s the database, and what may happen to it, that particularly exercises Paul-Olivier Dehaye, a Swiss mathematician and data activist who has been investigating Cambridge Analytica and SCL for more than a year. “How is it going to be used?” he says. “Is it going to be used to try and manipulate people around domestic policies? Or to ferment conflict between different communities? It is potentially very scary. People just don’t understand the power of this data and how it can be used against them.”

There are two things, potentially, going on simultaneously: the manipulation of information on a mass level, and the manipulation of information at a very individual level. Both based on the latest understandings in science about how people work, and enabled by technological platforms built to bring us together.

Are we living in a new era of propaganda, I ask Emma Briant? One we can’t see, and that is working on us in ways we can’t understand? Where we can only react, emotionally, to its messages? “Definitely. The way that surveillance through technology is so pervasive, the collection and use of our data is so much more sophisticated. It’s totally covert. And people don’t realise what is going on.”

Public mood and politics goes through cycles. You don’t have to subscribe to any conspiracy theory, Briant says, to see that a mass change in public sentiment is happening. Or that some of the tools in action are straight out of the military’s or SCL’s playbook.

But then there’s increasing evidence that our public arenas – the social media sites where we post our holiday snaps or make comments about the news – are a new battlefield where international geopolitics is playing out in real time. It’s a new age of propaganda. But whose? This week, Russia announced the formation of a new branch of the military: “information warfare troops”.

Sam Woolley of the Oxford Internet Institute’s computational propaganda institute tells me that one third of all traffic on Twitter before the EU referendum was automated “bots” – accounts that are programmed to look like people, to act like people, and to change the conversation, to make topics trend. And they were all for Leave. Before the US election, they were five-to-one in favour of Trump – many of them Russian. Last week they have been in action in the Stoke byelection – Russian bots, organised by who? – attacking Paul Nuttall.

“Politics is war,” said Steve Bannon last year in the Wall Street Journal. And increasingly this looks to be true.

There’s nothing accidental about Trump’s behaviour, Andy Wigmore tells me. “That press conference. It was absolutely brilliant. I could see exactly what he was doing. There’s feedback going on constantly. That’s what you can do with artificial intelligence. You can measure ever reaction to every word. He has a word room, where you fix key words. We did it. So with immigration, there are actually key words within that subject matter which people are concerned about. So when you are going to make a speech, it’s all about how can you use these trending words.”

Wigmore met with Trump’s team right at the start of the Leave campaign. “And they said the holy grail was artificial intelligence.”

Who did?

“Jared Kushner and Jason Miller.”

Later, when Trump picked up Mercer and Cambridge Analytica, the game changed again. “It’s all about the emotions. This is the big difference with what we did. They call it bio-psycho-social profiling. It takes your physical, mental and lifestyle attributes and works out how people work, how they react emotionally.”

Bio-psycho-social profiling, I read later, is one offensive in what is called “cognitive warfare”. Though there are many others: “recoding the mass consciousness to turn patriotism into collaborationism,” explains a Nato briefing document on countering Russian disinformation written by an SCL employee. “Time-sensitive professional use of media to propagate narratives,” says one US state department white paper. “Of particular importance to psyop personnel may be publicly and commercially available data from social media platforms.”

Yet another details the power of a “cognitive casualty” – a “moral shock” that “has a disabling effect on empathy and higher processes such as moral reasoning and critical thinking”. Something like immigration, perhaps. Or “fake news”. Or as it has now become: “FAKE news!!!!”

How do you change the way a nation thinks? You could start by creating a mainstream media to replace the existing one with a site such as Breitbart. You could set up other websites that displace mainstream sources of news and information with your own definitions of concepts like “liberal media bias”, like CNSnews.com. And you could give the rump mainstream media, papers like the “failing New York Times!” what it wants: stories. Because the third prong of Mercer and Bannon’s media empire is the Government Accountability Institute.

Bannon co-founded it with $2m of Mercer’s money. Mercer’s daughter, Rebekah, was appointed to the board. Then they invested in expensive, long-term investigative journalism. “The modern economics of the newsroom don’t support big investigative reporting staffs,” Bannon told Forbes magazine. “You wouldn’t get a Watergate, a Pentagon Papers today, because nobody can afford to let a reporter spend seven months on a story. We can. We’re working as a support function.”

Welcome to the future of journalism in the age of platform capitalism. News organisations have to do a better job of creating new financial models. But in the gaps in between, a determined plutocrat and a brilliant media strategist can, and have, found a way to mould journalism to their own ends.

In 2015, Steve Bannon described to Forbes how the GAI operated, employing a data scientist to trawl the dark web (in the article he boasts of having access to $1.3bn worth of supercomputers) to dig up the kind of source material Google can’t find. One result has been a New York Times bestseller, Clinton Cash: The Untold Story of How and Why Foreign Governments and Businesses Helped Make Bill and Hillary Rich, written by GAI’s president, Peter Schweizer and later turned into a film produced by Rebekah Mercer and Steve Bannon.

This, Bannon explained, is how you “weaponise” the narrative you want. With hard researched facts. With those, you can launch it straight on to the front page of the New York Times, as the story of Hillary Clinton’s cash did. Like Hillary’s emails it turned the news agenda, and, most crucially, it diverted the attention of the news cycle. Another classic psyops approach. “Strategic drowning” of other messages.

This is a strategic, long-term and really quite brilliant play. In the 1990s, Bannon explained, conservative media couldn’t take Bill Clinton down because “they wound up talking to themselves in an echo chamber”.

As, it turns out, the liberal media is now. We are scattered, separate, squabbling among ourselves and being picked off like targets in a shooting gallery. Increasingly, there’s a sense that we are talking to ourselves. And whether it’s Mercer’s millions or other factors, Jonathan Albright’s map of the news and information ecosystem shows how rightwing sites are dominating sites like YouTube and Google, bound tightly together by millions of links.

Is there a central intelligence to that, I ask Albright? “There has to be. There has to be some type of coordination. You can see from looking at the map, from the architecture of the system, that this is not accidental. It’s clearly being led by money and politics.”

There’s been a lot of talk in the echo chamber about Bannon in the last few months, but it’s Mercer who provided the money to remake parts of the media landscape. And while Bannon understands the media, Mercer understands big data. He understands the structure of the internet. He knows how algorithms work.

Robert Mercer did not respond to a request for comment for this piece. Nick Patterson, a British cryptographer, who worked at Renaissance Technologies in the 80s and is now a computational geneticist at MIT, described to me how he was the one who talent-spotted Mercer. “There was an elite group working at IBM in the 1980s doing speech research, speech recognition, and when I joined Renaissance I judged that the mathematics we were trying to apply to financial markets were very similar.”

He describes Mercer as “very, very conservative. He truly did not like the Clintons. He thought Bill Clinton was a criminal. And his basic politics, I think, was that he’s a rightwing libertarian, he wants the government out of things.”

He suspects that Mercer is bringing the brilliant computational skills he brought to finance to bear on another very different sphere. “We make mathematical models of the financial markets which are probability models, and from those we try and make predictions. What I suspect Cambridge Analytica do is that they build probability models of how people vote. And then they look at what they can do to influence that.”

Finding the edge is what quants do. They build quantitative models that automate the process of buying and selling shares and then they chase tiny gaps in knowledge to create huge wins. Renaissance Technologies was one of the first hedge funds to invest in AI. But what it does with it, how it’s been programmed to do it, is completely unknown. It is, Bloomberg reports, the “blackest box in finance”.

Johan Bollen, associate professor at Indiana University School of Informatics and Computing, tells me how he discovered one possible edge: he’s done research that shows you can predict stock market moves from Twitter. You can measure public sentiment and then model it. “Society is driven by emotions, which it’s always been difficult to measure, collectively. But there are now programmes that can read text and measure it and give us a window into those collective emotions.”

The research caused a huge ripple among two different constituencies. “We had a lot attention from hedge funds. They are looking for signals everywhere and this is a hugely interesting signal. My impression is hedge funds do have these algorithms that are scanning social feeds. The flash crashes we’ve had – sudden huge drops in stock prices – indicates these algorithms are being used at large scale. And they are engaged in something of an arms race.”

The other people interested in Bollen’s work are those who want not only to measure public sentiment, but to change it. Bollen’s research shows how it’s possible. Could you reverse engineer the national, or even the global, mood? Model it, and then change it?

“It does seem possible. And it does worry me. There are quite a few pieces of research that show if you repeat something often enough, people start involuntarily to believe it. And that could be leveraged, or weaponised for propaganda. We know there are thousands of automated bots out there that are trying to do just that.”

THE war of the bots is one of the wilder and weirder aspects of the elections of 2016. At the Oxford Internet Institute’s Unit for Computational Propaganda, its director, Phil Howard, and director of research, Sam Woolley, show me all the ways public opinion can be massaged and manipulated. But is there a smoking gun, I ask them, evidence of who is doing this? “There’s not a smoking gun,” says Howard. “There are smoking machine guns. There are multiple pieces of evidence.”

“Look at this,” he says and shows me how, before the US election, hundreds upon hundreds of websites were set up to blast out just a few links, articles that were all pro-Trump. “This is being done by people who understand information structure, who are bulk buying domain names and then using automation to blast out a certain message. To make Trump look like he’s a consensus.”

And that requires money?

“That requires organisation and money. And if you use enough of them, of bots and people, and cleverly link them together, you are what’s legitimate. You are creating truth.”

You can take an existing trending topic, such as fake news, and then weaponise it. You can turn it against the very media that uncovered it. Viewed in a certain light, fake news is a suicide bomb at the heart of our information system. Strapped to the live body of us – the mainstream media.

One of the things that concerns Howard most is the hundreds of thousands of “sleeper” bots they’ve found. Twitter accounts that have tweeted only once or twice and are now sitting quietly waiting for a trigger: some sort of crisis where they will rise up and come together to drown out all other sources of information.

Like zombies?

“Like zombies.”

Many of the techniques were refined in Russia, he says, and then exported everywhere else. “You have these incredible propaganda tools developed in an authoritarian regime moving into a free market economy with a complete regulatory vacuum. What you get is a firestorm.”

This is the world we enter every day, on our laptops and our smartphones. It has become a battleground where the ambitions of nation states and ideologues are being fought – using us. We are the bounty: our social media feeds; our conversations; our hearts and minds. Our votes. Bots influence trending topics and trending topics have a powerful effect on algorithms, Woolley, explains, on Twitter, on Google, on Facebook. Know how to manipulate information structure and you can manipulate reality.

We’re not quite in the alternative reality where the actual news has become “FAKE news!!!” But we’re almost there. Out on Twitter, the new transnational battleground for the future, someone I follow tweets a quote by Marshall McLuhan, the great information theorist of the 60s. “World War III will be a guerrilla information war,” it says. “With no divisions between military and civilian participation.”

By that definition we’re already there.

Additional reporting by Paul-Olivier Dehaye

Carole Cadwalladr will be hosting a discussion on technology’s disruption of democracy at the bluedot festival, Jodrell Bank, Cheshire, 7-9 July