From the Stephen Lawrence inquiry we learned that the police were institutionally racist. Can it be long before we learn that they are also institutionally corrupt? Almost every month the undercover policing scandal becomes wider and deeper. Today I can reveal a new twist, which in some respects could be the gravest episode yet. It surely makes the case for an independent public inquiry – which is already overwhelming – unarguable.

Before I explain it, here’s a summary of what we know already. Thanks to the remarkable investigations pursued first by the victims of police spies and then by the Guardian journalists Rob Evans and Paul Lewis (whose book Undercover is as gripping as any thriller), we know that British police have been inserting undercover officers into protest movements since 1968. Their purpose was to counter what they called subversion or domestic extremism, which they define as seeking to “prevent something from happening or to change legislation or domestic policy … outside the normal democratic process”. Which is a good description of how almost all progressive change happens.

Most of the groups whose infiltration has now been exposed were non-violent. Among them were the British campaign against apartheid in South Africa, the protest movements against climate change, people seeking to expose police corruption and the campaign for justice for the murdered black teenager Stephen Lawrence. Undercover officers, often using the stolen identities of dead children, worked their way into key positions and helped to organise demonstrations. Several started long-term relationships with the people they spied on. At least two fathered children with them.

Some officers illegally used their false identities in court. Some acted as agents provocateurs. Seldom did they appear to be operating in the wider interests of society. They collected intelligence on trade unionists that was passed to an agency which compiled unlawful blacklists for construction companies, ensuring that those people could not find work. The policeman who infiltrated the Stephen Lawrence campaign was instructed by his superiors to “hunt for disinformation” about the family and their supporters that could be used to undermine them. When their tour of duty was over, the police abandoned their partners and their assumed identities and disappeared, leaving a trail of broken lives. As the unofficial motto of the original undercover squad stated, it would operate By Any Means Necessary.

The revelations so far have led to 56 people having their cases or convictions overturned, after police and prosecutors failed to disclose that officers had helped to plan and execute the protests for which people were being prosecuted. But we know the names of only 11 spies, out of 100-150, working for 46 years. Thousands of people might have been falsely prosecuted.

So far there have been 15 official inquiries and investigations. They seem to have served only to delay and distract. The report by Sir Christopher Rose into the false convictions of a group of climate change protesters concluded that failures by police and prosecutors to disclose essential information to the defence “were individual, not systemic” and that “nothing that I have seen or heard suggests that … there was any deliberate, still less dishonest, withholding of information”. Now, after an almost identical case involving another group of climate activists, during which the judge remarked that there had been “a complete and total failure” to disclose evidence, Rose’s findings look incredible.

The biggest inquiry still running, Operation Herne, is investigating alleged misconduct by the Metropolitan police. Of its 44 staff, 75% work for, er, the Metropolitan police. Its only decisive action so far has been to seek evidence for a prosecution under the Official Secrets Act of Peter Francis, the police whistleblower who has revealed key elements of this story. This looks like an attempt to discourage him from testifying, and to prevent other officers from coming forward.

Bad enough? You haven’t heard the half of it. Last week, the activist John Jordan was told his conviction (for occupying the offices of London Transport) would be overturned. The Crown Prosecution Service refuses to reveal why, but it doubtless has something to do with the fact that one of Jordan’s co-defendants turns out to have been Jim Boyling, a secret policeman working for the Met, who allegedly used his false identity in court.

Jordan has now made a further claim. He alleges that the same man helped organise a street party that went wrong and turned into the worst riot in London since the poll tax demonstrations. The J18 Carnival Against Global Capitalism on 18 June 1999 went well beyond non-violent protest. According to the police, 42 people were injured and over £1m of damage was done. One building was singled out: the London International Financial Futures Exchange (Liffe), where derivatives were traded. Though protesters entered the building at 1.40pm, the police did not arrive until 4.15pm.

After furious recriminations from the Lord Mayor and the people who ran the Liffe building, the City of London police conducted an inquiry. It admitted that their criticisms were justified, and that the police’s performance was “highly unsatisfactory”. The problem, it claimed, was that the police had no information about what the targets and plans of the protesters would be, and had no idea that Liffe was in the frame. The riot was “unforeseen”.

Jordan was a member of “the logistics group that organised the tactics for J18. There were about 10 of us in the group and we met weekly for over six months.” Among the other members, he says, was Boyling. “The 10 of us … were the only people who knew the whole plan before the day itself and who had decided that the main target would be Liffe.” Boyling, he alleges, drove one of the two cars that were used to block the road to the building.

It is hard to think of a more serious allegation. For six months an undercover officer working for the Metropolitan police was instrumental in planning a major demonstration, which ended up causing injuries and serious damage to property. Yet the police appear to have failed to pass this intelligence to the City of London force, leaving the target of the protest unprotected.

Still no need for an independent public inquiry? Really?

---