Some of the marketing materials are the apparent work of a former senior employee who, while working for Oracle, held a visiting researcher position at China’s leading police academy. That man is described in documents as having led a team of people who marketed Oracle’s products to police and governments around the world. Oracle declined to comment for this article.

Oracle previously denied directly selling software to Chinese police for the explicit purpose of combing through citizens’ data. But it confirmed that the slide decks, which were hosted on its website in February, are genuine. Several of the documents have since been taken down. (The Intercept has linked to archived versions when possible.)

“Oracle wants to take the money from supplying tools to intelligence agencies, but then they want to argue that they have no culpability for the use,” said Jack Poulson, executive director of the accountability nonprofit Tech Inquiry, who compared the company’s China efforts to Google’s cloud sales to U.S. Customs and Border Protection, Israel, and Saudi Arabia. “Obviously a company would like to take all of the money they can with as little responsibility as possible. But why are we letting them do that?”

Sales to the Military and FBI

Endeca was founded in 1999 in Cambridge, Massachusetts, within a tech corridor that is sometimes compared to Silicon Valley. The company’s software excelled at analysis of unstructured data, raw information that isn’t organized into a database. Its launch was well-timed; the world soon entered the big data era. E-commerce search was an early application for the technology, and businesses have remained important customers. Endeca, whose name derives from a German word meaning “discover,” got an additional boost from the “war on terror,” which brought an infusion of money for increasing intelligence agencies’ surveillance powers.

In-Q-Tel announced a strategic partnership with Endeca in 2003, the same year that Palantir was founded. “We see significant opportunities to extend and apply their commercial technology in the pursuit of enhanced national security,” In-Q-Tel CEO Gilman Louie said in a press release at the time. According to Endeca, the deal entailed an investment of an undisclosed amount. Two years later, the CIA-funded venture capital firm announced a deal to introduce Endeca to the Defense Intelligence Agency.

Endeca became one of several tools used by government agencies to visualize different pools of data, for example, by plotting data entries on a searchable map. “This type of interface became the norm, even if Endeca didn’t become a household name,” said Poulson. “It’s something that the U.S. intelligence community has continued to invest in.”

An early success story for In-Q-Tel, Endeca eventually gained clients at the FBI, U.S. Cyber Command, and the Army, according to federal spending records. The software has even been used at Guantánamo Bay, the U.S. military base in Cuba where prisoners captured in Afghanistan and elsewhere have been held and tortured. In 2008, the Joint Task Force Guantanamo, a U.S. military unit housed at the base, contracted with a technology firm for “Endeca maintenance,” according to spending records. The unit paid for Endeca support until at least 2019. Some of the transactions are coded Operation Enduring Freedom, the official name used by the U.S. government for the war in Afghanistan. Past job listings for developers at Guantánamo Bay describe “Endeca/Oracle Guided Search” as a requirement.

As Endeca gained intelligence and defense clients in the early 2000s, Oracle was building its base of police clients. The company is known for its relational databases, used to crunch large volumes of data. Publicly available contracts show that one major project was CPD’s Citizen Law Enforcement Analysis and Reporting, or CLEAR, an Oracle data warehouse that today is at the center of a campaign by activists to erase discriminatory data on immigrants and people of color collected by police enforcement agencies.

Along with millions of arrest records, CLEAR houses data on purported gang members that activists say has been used by police to discriminate against people. “It’s racist,” said Xanat Sobrevilla, an organizer with Organized Communities Against Deportations in Chicago, one of the groups behind the campaign Erase the Database. “Ninety-five percent of the names are Black and Latinx. It doesn’t serve our purpose of reducing crime and keeping us safe. It just serves the purpose of being able to justify more policing and brutality.” Erase the Database has documented children as young as 9 who were listed in the database.

By 2011, when Oracle acquired Endeca for a $1.1 billion, U.S. police departments were showing interest in data-driven policing, in which surveillance feeds and personal data are used to identify potential criminals or crime hot spots. Oracle claims to house the world’s largest collection of third-party personal data, and it became one of several large tech companies to venture into the fraught field. “What the Oracles of the world are doing is trying to make a play to be the platform of policing,” said Andrew Guthrie Ferguson, a law professor at American University and author of “The Rise of Big Data Policing: Surveillance, Race, and the Future of Law Enforcement.” “The calculation is, ‘We have all this personal data. Police need personal data. We know how to do data analytics, and we’re doing it in the corporate realm. What if we move into the government space? Wouldn’t that be great?’”

Because large tech companies like Oracle offer a broad variety of technologies, their forays into policing are harder to track than those of more niche companies like Palantir. But their work is no less concerning, said Ferguson. “It should raise red flags when big tech enters the policing space because it takes two forms of unreflective power — big tech and policing — and puts them together.”

In Chicago, Oracle’s acquisition of Endeca came amid police paranoia about the Occupy movement. CPD appears to be one of the first departments to use the software to mine social media. An Oracle white paper calls Endeca’s adoption by CPD a “proof-of-concept” exercise and claims that Endeca helped police zero in on threats during the protests. By “melding Social Media [sic] information with departmental 911 and force deployment information,” the white paper says, “officers could be rapidly redeployed to preemptively address evolving threats.”

As the protests got underway, Tomlinson, the former Oracle employee, tweeted updates. On May 20, 2012, the same day that police drew scrutiny for striking demonstrators with batons, he tweeted that Endeca was processing 8,000 tweets an hour. One day later, he said that traffic had gone up to 20,000 tweets an hour.

A few months later, Tomlinson presented his takeaways. While appearing to click through actual data from the protests, he showed how police could use Endeca to conduct targeted searches, drill down to identify users by location, and draw on information from 15,000 incidents ingested from the National Counterterrorism Center. The software allowed police to sort posts by clout score, filter for “negative” sentiment, and zero in on specific user accounts, he said, noting, “You can start to search through the data based on anything that you might be interested in, such as the word ‘protests.’”

Tomlinson did not respond to requests for comment.

In another talk, also given in 2012, Tomlinson said that, in addition to mining social media posts, CPD could use Endeca to draw on years’ worth of geotagged crime incident reports and arrest notes. One dataset contained 4.8 million incidents in Chicago. Tomlinson showed how, in a few steps, police could home in on just 12 incidents of interest. Palantir has made similar claims about its software.

An Oracle slide deck prepared for a Chinese audience claims that CPD also used Endeca and other software to predict crimes within a three-day window, based on weather, the number of emergency calls, and arrest figures, among other data.

Scholars say that such calculations simply reinforce existing police biases. “It becomes a feedback loop, where the data tells the police to continue patrolling and stopping and surveilling the same people and the same neighborhoods that they’ve always been patrolling and stopping and surveilling,” says Andy Clarno, who coordinates the University of Illinois at Chicago’s Policing in Chicago Research Group and assists with Erase the Database. “And yet it appears as if it’s somehow objective and based in math, rather than based in a long history of racist practices.”

Sales Push in China

Oracle describes Endeca as an almost magical tool that could lead to investigative breakthroughs. One Oracle document says that Endeca helps address gaps in intelligence — what former Defense Secretary Donald Rumsfeld famously called the “unknown unknowns.” “Endeca Information Discovery is particularly powerful to address the pain of ‘we do not know what we do not know,’” the document claims. In one talk, Tomlinson describes the platform as a way for police to sniff around for suspicious activity, without any suspect or incident in mind. “They don’t know what questions they have up front, what they want to ask of the data — so they just start hitting search, they just start hitting guide nav, start drilling, and things pop up.”

That’s an alarming claim, say civil liberties advocates. “Sifting through data and social media posts in hopes of finding ‘unknown unknowns’ sounds more like dragnet surveillance than a targeted and responsible investigatory tactic,” said Matthew Guariglia, a policy analyst with the Electronic Frontier Foundation who focuses on surveillance and policing. “It’s irresponsible and careless from a civil liberties perspective to collect up a vast amount of data in hopes that something in the pile will be a precisely ‘unknown’ line of investigation.”

It is not clear how long Chicago police continued to use Endeca and whether they still use it today. CPD did not respond to a request to comment for this story.

As anger over police killings mounts, several tech companies have tried to downplay their work with U.S. police; Oracle openly boasts about it. “We proudly partner with law enforcement,” Oracle executive vice president Ken Glueck told The Intercept in an April interview, adding that in some cases Oracle takes those same products overseas. “It shouldn’t surprise anybody that if you’ve done something successful in one place that you’re going to market that success story elsewhere.”

Glueck and others have tried to distance the company from reports of collaboration with authorities in China. There, Oracle’s marketing contradicts the image that executives have tried to cultivate as patriotic servants of the U.S. government. (Last year, that image helped Oracle and Walmart edge out rivals in a bid to control U.S. operations for the Chinese-owned social media company TikTok, a deal that is now in limbo.) And yet documents show that after testing the software on the NATO protesters, the tech giant also marketed it for police use there.

In China, one slide deck marked “CONFIDENTIAL” promotes Endeca for use in a provincial “police cloud” project, an effort to centrally house police data that has serious implications for human rights. The document recommends Endeca as one of several Oracle products well-suited to behavior and language analysis and to a “real-time warning” system. A second deck intended for a Chinese government audience shows social media and other data feeding into Endeca and other Oracle software.

Several presentation documents mention Siebel, another Oracle product extensively used by U.S. government and defense clients. They also tout the use of Oracle or Oracle partner technology for speech, facial, and license plate recognition and analysis of telephone metadata.