UC Berkeley expert warns AI is shifting fraud risk toward proving reality

Generative AI is already changing fraud prevention by making synthetic images, voices and video easier to produce and harder for people to detect, UC Berkeley professor Hany Farid told the ACFE Global Fraud Conference in Boston this week. Farid said the operational risk for companies now includes both identifying manipulated content and proving that genuine records, calls and images are authentic.

Farid, an AI and deepfake detection specialist, told delegates that the speed of development has compressed sharply. “We used to measure change in technology in 12 to 18 months. We now measure it in 12 to 18 days,” he said.

He said commercially available systems can generate realistic media from text prompts, while a person’s identity can be cloned with limited material. “I can take an identity, and what I need to take that identity is about 15 seconds of your voice and one image,” Farid said, adding that such material is available online for many people.

Remote work and onboarding face new pressure

Farid said the threat has moved beyond manipulated social media posts into financial crime, identity theft, employment fraud, disinformation and fabricated evidence. He highlighted real-time impersonation on video calls as a particular concern for organisations that rely on remote hiring, vendor checks and executive approvals.

Fraudsters can use face-swapping and voice-cloning software to appear as another person on conferencing platforms including Zoom, Teams and Webex, Farid said. “Suddenly now the video calls you get, the FaceTime calls, the Zoom calls, the Teams calls, the WebEx calls, you don’t know who’s on the other end of that line,” he told the conference.

Farid cited cases involving North Korean operatives obtaining remote jobs with Western companies and said many large businesses have not assigned clear responsibility for AI-enabled fraud risk. In one boardroom discussion, he said, a chairman stated the company had not seen AI-fraud incidents before multiple executives contradicted him. “Ten hands went up in the room,” Farid said.

Human detection is falling short

Research by Farid’s team found that people shown a mix of real and AI-generated images, audio and video performed only slightly better than chance when asked to identify which was which, he said. Training did not solve the problem, according to Farid. “We’ve trained them. We’ve explained to them how this works. They’re basically at chance,” he said.

Farid said confidence was a poor guide to accuracy. “People who think they’re good at it are bad,” he told delegates. He also rejected common consumer advice about spotting obvious defects, such as extra fingers, saying model quality has improved enough to make those signs unreliable.

The same technology is also creating what Farid called the “liar’s dividend”, in which authentic material can be dismissed as artificial. He said he spent hours examining a photograph posted by US senator Mitch McConnell after online users claimed it was AI-generated. “We analysed the photo and, if you’re interested, it is real,” Farid said.

Farid said that erosion of trust affects organisations, democracies and courts. “Evidence in courts of law is complete and utter chaos,” he said, adding that legal systems are struggling to assess digital evidence.

Technical controls and governance

Farid said organisations should rely less on human judgment and more on technical controls, including content credentials, invisible watermarks from AI providers, digital signatures and forensic analysis of lighting, shadows, facial biometrics and voice patterns.

He said current AI systems often fail to model physical reality, creating inconsistencies in geometry, lighting and perspective that forensic tools can identify. He also described real-time monitoring systems that can flag suspicious virtual cameras, voice modulation software and other indicators during video conferences.

Farid urged companies to assign ownership for AI-related identity and fraud risks and to train staff before an incident forces action. For sensitive communications, he said individuals and organisations can also use basic verification procedures, including code words. Farid said he adopted that approach after fraudsters cloned his voice and contacted a lawyer working on a confidential case.

This story draws on original reporting from Finextra Research.