The police were hot on his trail for two murders and when pressured by the leader of his new gang to explain what he’d done, Trestan Brown confided in her. He figured he could trust her. She was, after all, a fellow criminal.

“Be up front with me and I’ll take care of you,” the gang leader, Steph, told Mr. Brown in a hotel room along the lakefront of Kingston, Ontario, according to court records.

So Mr. Brown, who is now 30, confessed, describing his role in the 2016 shooting deaths of two men in the Toronto area — and soon found himself under arrest.

Steph, it turned out, wasn’t a crime boss and her group wasn’t a gang. They were undercover officers conducting a “Mr. Big” operation, an elaborate Canadian policing tactic that is being challenged in court and that some opponents want banned.

An invention of the Royal Canadian Mounted Police, or R.C.M.P., the technique involves officers creating a fake criminal enterprise, then enticing a target to join and ultimately confess to previous crimes.

It has been used in hundreds of cold cases across Canada, with a 95 percent success rate in securing convictions, according to the Mounties. Law enforcement agencies in a handful of other countries have even copied the method.

But critics argue that the confessions obtained in Mr. Big operations are unreliable because they are often coerced.

In 2014, the Supreme Court of Canada established safeguards aimed at reducing the risk of wrongful convictions. But pitfalls persist, said Alison Craig, a criminal defense lawyer in Toronto who has represented several people ensnared by the technique, including Mr. Brown.

The police still use threats, inducements — such as financial incentives — promises and psychological control to elicit confessions, critics said.

“It’s just a recipe for wrongful convictions,” Ms. Craig said.

Defending the practice are the police themselves and the families of some victims. Cpl. Kim Chamberland, a spokeswoman for the Royal Canadian Mounted Police, said the tactic had also been used to exonerate suspects.

“The focus is on uncovering the truth, verifying facts and determining if someone is involved,” in major crimes, Corporal Chamberland wrote in an email.

In the United States, federal law enforcement agencies have used undercover informants to lure people who have expressed vague support for terrorism or terrorist groups into committing crimes by making it seem that the informants can, for example, provide arms or money.

The practice, which became more common after the Sept. 11 terrorist attacks, has been criticized by civil liberties groups and defense lawyers as a form of entrapment that can sweep up people who never had the resources or the actual intent to carry out violence.

The Mr. Big operations in Canada are typically broader than relying on outside informants and involve groups of law enforcement officers themselves working undercover and concocting an elaborate ruse to target people believed to have committed crimes.

In popular culture, “Mr. Big” refers to someone who heads a criminal organization. Since the 1980s, when the R.C.M.P. started using it, Mr. Big operations have been conducted more than 350 times in Canada and tend to follow the same playbook.

First, undercover officers contrive a situation to cross paths with the target of an investigation. In Mr. Brown’s case, the officers played the role of strangers buying his broken down car for parts.

The officers play up the illusion of a shared criminal history and befriend the target, who is steadily given greater criminal responsibilities by the fake gang and compensated for work.

Orchestrated violence, including simulated beatings, can help add legitimacy to the ruse, and stoke fear of Mr. Big. Some are especially graphic.

Al Potter, who was convicted of first-degree murder in 2019 following a Mr. Big operation in Newfoundland, helped undercover officers bury, in a remote cornfield, what he thought was the body of a man who owed a debt to the gang. It turned out to be a dead pig stuffed in a hockey bag.

Finally, the sting shifts toward eliciting a confession. The officer playing Mr. Big threatens to cut ties unless the target explains why the police are targeting him, reassured that Mr. Big is powerful enough to make his problem disappear.

Since the 2014 ruling by the Supreme Court of Canada, prosecutors seeking to use the secretly recorded confessions obtained through Mr. Big operations must first persuade a judge that they comply with certain standards.

But judges don’t seem to agree about how to apply the standards, according to a review of 61 cases published by the Manitoba Law Journal. The 2014 ruling “does not appear to have had a significant impact” on whether confessions are allowed in court, the authors concluded.

Some inmates who confessed under Mr. Big operations, especially before the 2014 ruling, have appealed their convictions and sued law enforcement officials, accusing them of misconduct.

One of the most publicized exonerations was that of Kyle Unger, who was vindicated by DNA evidence after spending 14 years in jail for a 1990 murder and sexual assault. In 2019, he reached an out-of-court settlement after claiming 14.5 million Canadian dollars (about $10.7 million) in damages against the police and various justice authorities.

In another well-publicized case, Canada’s justice minister, David Lametti, issued a rare decision and intervened in favor of an inmate, Wade Skiffington.

Mr. Lametti found that there was probably a miscarriage of justice in the conviction of Mr. Skiffington, who was found guilty of second-degree murder after confessing, during a Mr. Big sting, to the 1994 killing of his fiancée in British Columbia.

Mr. Lametti referred the case, which had exhausted its appeals, back to the province’s appeal court, where Mr. Skiffington’s lawyers will argue for his exoneration.

The necessity of Mr. Lametti’s intervention “gives renewed cause for concern about these types of investigations,” said Matthew Gourlay, a criminal lawyer in Toronto who has petitioned high courts against the use of Mr. Big operations.

The country’s Department of Justice announced in February that an independent commission would be created to hear cases from other defendants who have exhausted their court appeals.

A legal flaw with Mr. Big operations is that targets are effectively under state control, as if being detained, yet they cannot be warned of their right to silence or right to counsel, said Timothy Moore, a psychology professor at York University in Toronto.

“That’s the major wrinkle with Mr. Big,” said Professor Moore, who has studied the technique and been consulted on several Mr. Big cases. “The police can do and say things that they otherwise could not if the target was aware that he’s dealing with state agents.”

Still, the Mr. Big technique has spread beyond Canada to other countries, including Australia, New Zealand and the Netherlands.

In Australia, public support for it grew after the tactic was used to solve two high-profile child murders, said Michele Ruyters, a criminal law professor at RMIT University in Melbourne and founder of an organization that investigates claims of wrongful conviction.

“It’s very difficult to advocate against Mr. Big schemes because they appear to be so successful in these notorious cases,” Ms. Ruyters said.

The expense of running these investigations and the ensuing “thorny issues” make Mr. Big operations a tool of last resort, but an important one, said Richard Boyington, a retired sergeant from Brantford, Ontario.

“Think of yourself as the family of the victim,” said Mr. Boyington, who managed a Mr. Big sting in the early 2000s that led to a murder conviction. “You would want the police to do everything within their lawful power to bring this investigation to a successful conclusion.”

Mr. Brown was one of many defendants who never challenged their convictions after being stung by Mr. Big. He pleaded guilty to the 2016 killings and was sentenced in October to life in prison with no chance of parole for 16 years.

He shot and killed one of his victims, Abdullah Farah, in a case of mistaken identity. Mr. Farah, a college student in California who was visiting his family, was a funny and outgoing 20-year-old man “that everyone loved,” said his sister, Ifrah Farah.

She is apprenticing as a lawyer at a corporate law firm and remains ambivalent about Mr. Big stings but is relieved that her brother’s murder was solved after years when the investigation “felt kind of hopeless,” she said. “We would have just been left always wondering what happened.”