ProPublica

The Trump administration made false assertions to justify an executive order expanding police forces’ access to military equipment such as tanks and grenade launchers.

Attorney General Jeff Sessions announced on Monday that President Trump would make defensive gear available to police again by undoing a policy from the Obama administration. Trump then signed an executive order whose title emphasized that branding: “Restoring State, Tribal, and Local Law Enforcement’s Access to Life-Saving Equipment and Resources.”

“He is rescinding restrictions from the prior administration that limited your agencies’ ability to get equipment through federal programs, including life-saving gear like Kevlar vests and helmets and first-responder and rescue equipment like what they’re using in Texas right now,” Sessions said in the speech.

But that’s not what the Obama administration’s restrictions did, according to documentation from a unit inside of Sessions’ own Justice Department, the Bureau of Justice Assistance.

Kevlar vests were never subject to any restrictions. Most helmets weren’t, either. Riot helmets (defined as those with shields over the face), Humvees and helicopters that are sometimes used in rescue missions, were still available to police forces as long as they explained why they needed them and certified that they had protocols and training in place so officers would use them safely. That requirement was dropped for riot helmets last October.

“Kevlar vests were never on any lists. That part is simply lying about what we did,” said Roy Austin, who worked on the Obama policy as a deputy assistant to the president for the Office of Urban Affairs, Justice and Opportunity. “He was being untruthful about helmets as well.”

A Justice Department spokesman acknowledged that the items Sessions cited were never prohibited by the Obama administration and that only some of them were even subject to additional procedures. Still, the spokesman, who did not want to be named, said: “There is absolutely nothing misleading about what the Attorney General said.”

What the Obama administration did actually prohibit were tanks, weaponized vehicles, .50 caliber guns, grenade launchers, bayonets and digital-pattern camouflage uniforms. Those restrictions applied only to purchases using federal dollars through a Pentagon surplus program. Police departments remained free to buy them with state, local or private funds.

“The attorney general and whoever advised him on these policies didn’t read them carefully and didn’t understand what they actually did,” said Ed Chung, who worked on the Obama administration policies in the Justice Department’s Office of Justice Programs. “The safety of officers was never jeopardized because the overwhelming majority of equipment, including the ones cited by the attorney general, were still available to law enforcement.”

The Obama administration implemented the restrictions in the spring of 2016 in response to public concerns over the display of military-style hardware deployed by police to control riots in Ferguson, Missouri, after an officer fatally shot Michael Brown. Chung said the administration worked with civil rights organizations and police groups in an effort to meet law enforcement agencies’ needs while improving their relations with the communities they protect.

Sessions, in his speech, harshly dismissed such considerations. “We will not put superficial concerns above public safety,” he said.

The attorney general asserted that the types of equipment limited by the Obama administration saved an officer from a bullet at the Pulse nightclub shooting in Orlando and helped police pursue the shooters in San Bernardino. But that’s not accurate, either. As shown in the very articles that Justice Department officials provided to reporters to support Sessions’ speech, the Orlando officer’s helmet that blocked the bullet wasn’t a riot helmet, so it never faced any restrictions. And the armored vehicles used in San Bernardino had wheels, so the Obama policy still permitted them, with the additional precautions described above. The Obama administration only prohibited buying tanks (armored vehicles that run on tracks instead of wheels) using federal resources.

Sessions shared an anecdote in which a sheriff told him that the Obama administration “made his department return an armored vehicle that can change the dynamics of an active shooter situation.” The Obama administration did ask those few departments that had tanks to return them, according to Chung and Austin, but offered to replace them with equivalent wheeled vehicles.

“For every tracked vehicle, we replaced it with a wheeled vehicle so they didn’t lose anything,” Austin said. “His anecdote makes no sense because it skips the whole part that if they really needed this thing, all they had to do was say we want a replacement and we’d provide a replacement.”

Sessions’ anecdote referred to Sheriff Mike Bouchard of Oakland County, Michigan, who is also vice president of government affairs for the advocacy group Major County Sheriffs of America. In an interview, Bouchard confirmed that the Obama administration replaced his tracked armored personnel carrier with a wheeled one, but he said the wheeled version wasn’t as useful on the sandy and marshy terrain he has in his county.

The Justice Department spokesman said the Obama administration’s additional measures were onerous because departments had to demonstrate a “clear and persuasive” need for the equipment, certify approval from a civilian governing body and keep records of incidents when the equipment was used.

By repealing the Obama administration’s policy, the Trump administration also abolished reporting procedures to help the government keep better track of the equipment it distributes. The Government Accountability Office recently tested the controls in the same program by creating a fake agency that was able to obtain $1.2 million worth of night-vision goggles, and simulated pipe bombs and other potentially lethal items.

“We viewed the Obama reforms as very modest,” said Kanya Bennett, a lobbyist for the American Civil Liberties Union. “We’re certainly disappointed not only by the action that the administration took on Monday but again in suggesting to the public that the previous administration had taken away critical tools for law enforcement. In reality, there was very little that the Obama administration did with respect to taking military weapons and equipment out of rotation for state and local law enforcement use.”

Sessions’ speech cited studies showing that such equipment reduces crime, assaults on officers and complaints against them. In the materials provided to reporters, Justice Department officials pointed to a pair of studies in the American Economic Journal. But those studies considered all equipment supplied through the Pentagon’s surplus program, without distinguishing the items the Obama administration restricted. They also studied the years 2006 to 2012, before the Obama policies took effect.

“Our findings do not necessarily mean that saturating our local law enforcement agencies with military hardware is good policy,” researchers at the University of Tennessee wrote in one of the papers. “These results should not be used to diminish concerns about police-community relations, the role of police in our society, violence against civilians by police, or vice versa.”

The other study observed that the type of equipment that had the biggest effect on reducing crime was nonlethal non-military gear, such as computers and office supplies. The researchers suspect that’s because those supplies free up time and money for officers to spend on policing.

“There is a tendency to conflate military equipment with weapons,” the lead researcher, Vincenzo Bove at the University of Warwick of England, told ProPublica in an email. “Whereas we find that weapons is an unproductive category (it does not significantly affect crime) the miscellaneous category has the strongest effect, followed by vehicles and gear.”

“We do not explore the effect on crime of the items prohibited under the Obama policy,” Bove added. “But, as most of the prohibited items were weapons, note they do not seem to have (in aggregate) any effect on crime.”

Sessions announced the change of policy at a meeting of the Fraternal Order of Police, which viewed repealing the Obama-era restrictions as one of its top-priority asks for the Trump administration. “The previous administration was more concerned about the image of law enforcement being too ‘militarized’ than they were about our safety,” the organization’s president, Chuck Canterbury, said in a statement.

But not all police groups agree. The Obama policy was sensible to restrict non-essential lethal equipment like bayonets and to encourage police to consult local elected officials before acquiring equipment that could be controversial, said Jim Bueermann, president of the Police Foundation and a retired police chief in Redlands, California.

“The Obama executive order represented a best practice for acquiring military surplus equipment for local policing, and I still think it could serve and should serve as a best practice even though the rules have changed,” he said. “I was supportive of the overarching goals of the previous executive order, but I also understand what this president is trying to do. I view this through the lens of a police chief, and I would say to my city manager and council, ‘Ignore all of this that’s the federal government’s activity — under both sets of rules, we should still go to the public to explain our rationale.’”

Even before the demonstration in Virginia began last weekend, the police there knew they weren’t going to be able to handle what was coming.

Charlottesville police officers, including Sgt. Jake Via of the investigations bureau, had been contacting organizers and scanning social media to figure out how many demonstrators were headed their way and whether they would be armed.

“The number each group was saying was just building and building,” Via said. “We saw it coming. … Looking at this, I said, ‘This is going to be bad.’”

The protesters’ numbers were too large and the downtown park too small. City officials tried to get the demonstration moved to another, more spacious location, but lost in court after the rally’s organizer, backed by the American Civil Liberties Union, alleged his freedom of speech was being infringed.

The protests, of course, ended tragically. Local law enforcement was widely blamed for losing control of the event and standing back even as people were attacked.

Via maintains that nothing the police did could have stopped the violence between the two sides. “No hours and hours and hours or even months of planning is going to stop the radicals from both sides wanting to go at it,” he said.

With more demonstrations planned in cities across the country, ProPublica interviewed law enforcement experts in the United States and Europe to ask what more can be done to prevent bloodshed at protests where people are spoiling for a fight. The consensus was that additional steps can be taken.

But many of the tactics come at a price. Some could be viewed as impinging on civil liberties and the constitutionally enshrined rights to free assembly and protest. Others require funding and coordination that is difficult to achieve within the fragmented framework of American policing. A few are as simple as strategically placed blockades that keep the two sides separate. Here are some of the top approaches and how they might — or might not — be deployed in the U.S.

Drones, Anti-Mask Laws and Open-Carry Restrictions

Local police forces will increasingly institutionalize the use of drones at mass demonstrations. That’s the prediction of Brian Levin, a criminal justice professor at California State University, San Bernardino. Cameras in the air with real-time feeds transmitted to officers on the ground would allow police to cover more terrain and in some cases, identify potential conflicts before they erupt.

“Demonstrations spread, and these violent confrontations can take place in disparate areas,” said Levin. “It’s like when a hammer hits mercury.”

Drones can also be safer than helicopters. In Charlottesville, a helicopter monitoring the demonstration went down, killing two state troopers aboard.

But police drone use has been met with opposition from civil liberties groups. Drones donated to the Los Angeles Police Department have gone unused for years amid privacy concerns. Activists have argued that access to the devices, which make surveillance cheaper and more efficient, will lead police to more routinely surveil private citizens.

Earlier this month, LA’s police commission gathered to discuss relaunching the program only to be met by chanting activists who shut down the conversation twice. Similar stories have played out in Seattle and elsewhere.

Another tool cities and states (including Virginia) have used is anti-mask laws, which bar groups of people from disguising themselves in public. Violent demonstrators will sometimes arrive in ski masks or scarves wrapped around their faces. New York City has a ban, with exceptions including for Halloween. So does Alabama, a rule it instituted in 1949 to unmask the Ku Klux Klan. A similar restriction in California, though, was struck down after Iranian Americans hoping to safely (and non-violently) protest the post-revolutionary regime back home sued on First Amendment grounds.

Another challenge in Charlottesville was the number of demonstrators who came with guns, and were allowed to do so lawfully, because of Virginia’s open-carry laws.

Even in states with such statutes, the authorities have some options. Erwin Chemerinsky, dean of the University of California, Berkeley’s law school, said the Supreme Court has upheld the right to have guns at home, but not necessarily in public. “Think of curfews. The government has the ability to take steps to protect public safety,” Chemerinsky said. “The more evidence there is that it’s a threat to public safety, the more sympathetic the courts would be.”

The evidence could consist of past rallies that broke out into violence, or intelligence that an armed group is planning to employ force in the future.

Still, attempts to temporarily restrict gun rights have floundered in the past. Before the most recent Republican National Convention in Cleveland, the head of Cleveland’s largest police union and others called for the state’s open-carry laws to be tightened during the convention. Gov. John Kasich refused, saying “Ohio governors do not have the power to arbitrarily suspend federal and state constitutional rights.”

A more radical approach comes from Philip Zelikow, a history professor at the University of Virginia and former executive director of the 9/11 Commission. In 1981, he worked with the Southern Poverty Law Center, a civil rights nonprofit, to ask a federal judge to shut down a group called the Knights of the Ku Klux Klan, which was showing up armed and in Army-style clothing on the Texas Gulf Coast to harass Vietnamese fishermen.

With the support of the Texas attorney general, Zelikow and his team invoked a 19th-century law that forbids “military companies” not authorized by the governor. They argued that the Klan qualified because it was not government-regulated but had “command structure, training and discipline so as to function as a combat or combat support unit.” The lawyers prevailed, and the Klan was forced to leave its weapons at home.

A similar argument also succeeded soon after in North Carolina, and Zelikow said groups like those in Charlottesville that are mixing weaponry and political activism could be subject to similar legal challenges. “These problems haven’t come up much in recent decades,” Zelikow said. “The issue subsided and memory fades but here we are again.”

Most states have restrictions on private military-like groups. Zelikow was contacted by lawyers from Oklahoma this week, asking if their state had such a law on the books. “It took me about five minutes to find,” he said. Zelikow is now trying to form a team of lawyers to bring a case in Virginia.

Looking Abroad

Thousands of people, divided into two opposing sides, squaring off in public. Some come armed, looking to damage property and wreak havoc. Many filter in from out of town, complicating efforts by police to negotiate peace in advance.

It’s a scenario European authorities know well, though with a different kind of group: soccer hooligans.

Maria Haberfeld, a professor at the John Jay College of Criminal Justice in New York and a former supervisor in the Israel National Police, said cops in the Netherlands use a “situation-oriented” model to keep violent rival soccer fans under control.

That framework trains officers in perceptual skills, helping them develop the emotional intelligence to read members of crowds and make sound judgments about which situations are truly dangerous. Officers are put through simulations in which they can achieve positive, nonfatal outcomes. Trainings are handled in groups, not individually, so that in the field, officers are less likely to misinterpret any of their colleagues’ motions or actions.

“European police forces are light years ahead of us in terms of training.” Haberfeld said. “It’s not something you can train police officers to do in half an hour. It’s a serious commitment.”

Reaching that level of training may not be feasible in the U.S., where local municipalities set their own academy protocols. (And demonstrations are less frequent than soccer matches.) The training in the U.S. typically lasts just a few months, compared to the couple of years that European police cadets get.

In Germany, police forces commonly have specialized units assigned to each side of a potentially violent protest. Officers meet with the groups’ leaders in advance and discuss plans for the protest in detail, including symbols that are forbidden for display by the government.

Protest leaders can be denied permits to demonstrate because of criminal records, forcing them to turn leadership of the event over to another member of the group. They’re also asked to assign deputies from within their organization who can help the group’s leader keep things under control. Those assistants also have their records vetted by the police before being approved.

Once at the event, the specialized police units show up in distinctive yellow vests, and without riot gear, so they can mix in with the demonstrators less threateningly. When officers see someone with a banned weapon, they sometimes will only film the demonstrator and make an arrest later.

“It is important for us, is not to have a negative solidarity spillover effect. … If we disarm a person or act against a small group of potentially violent protestors, other people around solidarize with them against the police,” said Elke Heilig, head of the anti-conflict team in Pforzheim, Germany. “This leads towards escalation.”

Keeping Peaceful Protesters Away

Social media gives hate groups a new megaphone for getting the word out about their rallies, opening up communication with many previously fragmented niche groups and helping lead to larger gatherings, experts said. A big crowd is inherently harder to police, but what makes the scenario even more vexing for law enforcement is that they’re now dealing with not just one or two groups, but many, along with unaffiliated individuals.

“People are coming in from disparate places and disparate groups who don’t answer to any single authority. A Klan leader can tell his folks to stand down,” said Levin, a former NYPD officer. “Social media has been a magnet not only for haters but for unstable haters.”

Some municipalities are responding by using social media tools to dissuade some activists from showing up. City officials in Berkeley, California, have experimented with discouraging peaceful protesters from attending demonstrations they expect to be violent.

In March, fights broke out between supporters and opponents of the president at a demonstration near the Berkeley campus. Some of the unruly counter-protesters were believed to be affiliated with black bloc, an anarchist group whose members are known to wear black and mask their faces. Mayhem ensued. In one case, a man wearing a “Trump is My President” shirt had his face bloodied.

“There are people who come intent on committing violence and they look for ways to subvert whatever you set up,” said city spokesman Matthai Chakko. “There are people who use peaceful protesters as shields. They blend into crowds after they commit their acts.”

In April, before another planned demonstration, the city launched a messaging campaign suggesting peaceful protesters keep their distance. “Consider whether the approach others advertise is the style and venue for you,” one alert read, warning of violent protesters. “Reaching out to organizations or individuals in need is an alternative to conflict. When people at an event act in a way that compromises your values and goals, separate yourself.”

The number of peaceful protesters dropped significantly, Chakko said, and the city is taking a similar approach with an unpermitted, white nationalist demonstration expected later this month. The alert the city sent out Wednesday was direct: “The best response for those seeking to safeguard our community is to stay away.”

Barriers and Chain-Link Fences

Miriam Krinsky, a former federal prosecutor who has worked on police reform efforts in Los Angeles, said the most fundamental strategy for dueling demonstrations is keeping the two sides separate, with physical obstacles and police in between. “Create a human barrier so the flash points are reduced as quickly as possible,” she said.

Law enforcement will sometimes quarantine protesting hate groups inside concentric chain link fences, creating a large empty space between opposing groups. Those entering the inner ring are sent through metal detectors.

At an anti-Sharia protest in San Bernardino, California earlier this year, the two groups were kept on opposite sides of the street, with horse-mounted cops there to prevent protesters from crossing over.

The lack of space to separate the factions was a widely noted problem in Charlottesville. The massive demonstration was allowed to take place inside a small downtown park, making it more difficult for police to insert themselves and separate the two sides. “The two groups are both trying to occupy the same area and this doesn’t give police a lot of maneuverability,” said John Kleinig, professor emeritus at John Jay College of Criminal Justice.

Demonstrators ended up spilling out beyond the park, and one counter-protester was killed when an Ohio man allegedly plowed his car into a crowd a few blocks away from the park.

Demonstrations can in some ways be easier to control in concentrated urban areas, where police use tall buildings with little or no space in between them as barriers. And smaller city police forces generally have less training in large crowd control.

“I’m former NYPD,” Levin said. “We had grid patterns and streets we could block off, put a wedge in when we had an unruly crowds. … You have people hemmed in by structures and street grid patterns. In smaller places, people can spread out in all different directions.”

Since the weekend, amid criticism of their handling of the demonstration, Charlottesville Police Chief Al Thomas acknowledged that crowd’s spread led to problems.

“We had to actually send out forces to multiple locations to deal with a number of disturbances,” he said. “It was certainly a challenge. We were spread thin once the groups dispersed.”

Special correspondent Pia Dangelmayer in Germany contributed to this story.

The USDA’s research section studies everything from climate change to nutrition. Under the 2008 Farm Bill, its leader is supposed to serve as the agency’s “chief scientist” and be chosen “from among distinguished scientists with specialized or significant experience in agricultural research, education, and economics.”

But Sam Clovis — who, according to sources with knowledge of the appointment and members of the agriculture trade press, is President Trump’s pick to oversee the section — appears to have no such credentials.

Clovis has never taken a graduate course in science and is openly skeptical of climate change. While he has a doctorate in public administration and was a tenured professor of business and public policy at Morningside College for 10 years, he has published almost no academic work.

Clovis is better known for hosting a conservative talk radio show in his native Iowa and, after mounting an unsuccessful run for Senate in 2014, becoming a fiery pro-Trump advocate on television.

Clovis advised Trump on agricultural issues during his presidential campaign and is currently the senior White House adviser within the USDA, a position described by The Washington Post as “Trump’s eyes and ears” at the agency.

Clovis was also responsible for recruiting Carter Page, whose ties to Russia have become the subject of intense speculation and scrutiny, as a Trump foreign policy adviser.

Neither Clovis, nor the USDA, nor the White House responded to questions about Clovis’ nomination to be the USDA’s undersecretary for research, education and economics.

Catherine Woteki, who served as undersecretary for research, education and economics in the Obama administration, compared the move to appointing someone without a medical background to lead the National Institutes of Health. The USDA post includes overseeing scientific integrity within the agency.

“This position is the chief scientist of the Department of Agriculture. It should be a person who evaluates the scientific body of evidence and moves appropriately from there,” she said in an interview.

Woteki holds a Ph.D. in human nutrition and served as the first undersecretary for food safety at the USDA during the Clinton administration. She was then the dean of the school of agriculture at Iowa State University before becoming the global director of scientific affairs for Mars, Inc.

Clovis has a B.S. in political science from the U.S. Air Force Academy, an MBA from Golden State University and a doctorate in public administration from the University of Alabama. The University of Alabama canceled the program the year after Clovis graduated, but an old course catalogue provided by the university does not indicate the program required any science courses.

Clovis’ published works do not appear to include any scientific papers. His 2006 dissertation concerned federalism and homeland security preparation, and a search for academic research published by Clovis turned up a handful of journal articles, all related to national security and terrorism.

As undersecretary for research, education and economics, Woteki directed additional resources to helping local farmers and agricultural workers address the impacts of severe drought, flooding and unpredictable weather patterns. She chaired the “Global Research Alliance to Reduce Agricultural Greenhouse Gasses,” which brings together chief agricultural scientists from across the globe. Under her leadership, the USDA also created “Climate Hubs” across the country to help localized solutions for adapting to climate change.

Clovis has repeatedly expressed skepticism over climate science and has called efforts to address climate change “simply a mechanism for transferring wealth from one group of people to another.” He has indicated the Trump administration will take a starkly different approach at the USDA. Representing the campaign at the Farm Foundation Forum in October, Clovis told E&E News that Trump’s agriculture policy would focus on boosting trade and lessening regulation and not the impact of climate change.

“I think our position is very clearly [that] Mr. Trump is a skeptic on climate change, and we need more science,” he said. “Once we get more science, we are going to make decisions.”

The USDA’s undersecretary for research, education and economics has historically consulted on a wide range of scientific issues. Woteki, for example, said she was asked for input on the Zika and Ebola outbreaks because of the USDA’s relevant research and was frequently called upon to offer guidance on homeland security issues related to food safety.

“Access to safe food and clean air and water is absolutely fundamental to personal security,” she said, adding that a scientific understanding of food safety is critical to success in the job. “Food systems are widely recognized by the national security community as being part of critical infrastructure.”

Clovis’ academic background includes years of study on homeland security, but focused almost exclusively on foreign policy. A biography he provided to the 2016 Fiscal Summit at which he was a speaker indicates he is “a federalism scholar” and “an expert on homeland security issues,” with “regional expertise in Europe, the former Soviet Union, and the Middle East.” Neither this biography nor any other publicly available biographies list any experience in food safety, agriculture or nutrition.

Clovis first became well-known in Iowa through his radio show, “Impact with Sam Clovis.” He finished a distant second in the 2014 Republican Primary for an Iowa Senate seat ultimately won by Joni Ernst. During the race, his outlandish statements often made headlines. In one instance, he said the only reason President Obama hadn’t yet been impeached was because of his race.

While he initially signed on as former Texas Gov. Rick Perry’s top Iowa adviser, he left in August 2015 to become the Trump campaign’s national co-chair and chief policy adviser. Emails leaked by the Perry campaign to The Des Moines Register show Clovis slamming Trump in the months before, questioning his faith. “His comments reveal no foundation in Christ, which is a big deal,” Clovis wrote. He also praised Perry for calling Trump a “cancer on conservatism.”

Still, Clovis subsequently became one of Trump’s best-known advocates on cable television, where he relentlessly defended his new boss. On “Morning Joe,” he said Democratic presidential nominee Hillary Clinton had failed to “control the sexual predation that went on in her own home.” On SiriusXM, he said Republicans who were abandoning Trump were “weak-kneed” and “lily-livered.”

Trump’s call for a “total and complete shutdown” of entry of Muslims into the United States in December 2015 put Clovis’ job as a tenured professor at risk.

“If he played a role in drafting or advising the Trump campaign on this issue, we will be outraged and extremely disappointed in Dr. Clovis,” Morningside College spokesman Rick Wollman told Iowa Starting Line, before pledging to look “more closely” at the issue.

Clovis went on unpaid leave from the college in the summer of 2015 and resigned after Trump’s win in November.

Do you have information about this case or other Trump administration appointees? Contact Jessica Huseman at jessica.huseman@propublica.org or via Signal at 972-268-1882.

One day recently, we visited Amazon’s website in search of the best deal on Loctite super glue, the essential home repair tool for fixing everything from broken eyeglass frames to shattered ceramics.

In an instant, Amazon’s software sifted through dozens of combinations of price and shipping, some of which were cheaper than what one might find at a local store. TheHardwareCity.com, an online retailer from Farmers Branch, Texas, with a 95 percent customer satisfaction rating, was selling Loctite for $6.75 with free shipping. Fat Boy Tools of Massillon, Ohio, a competitor with a similar customer rating was nearly as cheap: $7.27 with free shipping.

The computer program brushed aside those offers, instead selecting the vial of glue sold by Amazon itself for slightly more, $7.80. This seemed like a plausible choice until another click of the mouse revealed shipping costs of $6.51. That brought the total cost, before taxes, to $14.31, or nearly double the price Amazon had listed on the initial page.

What kind of sophisticated shopping algorithm steers customers to a product that costs so much more than seemingly comparable alternatives?

One that substantially favors Amazon and sellers it charges for services, an examination by ProPublica found.

Amazon often says it seeks to be “Earth’s most customer-centric company.” Jeffrey P. Bezos, its founder and CEO, has been known to put an empty chair in meetings to remind employees of the need to focus on the customer. But in fact, the company appears to be using its market power and proprietary algorithm to advantage itself at the expense of sellers and many customers.

Unseen and almost wholly unregulated, algorithms play an increasingly important role in broad swaths of American life. They figure in decisions large and small, from whether a person qualifies for a mortgage to the sentence someone convicted of a crime might serve. The weightings and variables that underlie these equations are often closely guarded secrets known only to people at the companies that design and use them.

But while the math is hidden from public view, the effects of algorithms can be vast. With more than 300 million active customer accounts and more than $100 billion in annual revenue, Amazon is a shopping giant whose algorithm can make or break other retailers. And so ProPublica set out to see how Amazon’s software was shaping the marketplace.

We looked at 250 frequently purchased products over several weeks to see which ones were selected for the most prominent placement on Amazon’s virtual shelves — the so-called “buy box” that pops up first as a suggested purchase. About three-quarters of the time, Amazon placed its own products and those of companies that pay for its services in that position even when there were substantially cheaper offers available from others.

That turns out to be an important edge. Most Amazon shoppers end up clicking “add to cart” for the offer highlighted in the buy box. “It’s the most valuable small button on the Internet today,” said Shmuli Goldberg, an Israeli technologist who has extensively studied Amazon’s algorithm.

Amazon does give customers a chance to comparison shop, with a listing that ranks all vendors of the same item by “price + shipping.” It appears to be the epitome of Amazon’s customer-centric approach. But there, too, the company gives itself an oft-decisive advantage. Its rankings omit shipping costs only for its own products and those sold by companies that pay Amazon for its services.

We found that the practice earned Amazon-linked products higher rankings in more than 80 percent of cases. Amazon’s offer of the Loctite glue, a respectable No. 5 on the comparison list, dropped to the 39th best deal when shipping was included. (The prices Amazon shows are ranked correctly for those who pay $99 per year for Amazon’s Prime shipping service and for those who are buying $49 or more in eligible items.)

Erik Fairleigh, a spokesman for Amazon, said the algorithm that selects which product goes into the “buy box” accounts for a range of factors beyond price. “Customers trust Amazon to have great prices, but that’s not all— vast selection, world-class customer service and fast, free delivery are critically important,” he said in an e-mailed statement. “These components, and more, determine our product listings.” (Read Amazon’s original statement and the statement Amazon sent after this story was published.)

Fairleigh declined to answer detailed questions, including questions about why Amazon’s product rankings excluded shipping costs only for itself and its paid partners.

The decision to allow non-Amazon companies to sell products on the site was controversial within the company. But Bezos pushed ahead, saying he was willing to lose sales if it made his company more competitive in the long run. “If we side with the consumer on that kind of decision,” he said at the 2007 annual shareholder meeting, “over time it will force the right kind of behaviors on ourself.”

At that meeting, a shareholder asked about Amazon’s practice of promoting products sold by other companies on its website. Bezos replied that the company had “very objective customer-centered algorithms‘’ that automatically award the “buy box” to the lowest price seller, provided “they actually have it in stock and can deliver it.”

It is not clear why Amazon’s algorithm now pushes its own products ahead of better deals offered by others. Perhaps Amazon is taking the view that its widely admired shipping and delivery offers the best possible satisfaction for customers, even if it costs more.

Another possibility is that the company is trying to encourage shoppers to join the Prime program, which offers free shipping on many items (including the Loctite super glue). When non-Prime customers initially view Amazon products, they are offered “FREE Shipping on eligible orders.” When they reach the final page on checkout, the shipping fees are revealed along with an advertisement to avoid such fees by joining Prime.

The costs of simply buying the algorithm-selected choice can add up. The average price difference between what the program recommended and the truly cheapest price was $7.88 for the 250 products we tested. An Amazon customer who bought all the products on our list from the buy box would have paid nearly 20 percent more — or about $1,400 extra — than if they had bought the cheapest items being offered by other vendors.

Amazon’s algorithm also takes a toll on outside companies hoping to sell products on the website. To increase their chances of winning the buy box, many sellers are paying Amazon to warehouse and ship their products through a program called “Fulfilled by Amazon.” The fees for the program, which vary by size and weight of the items being shipped, can amount to 10 to 20 percent of sales.

Paying Amazon appears to be a sound strategy. Fulfilled by Amazon vendors and Amazon itself were just about the only sellers — 94 percent of the cases we analyzed — that ever won the buy box without having the cheapest product.

Through its rankings and algorithm, Amazon is quietly reshaping online commerce almost as dramatically as it reshaped offline commerce when it burst onto the scene more than 20 years ago. Just as the company’s cheap prices and fast shipping caused a seismic shift in retailing that shuttered stores selling books, electronics and music, now Amazon’s pay-to-play culture is forcing online sellers to choose between paying hefty fees or leaving the platform altogether.

Consider BareBones WorkWear, a Sacramento clothing retailer that has been selling on Amazon since 2004. This year, the company removed nearly all of its items from Amazon, and shuttered a warehouse and call center that were devoted to Amazon sales.

“Competition between us and Amazon is just insurmountable,” BareBones chief operating officer Mason Moore said. The profit margins for most clothing items were too low, he said, to allow for the company to sell through the Fulfilled by Amazon, or FBA, program. But, he said, “FBA is really the only avenue that we see as any feasible way to do business with Amazon.” This week, BareBones has just five items listed on Amazon — all of them fulfilled by Amazon.

Last Christmas, so many vendors joined Fulfilled by Amazon that the company ran out of space in some of its warehouses. This year, the company has doubled its number of warehouses.

In July, Amazon reported record profits and the company’s chief financial officer Brian Olsavsky told investors that Fulfilled by Amazon growth was “really strong.”

Tech companies’ practice of favoring their own listings has occasionally earned regulators’ scrutiny. The European Commission, for example, has accused Google of violating EU antitrust rules by favoring its own shopping service over those of other vendors.

---

Amazon didn’t start as an open marketplace for online sellers. When it opened its virtual doors in 1995, Amazon sold only its own products. It began letting other vendors onto its product listings in 2000.

“Our judgment was simple,” Bezos wrote of that decision, in a 2005 letter to shareholders. “If a third party could offer a better price or better availability on a particular item, then we wanted our customer to get easy access to that offer.”

For merchants, listing their wares on Amazon was a great opportunity. Amazon attracted legions of customers, well worth the 6 to 25 percent commission merchants paid the online colossus on each sale.

By 2007, more than 1 million third-party sellers had joined Amazon. Collectively, they generated about 30 percent of unit sales, the company said at the time.

One of the sellers Amazon attracted was Kate Erkavun. She had recently received a master’s degree in industrial engineering, but also sold cosmetics on eBay out of her apartment while her husband worked at an engineering company.

“I was young, I had just graduated,” she said. “I worked in a pharmaceutical company and I quit after seven months. I realized it wasn’t for me.” She started selling makeup on eBay, and then migrated to Amazon, too.

At first, Erkavun’s listings on Amazon were similar to her eBay listings — one page for each product that customers could page through at leisure. But over time, Amazon simplified the design so that customers would only be presented with one default vendor for each product.

Amazon’s algorithm would choose which seller would win that default position — the buy box. While the exact formulas used to pick the winner were secret, Amazon’s website advises sellers that they can increase their chances by having low prices, having items in stock, offering free shipping and getting excellent customer service ratings.

To optimize their chances, many sellers starting using algorithmic software to constantly change prices to adapt to competitors’ moves. Soon, Amazon became a highly dynamic marketplace, similar to a stock-trading floor, where prices for products changed as often as every 15 minutes.

Erkavun and her husband, Gokhan, were determined to increase their chances of sales. Gokhan, who quit his job to help run the business, worked with a programmer to write software that repriced their products throughout the day. And the couple bought a building in Nutley, New Jersey, to store their approximately 10,000 shampoos, lipsticks, lotions and other cosmetics in a temperature-controlled warehouse. They offered free shipping, quick turnaround times and worked hard to keep their customer service ratings high.

At first, their techniques seemed to be successful. In 2010, Amazon sales were half of their revenue and profits were at a record high. “We were very happy,” said Gokhan.

But in 2011, he said, Beauty Bridge’s sales started slipping as Amazon entered the cosmetics business and began consistently winning the buy box. “If you don’t win the buy box, your chance of selling is low,” Kate said.

Sellers who don’t win the buy box are placed on a page called “More Buying Choices,” on a list that Amazon describes as ranked by price plus shipping. However, since Amazon doesn’t include the cost of shipping for itself and its fulfillment partners, the rankings on that page can be misleading.

One day recently, for instance, Amazon was listed as the top-ranked seller — both in the buy box and at the top of the buying choices page — for a self-tanning lotion called Vita Liberata. Beauty Bridge was offering the lowest price at $27.03, but Amazon had won the buy box with an offer of $29.98.

When a customer put the lotion from Amazon in her cart, the added shipping cost brought the total to $35.46. Beauty Bridge was offering free shipping — so with or without shipping, its offer should have been listed higher than Amazon’s. But it was not.

When Gokhan Erkavun was told of his lotion’s poor ranking despite its cheaper price, he just sighed and said, “Amazon is not really fair in terms of competition, but we don’t have much choice. We have to be there.”

On its Canadian website, Amazon discloses that its own items are ranked without shipping price. But in the United States, Amazon’s website states that “the default sort order of the offer listing is ascending Price + Shipping.”

Of course, most Amazon customers never make it to the More Buying Choices page where Beauty Bridge’s listing was ranked poorly. Among the countless consultants and conferences devoted to winning the buy box, it’s well known that Amazon’s algorithm gives an advantage to itself, and to sellers who pay to join the Fulfilled by Amazon program.

“Amazon definitely does weight things in favor of the FBA seller,” said Michael Butcher, senior account manager at SellerEngine Software, which sells algorithmic pricing software for Amazon sellers. “It does seem unfair and it is sometimes hard for merchants.”

For a few years, Amazon even advertised the advantage it offered to its paid partners. According to Web pages stored by the Internet Archive, the Amazon website said: “Because most FBA listings are ranked without a shipping cost, you get an edge when competing!” The language remained on the page from February 2013 through December 2015.

This year, the language has been changed to: “As you grow your competitive edge, you can increase your chance of winning the Buy Box.”

But Beauty Bridge’s Kate and Gokhan Erkavun didn’t want to pay the fees to join the program. They had their own warehouse and didn’t need Amazon’s. And they estimated the program would cost them at least an additional 15 percent of sales.

They held out until 2014. By then, sales had slid 30 percent from the peak in 2010. In 2014, “we got to a point where we couldn’t survive without doing FBA,” Gokhan said. Since joining the “Fulfilled by Amazon” program, Gokhan says the company’s sales have recovered, but profits have not because of the fees.

Gokhan is now hoping that Wal-Mart’s recent purchase of online shopping website Jet.com will increase the pressure on Amazon to give small online retailers a better deal.

“We need Wal-Mart to really get serious about competing with Amazon,” he said. “Otherwise in 10 years, we aren’t going to have many retailers left.”

Lauren Kirchner and Jeff Larson contributed research to this story.

Update, Sept. 20, 2016: This story has been updated to include a statement Amazon sent after publication.

After assembling suicide bomb vests for the attacks that slaughtered 130 people in Paris last November, Najim Laachroui went underground in his native Brussels.

The 24-year-old explosives expert wasn’t just hiding from the biggest manhunt in Europe’s recent history. He was plotting. In a dingy apartment converted into a bomb factory, Laachroui exchanged a series of messages in French with Abu Ahmed, a shadowy commander in the Islamic State based in Syria.

If law enforcement agencies had intercepted the communications, they would have been immediately alarmed. Laachroui asked militants in Syria to test chemical mixtures so he could assemble powerful bombs. He discussed his hopes to strike France again and disrupt a soccer championship there. He reported that he and half a dozen other fugitives from the Paris attacks had split up among three safe houses, according to Belgian and French counterterror officials.

Although U.S and European spy agencies were scouring the internet for any trace of Laachroui, they failed to intercept those exchanges. The reason, U.S. and European counterterror officials say: during Laachroui’s four months on the run, he and Abu Ahmed communicated through Telegram, an encrypted messaging application, and other widely available tools for secure communications.

On March 15, Belgian police raided a safe house and killed another leader of the terrorist cell in a gunfight. A worried Laachroui sent a message to Abu Ahmed reporting that the raid had cost the plotters the stash of ammunition for their AK-47 rifles, according to Belgian and French counterterror officials.

“The original plan at the airport was for them to do an attack more like Paris: shoot a lot of people first, and then set off the bombs,” a Belgian counterterror official explained in an interview in April. “But they didn’t have ammo because it was left behind in the safe house. Laachroui says: ‘We don’t have chargers for our guns. What do we do?’ They were told to go ahead and attack just with bombs’.”

On March 22, they did just that. Laachroui and two other suicide bombers killed 32 people at the airport and at a subway station in Brussels. Afterward, investigators found a laptop computer that helped them reconstruct Laachroui’s encrypted audio and text exchanges with his commander in Syria, according to European counterterror officials.

The communications were described by European and U.S. counterterror officials to ProPublica, which is preparing a documentary about terrorism in Europe in collaboration with the PBS program Frontline. ProPublica interviewed counterterror officials in Europe and the United States, some on condition of anonymity, and reviewed intercepted conversations documented in European court cases.

Taken together, the voices of the Islamic State offer insights into the day-to-day workings of an organization that has carried out lethal attacks in Baghdad, Bangladesh, and Turkey in the past two weeks.

The culture of ISIS mixes the centralized control that characterized al-Qaida with a more freewheeling approach that gives its operatives considerable latitude. The group’s use of digital propaganda to inspire “self-radicalizing” terrorists has drawn attention with attacks on U.S. soil in San Bernardino and Orlando. But the communications collected in Europe show the group provides direct long-distance instructions to operatives it dispatches from its base in Syria, and they rely heavily on that guidance.

The European communications also clearly establish the importance of encryption to ISIS operations.

“We are dealing with a challenge right now: New technologies that enable encryption and allow them to be fairly confident that they are communicating in a way that can’t be detected,” a senior U.S. intelligence official said. “They know how to communicate securely. Often we are inhibited: We know the fact of the communications taking place without knowing the content.”

In April, Italian police overheard a senior figure in Syria urging a Moroccan suspect living near Milan to carry out an attack in Italy, according to a transcript. Although the voice message had been sent through an encrypted channel, the Moroccan played it back in his car, where a hidden microphone recorded it.

In the message, the unidentified “sheik” declared: “Detonate your belt in the crowds declaring Allah Akbar! Strike! (Explode!) Like a volcano, shake the infidels, confront the throng of the enemy, roaring like lightning, declare Allah Akbar and blow yourself up, O lion!”

The suspects exchanged recorded messages over WhatsApp, an encrypted telephone application that is widely used in Europe, the Arab world and Latin America. FBI Director James Comey and other counterterror officials have publicly expressed concern about extremists in the United States using such techniques to elude monitoring.

“We’ll be monitoring a couple of guys in an internet chatroom,” a former FBI counterterror official said in an interview. “Then you’ll see one of them says: OK, reach out to me on WhatsApp.’ At that point, we can’t do anything.”

Executives at WhatsApp and Telegram defend encryption as a vital shield to privacy. Reached for comment last week, a spokesperson at WhatsApp said the company complies with U.S. laws requiring cooperation with law enforcement agencies. The spokesperson cited a statement by executives in April when WhatsApp implemented “end-to-end” encryption that will conceal the content of users’ communications even from the company itself.

“Encryption is one of the most important tools governments, companies, and individuals have to promote safety and security in the new digital age,” said Jan Koum, the company’s founder, in a blog post in April. “While we recognize the important work of law enforcement in keeping people safe, efforts to weaken encryption risk exposing people’s information to abuse from cybercriminals, hackers, and rogue states.”

(Facebook, which owns WhatsApp, announced Friday that it would add end-to-end encryption for some photo and text messages on its Messenger application.)

Telegram did not respond to a request for comment for this article last week. But company executives have publicly addressed concerns about encryption by saying that technology comes with an inevitable dark side. The company says it has shut down more than 660 public channels on its application that were being used by the Islamic State.

Intelligence officials say the Islamic State’s failure to launch Paris-style attacks in the United States reflects differences in both geography and demography. American Muslims are less radicalized and less numerous than those in Europe, and U.S. border security makes it harder for would-be terrorists to enter the country, according to Western counterterror officials.

In Europe, the Islamic State has found support in large and restive Muslim communities, especially among criminals who radicalize more rapidly today than previous generations of hoodlums-turned-jihadis. Investigators say intensified Western military pressure in Iraq and Syria has prompted the group to order European recruits to strike immediately rather than make the pilgrimage to the caliphate.

“In the context of the current strategy of the Islamic State, it’s clear that their focus is causing casualties here,” said Claudio Galzerano, commander of a counterterrorism unit of the Italian police.

The ISIS strategy toward the West has evolved since the Islamic State conquered a swath of territory in Syria and Iraq and declared the caliphate two years ago, causing thousands of militants to flock to Syria.

Previous generations of aspiring jihadis passed through a series of filters as they journeyed to al-Qaida training camps in South Asia, often with a first stop at radical mosques in London. This selective, secretive approach allowed al-Qaida to vet prospective holy warriors and detect attempts at infiltration by intelligence services.

In contrast, the flow to Syria has been larger, faster and less security-conscious. Taking advantage of Europe’s proximity and ease of travel, fighters who rushed to Syria posted photos of themselves online brandishing guns. Their ranks included criminals and thrill-seekers with little religious knowledge, according to Marc Trevidic, a veteran French counterterror judge.

The Islamic State “has accepted for strategic reasons, because it wanted to impose itself on other groups in the region, the recruitment of anyone,” Trevidic said. “Methods will be created afterwards to check that … they are not spies, etcetera, but initially there are no filters.”

An Italian investigation begun in 2014 documented that hectic period. Tracking jihadis from Italy, police intercepted the cell phones of senior figures in Turkey and Syria, according to a 44-page report by a Milan investigative magistrate dated June 12, 2015. A Turkish phone was used primarily by Ahmed Abu al Harith, “a significant member of the terrorist organization with the role … of coordinating volunteers arriving in Turkey and headed to join the Islamic State,” the report says.

Monitored in late 2014 and early 2015, Ahmed Abu al Harith and fellow coordinators spoke multiple languages with callers from 22 countries including Afghanistan, Saudi Arabia, Georgia, San Marino and Sweden.

They explained “concrete rules for joining the Islamic State already described on the internet in a manual titled ‘Hijrah [Pilgrimage] to the Islamic State 2014 what to bring, whom to contact, where to go’,” according to the Italian report.

The coordinators didn’t use encryption or coded language. But they banned recruits from traveling with “latest-generation mobile devices” in order to “avoid being located” by spy agencies. The Islamic State wanted recruits to leave behind smartphones, tablets and other devices with existing trails of activity that make them easier to trace, and use new, disposable cell phones instead, the report says.

Some recruits were less than sophisticated about the directives.

On Jan. 4, 2015, an exasperated coordinator repeatedly explained to a befuddled caller with a Lebanese accent that he could only bring a basic cell phone to Syria, according to a transcript.

“The important thing is that when you arrive in Turkey you have a small cell phone to contact me,” the coordinator said. “Don’t bring smart phones or tablets. OK, brother?”

For the fourth time, the recruit asked: “So we can’t have cell phones?”

“Brother, I said smart phones: iPhone, Galaxy, laptop, tablet, etcetera.”

Sounding a bit like a frustrated gate agent at a crowded airport, the coordinator added: “Each of you can only bring one suitcase. If you come alone, just bring one suitcase. That is, a carry-on and one suitcase.”

“I didn’t understand the last thing, could you explain?”

“Brother, call me when you get to Turkey.”

In Syria, new arrivals were interviewed by Islamic State militants seated at computers, according to Western counterterror officials. The militants asked a checklist of questions including blood type, mother’s name, level of religious education, and preference for becoming a “warrior” or a “martyr,” according to copies of Islamic State intake forms obtained by ProPublica. The authenticity of the documents was confirmed by U.S. counterterror officials.

Although the culture of the Islamic State is repressive and bureaucratic, the reality on the ground can be rather anarchic. The restrictions on high-tech devices described in the Italian investigation were by no means uniformly imposed or obeyed. Militants in the self-styled caliphate have access to computers, smart phones and social media. Some have posted a barrage of messages and images, including videos of atrocities.

In response, leaders of the Islamic State have told foreign fighters to curtail their activity on social media because it exposes them to eventual prosecution back home or to being targeted in Syria. The use of social media has continued, however, according to European counterterror officials.

In fact, the cacophony of voices from Syria has been crucial to recruitment.

“In contrast to what has happened with other conflicts, the recruitment and the propaganda aren’t just in hands of the public communications apparatus,” said a counterterror chief of the Spanish police. “Each fighter has a phone and narrates his day-to-day life, his blog … A lot of these terrorists have circles of associates in Europe because they came from there, so this is effective publicity.”

Tapping into such communications, Italian police gained insight from their investigation of a family of Muslim converts from the Milanese suburb of Inzago.

Maria Giulia Sergio was 28. In September of 2014, she married an Albanian extremist she barely knew so they could join the Islamic State, according to investigators. The couple traveled with his mother to the Syrian city of Sed Forouk and met up with Albanian relatives living there, including children. The husband’s brother died in combat, according to the report.

The newlyweds encountered “numerous daily obstacles” to staying in touch with people back home because of “rigid rules imposed by the Islamic State as well as the objective technical difficulties, in a country devastated by years of civil war,” the report says. “This had a positive impact on the investigations because it had the practical effect of multiplying calls among relatives” in Italy and Albania when they heard from the militants.

Sergio talked via Skype because the suspects believed it was “more secure,” the report says. Police intercepted the conversations nonetheless. Sergio described her husband’s stint in a training camp in Iraq. She talked about child care, Koranic classes, decapitations and a stoning, and implored her family to make the “hijrah,” or pilgrimage to Syria.

“I am speaking to you in the name of the Islamic State,” she said. Scolding her father for remaining in his job in Italy, she said: “It makes no sense for you to work for them. They are the ones who must be our slaves.”

In April of last year, the parents announced they would join her. The father asked if he should bring his driver’s license and if he could buy a car in the “Caliphate,” according to the transcript.

Police arrested the suspects before they could depart.

Meanwhile, authorities across Europe struggled to intercept a smaller flow of militants traveling in the opposite direction.

Western intelligence officials estimate that the Islamic State dispatched between 60 and 180 operatives to attack targets in Europe. The strategy appears to have been to overwhelm the security forces with sheer numbers. Even if most of the strikes failed, something would eventually succeed. The results were graphically evident in France, where authorities foiled 11 attacks in 2015.

Officials say the threat has changed since the days of al-Qaida. Osama bin Laden’s group had the flexibility common to Islamist terror networks, often developing plots based on the initiative, expertise and availability of recruits who reached its secret compounds in Afghanistan and Pakistan.

Al-Qaida created a team to oversee attacks overseas. The chief plotters were Middle Easterners or Pakistanis, and they guided operatives to targets with instructions via phone and email. U.S. counterterror agencies identified external operations chiefs and eliminated a series of them with drone strikes and captures.

Today, the counterterror community is still mapping out the Islamic State’s leadership, especially those involved in foreign plots, according to the senior U.S. intelligence official.

“The structure that promotes attacks is wider and deeper, but to some degree also more autonomous, than what we saw with al-Qaida,” the official said. “It isn’t the case where we can home in on individuals and have a fair degree of confidence that if we neutralize them we will have had a considerable impact on the threat. That’s not the situation with ISIS. We had a fairly comprehensive view of the structure of al-Qaida … With ISIS, we don’t have an exact picture. It’s an intelligence collection challenge that we are working hard to address.”

The Islamic State’s top echelons are dominated by Gulf Arabs, Syrians and Iraqis, including former military and intelligence officers. Foreign fighters serve in units known as “katibas” organized by nationality and language. Large Francophone katibas field hundreds of French and Belgians, many of North African descent, and thousands of Moroccans and Tunisians. Senior foreign fighters have the resources of a quasi-state at their disposal: money, technology, identity documents, training facilities. But they are also given considerable autonomy to develop plots, officials say.

“There is … leeway to foreign fighters and operatives to choose targets and methods on turf they know best,” the senior U.S. intelligence official said. “The foreign fighters know what the organization wants to see happen and they act on it.”

The attacks on Paris in November briefly made Abdelhamid Abaaoud, a 28-year-old Belgian ex-convict, an internationally known leader of the external operations unit. He participated in the massacre and died in a police raid days later in the gritty suburb of Saint Denis. Yet some investigators now believe his stature within the Islamic State has been overstated.

“To me he was an average leader,” said Judge Trevidic, who led investigations of plots in which Abaaoud surfaced. “Have you ever seen a general on the front in Saint Denis? That is all right for a lieutenant, a captain, but not for people above. 2018′

Based in Syria, Abaaoud selected, trained and deployed jihadis to Europe in 2014 and the first half of 2015. In addition to guns and grenades, he taught trainees about secure communications 2014 encrypted applications such as Telegram, WhatsApp and Truecrypt 2014 and set up protocols to contact them when they were in place.

Secure communications technology has been a recurring feature in recent attacks and foiled plots in France, according to French Interior Minister Bernard Cazenueve.

“Encryption is a crucial issue,” Cazeneuve told a small group of U.S. reporters in March. “All the attacks last year used encrypted phones or computers … It is a difficult problem for us.”

(For background on the use of encryption by terrorists, read this article in the June issue of The CTC Sentinel, the monthly publication of the Combating Terrorism Center at West Point.)

Abaaoud’s operatives did not always follow security procedures, however. In June of last year, Turkish immigration authorities detained Tyler Vilus, a French plotter en route to Paris with someone else’s Swedish passport. Allowed to keep his cellular phone in a low-security detention center, Vilus brazenly sent an unencrypted text message to Abaaoud in Syria, according to a senior French counterterror official.

“I have been detained but it doesn’t seem too bad,” the message said, according to the senior official. “I will probably be released and will be able to continue the mission.”

Instead, U.S. spy agencies helped retrieve that text and French prosecutors charged Vilus with terrorist conspiracy.

In another case linked to Abaaoud in early 2014, the NSA played a central role in helping French police track down Ibrahim Boudinah, a plotter captured in a Cannes safe house as he prepared an attack with explosives and guns.

Officials said the Paris plot did not involve much long-distance direction. Instead, Abaaoud and two other field coordinators travelled separately from Syria to Europe to prepare the attack. Abaaoud entered via Greece in September, probably melting into a vast, chaotic flow of illegal immigrants that the Islamic State has used to infiltrate operatives during the past two years.

The other two leaders made their way to Budapest, where they sheltered in crowds of refugees camped out in the train station, according to Belgian prosecutors. On September 9, they were picked up at the train station by a Belgian accomplice named Saleh Abdeslam, who drove them to Belgium, prosecutors say.

The pair had different skills. Laachroui, a former electrical engineering student from Brussels, was the explosives expert. Belkaid, an Algerian petty criminal who lived in Sweden before joining the Islamic State, had more ideological preparation and religious knowledge than the others in the attack squad, officials say.

In Brussels, the duo set up a remote command post in a safe house and coordinated the Paris operation by phone. The gunmen and bombers used a number of disposable cell phones during the attacks. Although police reconstructed the activity of a phone found near the scene of the attack on the Bataclan concert hall, plotters exchanged encrypted messages before the attacks that French intelligence could not detect, according to Interior Minister Cazenueve.

Afterward, the coordinators in Brussels made calls and wired money to help Abaaoud, who was on the run, rent the shabby apartment north of Paris where police eventually killed him. Intercepted calls suggested that Laachroui and Belkaid “had some rank or influence,” a senior Belgian counterterror official said.

Laachroui and Belkaid became targets of a massive manhunt along with Abdeslam, the lone surviving attacker from Paris, and several other accomplices. They were sheltered for four months in Brussels by networks based on clan, ethnicity and criminality and ruled by a code of silence. Police monitored at least 89 phones. An intercept picked up ominous chatter.

“It was possibly Saleh [Abdeslam] or an associate,” the Belgian counterterror official said. “He’s saying: ‘The cops are chasing me. Things are hot here. We are going into action.’ And the voice in Syria says: ‘Yes, go into action.’ But there wasn’t intelligence of a precise plan.”

That snippet of intelligence was among the leads that prompted Belgian authorities to shut down their capital for five days in November. But the attack didn’t happen for a while.

During the next four months, the bombmaker Laachroui stayed in close contact with the Islamic State using encrypted methods, principally Telegram, according to Western counterterror officials. The fugitives also communicated with Syria and each other using WhatsApp, Skype and the mobile application Viber on devices including laptops, tablets and phones, counterterror officials said.

Laachroui exchanged audio recordings and text messages with Abu Ahmed, a French-speaking “emir” of foreign fighters in Syria, counterterror officials said. Investigators believe Abu Ahmed was based in Raqqah, the Islamic State’s headquarters, and that he also played a role in overseeing the Paris plot.

It’s not clear if Western intelligence officials have identified Abu Ahmed yet. Several veteran French jihadis in Syria are suspected of overseeing external operations. They include the converts Fabien and Jean-Michel Clain, brothers from Toulouse (Fabien Clain is believed to have issued the Islamic State’s claim of responsibility for the Paris attacks); Boubaker el Hakim and Salim Benghalem, Parisian veterans of a cell that first waged jihad in Iraq in 2004; and a blond convert known as Abu Sulayman al Fransi, described as a former physical education teacher and father of two.

At one point, Laachroui told Abu Ahmed there were tensions among the fugitives in Brussels, according to officials.

“They were squabbling,” the Belgian counterterror official said. “They told the emir: ‘We have split up. We were arguing before. It’s better now.'”

As Laachroui worked with TATP, a highly volatile explosive, in his hideout in the Schaarbeck neighborhood, he asked Abu Ahmed for technical help from bomb experts in Syria.

“He has them check different mixtures and adjusts his work based on what they tell him,” the Belgian counterterror official said.

The fugitives and the emir discussed potential plans including a massive attack with 600 kilos of explosives that did not take place. The goal was to wait and strike France again in hopes of disrupting or cancelling the month-long Euro 2016 soccer championship scheduled to begin in June, according to counterterror officials.

European spy agencies and their allies in the United States and Britain deployed the full weight of their sophisticated technology in the search for the plotters. But neither the NSA nor Britain’s Government Communications Headquarters (GCHQ), spotted the digital footprints, officials said.

“Everyone was trying to find these guys,” the senior French counterterror official said. “They were able to elude us. But they were able to elude the Americans, too, and that shows you what a problem encryption is.”

Police work on the ground produced a breakthrough on March 15. A forgery investigation turned up fake identity cards used to rent apartments. Police went to what they thought was an abandoned safe house near a Renault factory in the Forest neighborhood.

The raiders used a hand-held battering ram on the door. They discovered Belkaid aiming an AK-47 at them. The firefight wounded him and several investigators, who tumbled down a flight of stairs seeking cover. A SWAT team finished off Belkaid. Police found the cache of ammunition for AK-47 rifles–and DNA traces of Abdeslam, who had fled while Belkaid held off the officers.

The bad news reached Laachrooui, who sent the urgent message to Syria. Abu Ahmed told him just to use bombs, and the fugitives decided to hit Belgium rather than France because time was running out. Once again, the Islamic State let them decide targets and timing, counterterror officials say.

On March 22, Laachroui and another suicide bomber blew up at the Brussels airport. A third bomber struck at a subway station. Thirty-five people died and more than 300 were wounded.

It could have been worse. Two other bombers lost their nerve and fled. The improvised nature of the attack raises the possibility that it might not have succeeded without the guidance from afar.

Airstrikes and battlefield losses have intensified pressure on the Islamic State. It has become more difficult for plotters to enter Europe illegally and for recruits to reach Syria, counterterror officials say.

But terror groups often accelerate attacks abroad in response to defeats at home, as reflected by a recent flurry of strikes in Turkey, Iraq and Bangladesh.

An Italian investigation this year gives glimpses of a changing landscape. The chief suspect was Abderrahim Moutaharrik, a Moroccan immigrant living north of Milan with his wife and two children. A professional kickboxer, he was so radical that he posted a photo of himself in the ring wearing a t-shirt with the Islamic State logo, investigators say.

Moutaharrik and his associates used WhatsApp extensively, according to a 72-page Milan prosecutor’s report dated April 19, 2016. The police secreted microphones in homes and cars and implanted mobile devices with spyware, the report says.

Moutaharrik idolized Mohamed and Osama Koraichi, brothers fighting for the Islamic State. His desire to emulate his friends increased after Osama died in an air strike in Iraq last year. Moutaharrik sang jihadi anthems and ranted about killing infidels in front of his four-year-old son, the report says. His wife obtained a bank loan of $7,800, allegedly to finance their imminent move to Syria.

But there were problems. Mohamed Koraichi was difficult to reach and couldn’t find the family a smuggler in Turkey.

On March 15, Moutaharrik finally received an audio message from Koraichi via WhatsApp from an Indonesian phone in Syria. They discussed the kickboxer’s hopes of making the pilgrimage.

On March 20, Moutaharrik received another WhatsApp message from Syria. But this time a new voice addressed him by name: a high-ranking militant who spoke flowery classical Arabic. He wanted the kickboxer to wage jihad right away.

The “sheik” said the Islamic State was under attack by “Christian” armies from “bases in your homes, European bases,” according to the transcript.

“You must take revenge on them, revenge for the Muslims,” the sheik said. Praising “lone-wolf” operations, he continued: “in the Christian nations, one operation gives more satisfaction than dozens of bombs … You will be among those who do this good deed in the lands of the Christians, in Rome, in Italy …”

“We will take revenge,” Moutaharrik replied, according to the transcript. “God willing here they will only know massacres and killings.”

The police went on red alert. Suicide bombers struck Brussels two days later. Moutaharrik and his wife celebrated, according to transcripts. The sheik and Koraichi urged the increasingly agitated kickboxer to attack right away.

Italy “is the capital of those who carry the cross, my brother,” Koraichi said. “Until now there hasn’t been any operation done [in Italy], know that if you do an attack it is a great thing.”

On April 8, the sheik sent a soliloquy that he titled “Bomb Poem.” It evoked images of fire, suicide bomb belts, and Islamic battalions “annihilating the infidels.”

Police arrested Moutaharrik and three other suspects 11 days later.

ProPublica is a Pulitzer Prize-winning investigative newsroom. Sign up for their newsletter.

The overdose death toll from opioids, both prescription drugs and heroin, has almost quadrupled since 1999. In 2014 alone, 28,000 people died of opioid overdoses, more than half from prescription drugs.

Just last month, public awareness of the opioid epidemic reached a new level when Prince was found dead with prescription narcotics on him and authorities began to investigate their role in his demise. In recent weeks, lawmakers and regulators have moved to augment treatment options for addiction and to require more education for doctors who prescribe opioids. The U.S. House of Representatives is voting on a package of bills this week; the Senate passed its own bill in March.

Also in that span, the Los Angeles Times has published an investigation of Purdue Pharma, the maker of the blockbuster pain pill OxyContin, and CNN held a town hall meeting on the consequences of addiction to narcotics. Dr. David A. Kessler, former commissioner of the Food and Drug Administration, wrote an op-ed in the New York Times, calling the embrace of opioids “one of the biggest mistakes in modern medicine.”

Today, ProPublica added warnings labels to the pages of narcotic drugs in our Prescriber Checkup news app, prompted by indications that some readers are using the tool to find doctors who will prescribe these drugs with few or no questions asked (See our editor’s note).

The effectiveness of any of these steps remains to be seen. There is broad consensus on the need for more treatment options, more education, more careful prescribing by doctors. But there’s still much debate about the details—and funding–for each of those steps.

What’s clear is that in recent months there has been an increasing emphasis on the role of health providers and the agencies that oversee them to stem access to widely abused prescription drugs:

• In March, the Centers for Disease Control and Prevention released guidelines on prescribing of opioids for chronic pain, defined as pain that lasts for more than three months (excluding pain related to cancer, end-of-life and palliative care.) The guidelines call on doctors to choose therapies other than opioids as their preferred option; to use the lowest possible doses; and to monitor all patients closely.
• That same month, the FDA announced tougher warning labels on immediate-release opioids, such as fentanyl, hydrocodone, and oxycodone, to note the “serious risks of misuse, abuse, addiction, overdose and death.”
• Nonprofit groups and medical experts in April asked the federal Centers for Medicare and Medicaid Services to remove questions about pain control from a survey of hospital patients’ satisfaction to remove any incentive to overtreat pain. And they asked The Joint Commission, which accredits health facilities, to revise its standards to deemphasize “unnecessary, unhelpful and unsafe pain treatments.” The commission pushed back, saying its standards do no such thing.

Just yesterday, Dr. Steven J. Stack, president of the American Medical Association, called on doctors to do more. He encouraged doctors to use their state’s Prescription Drug Monitoring Program to ensure their patients aren’t shopping for multiple doctors to prescribe them drugs. He called on them to co-prescribe a rescue drug, naloxone, to patients at risk of overdose. And he told them to generally avoid starting opioids for new patients with chronic, non-cancer pain.

“As physicians, we are on the front lines of an opioid epidemic that is crippling communities across the country,” Stack wrote in a statement, published on the Huffington Post. “We must accept and embrace our professional responsibility to treat our patients’ pain without worsening the current crisis. These are actions we must take as physicians individually and collectively to do our part to end this epidemic.”

ProPublica is a Pulitzer Prize-winning investigative newsroom. Sign up for their newsletter.
Read original article
As Opioid Epidemic Continues, Steps to Curb It Multiply