U.S. Bank, the fifth-largest commercial bank by assets in the United States, was charged by federal authorities Thursday with failing to guard against illegal activity and, in at least one instance, even abetting it.

The Justice Department accused U.S. Bank, which is based in Minneapolis, of severely neglecting anti-money laundering rules, helping a payday lender operate an illegal business and lying to a regulator about its plans for tracking potential criminal activity by bank customers.

Federal prosecutors in Manhattan reached an agreement with U.S. Bank to defer prosecution as long as the bank could show it had improved its monitoring of customer transactions. To settle the Justice Department charges and cases brought by other regulators, the bank agreed to pay various fines and penalties totaling $613 million.

In a statement, the bank’s chief executive, Andy Cecere, said that U.S. Bank had made “significant investments” toward improving its program for combating money-laundering.

“Our culture of ethics and integrity demands that we do better,” he said.

The action against U.S. Bank was modest compared with cases involving banking giants such as HSBC and Standard Chartered that authorities found had done business with drug gangs and countries such as Cuba and Iran that had been barred from the U.S. financial system under international sanctions. In 2012, for instance, federal prosecutors settled a money-laundering case against HSBC for $1.9 billion.

Unlike the earlier cases, the one involving U.S. Bank focused mainly on warning signs that the bank was ignoring.

For about five years, senior bank employees tried to balance their security system’s constant alerts about suspicious customer activity against internal staffing limitations, authorities said. In a late-2009 memo, court documents show, U.S. Bank’s anti-money-laundering officer warned the chief compliance officer that employees responsible for investigating the alerts were “stretched dangerously thin.”

But senior bank officials knew they should not be curtailing money-laundering inquiries just because they did not have enough employees, so they hid the practice from federal bank examiners, according to prosecutors. One U.S. Bank employee left references to the staffing problem out of the minutes of internal meetings, fearful of what regulators would think, the Justice Department said. Tests of the alert system were halted after they showed that the bank had failed to report suspicious customer activity to regulators.

Dana Ripley, a U.S. Bank spokesman, said that most of the senior employees involved in the practices at issue had left by 2014, when the bank overhauled its anti-money laundering program.

Jeffrey Alberts, a former federal prosecutor who specializes in issues related to white-collar crime and bank regulation, said that scrutinizing potential criminal activity was a challenge for banks. “There’s no level of monitoring you can impose that will catch every single suspicious transaction,” he said.

Alberts, a partner in the law firm Pryor Cashman, said that banks often struggled to determine how precisely they needed to monitor customer transactions to satisfy regulators. Nonetheless, he said after reviewing the charges, “There are some things that are alleged here that sound clearly wrong.”

For example, prosecutors said the bank had failed to report strange behavior by one longtime customer, Scott Tucker, who was convicted in federal court in October of charges stemming from an illegal payday lending business he operated.

According to the charges announced Thursday, the bank kept open numerous accounts controlled by Tucker even after it received a subpoena from regulators for information about him. Tucker was using corporate accounts whose ownership he misrepresented to pay for millions of dollars spent on personal items like a vacation home in Aspen, Colorado, and a Ferrari racing team.