Public trust in our judiciary is critical to American democracy. Unlike members of the executive and legislative branches, whom we can vote in or out of office, federal judges serve for life. As a result, their power hinges on the public’s confidence in their decisions.

Even the slightest hint of bias or partiality threatens the integrity of the judiciary and the rule of law. It isn’t enough that judges believe they can be impartial, they must also appear to be impartial.

In recent months, we have heard calls for judicial reform, with critics advocating new rules and structural changes to enhance the courts’ transparency and accountability. Beyond mandatory codes of conduct and term limits, one focus of critics’ concern is recusal — a judge’s withdrawal from a case to avoid a conflict of interest or the appearance of a conflict.

On this matter, judges are bound by a federal law, 28 U.S.C. Section 455(a), which states that “any Justice [or] Judge … of the United States shall disqualify himself in any proceeding in which his impartiality might reasonably be questioned.” The word “shall” makes recusal mandatory, while the word “might” underscores that judges must err on the side of caution in close calls. Notably, however, there is no recusal police.

During my time as a judge on the nation’s second-highest court, the U.S. Court of Appeals for the District of Columbia Circuit, I adopted what I called a “5-minute rule” in deciding whether to recuse myself from a case. The rule was simple: If I had to spend more than 5 minutes thinking about how my participation in a case might be perceived, I chose to recuse.

Once, though, a case tested that confidence in an especially personal way. I was scheduled to sit on a case called American Council of the Blind v. Paulson, and — for the first and only time in my judicial career — I recused myself because I’m blind. The case was about money — and not just in the abstract. It was about the 1s, 5s and 20s we carry in our pockets. The legal question was how quickly the Treasury Department had to comply with a federal law that required currency to be accessible to the visually impaired, either through tactile markers or size variations.

As a blind person, I had a unique perspective on the case. When I travel in Mexico, I can distinguish bills easily because they increase in size with their denomination: a 500-peso note is larger than a 100, and so on. But here in the United States, I can’t tell a $5 bill from a $20 bill without assistance. When my kids were little, they helped me organize my wallet, using a folding system to keep track. Now that they are grown, my wife has taken over. Still, if my bills get mixed up, I need to ask a merchant or cabdriver, “Is this a 10?” I don’t think I’ve ever been cheated, but why does the government put me in this vulnerable position?

So, I had strong opinions about the case — I wanted accessible currency, and I wanted it quickly. But I was also certain I could set my personal views aside and rule as the law required. I did that every day. Still, I couldn’t ignore what the public might think about my decision. If I ruled that Treasury must speed up its efforts, would people assume I’d done so out of self-interest, regardless of how clear the law might be? I didn’t want to give anyone reason to believe that a blind judge had ordered the government to make costly changes for his own gain.

The idea that judges might be ruling for reasons other than what they believe the law requires is a corrosive one. Maybe the public would have appreciated that a judge with relevant personal experience had decided the case. But I couldn’t be sure. So I knew what I had to do: Because I had to think about it for more than five minutes, I followed my five-minute rule and recused.