There is a rule for stutterers interacting with society, and there's suffering that goes along with having a speech impediment. You might have difficulty speaking, then some people think you're retarded. The tone of your voice might not match how you feel inside, then people react inappropriately from ignorance. Or, your facial expressions are misinterpreted.
All of the above are part of the speech impediment.
Thinking it's only about repeating syllables is oversimplification. The rule for stutterers interacting with society is simple but potentially depressing: when it becomes more obvious, the stutterer will get mocked and rejected - it just depends on party affiliation.
So too, those calling for him to step down aren't doing so because of his age or incompetence, they are biased against anyone having a speech impediment (note: below article is from 2020).
(bold mine)
https://niemanreports.org/articles/biden-stutter/
I can’t say with any certainty whether Joe Biden, the current front-runner for the Democratic nomination for president, is in “cognitive decline,” as many journalists and critics have begun to ponder aloud. I can say with certainty that what you are seeing of Biden during debates and on the stump can plausibly be explained by his lifelong battle with stuttering. I also know why it is easy for anyone to point to a few clips of Biden to make a seemingly compelling yet misleading case that his verbal stumbles are really about cognitive problems.
Biden has spoken openly about his stutter, which he’s struggled with since childhood, only occasionally. Words beginning with “s” were particularly difficult when he was growing up, Biden told The Atlantic’s John Hendrickson, who also stutters. Biden wrote a letter for National Stuttering Awareness week in 2015 for the National Stuttering Foundation and mentors young people who struggle with their speech. Journalists who don’t know this history or don’t take the time to understand the complexity of the speech disorder will likely inadvertently mislead their audiences during any discussion about Biden’s mental capabilities.