The practical difference between ethics and morality is that to see something as moral, you need to have guilt feelings about not doing it. The converse is the same; if something is felt to be immoral, it's because you feel guilty about doing it. Ethics, you can think about, and extend as necessary to confront situations outside of your usual framing. (Did I say it was easy? No, and I won't. Ethics is vastly more complex than morality.)
Worse, of course, is the feeling that someone who crosses the bounds of your morality is somehow a lesser being than you are, but that feeling can be put down to pure inverted jealousy: Why does that person get away with doing something I'm not supposed to, without negative consequences? This applies even if you can see why the specific thing isn't a good idea (long term thinking), because what's being kicked in the stomach is the restrictions you believe you must accept to be acceptable. Here is a person who is acceptable, and yet they don't have to carry the same burdens that you do in order to warrant it. Grrr...
Morality isn't necessarily a religious phenomenon. Guilt trips of all kinds can be behind the structure. It's just that religion is most often the obvious companion for the exercise, and it's ultimately a better defender of the practice. It is easier to sort out and come to terms with prejudices your parents and friends may have tried to instill in you than it is to argue with the seeming authority of a God.
It's possible that morality, and its associated guilt, is also the base cause of "othering", and what a number of diarists blithely refer to as "tribalism". There's nothing like a strong feeling of guilt about your own feelings and actions to produce a tendency to hang around with other people who feel the same structure of guilt, so that people who infringe on it won't make you uncomfortable.
It's all too common for people to define themselves, and others, in terms of the limitations they perceive. It's also much simpler than trying to define anyone in terms of their potential, in that it is at least a finite description, where an attempt to include all positive possibilities is doomed by the sheer size of the field, much of which is likely unknowable at any given time.
Perhaps it is thinking you have to define yourself at all that sets up the whole problem to begin with. Or perhaps it's the presumption that you can....
More below...
I like the story of the blind men and the elephant; it's applicable to so many, many things.
Like human beings, as seen by other human beings. Each of us has our own snapshot of every person we come into contact with. And a whole scrapbook of snapshots of self, what we call "I", as we do different things, or with different people, or circumstances, and as we change from day to day. Many of them are on separate pages, and wholly irreconcilable with others, but as long as the pages are separate we can ignore it. The elephant never has to be looked at, and it's much more comfortable that way.
The prospect of dealing fairly with other human beings is a daunting one, if you try to extend it out beyond your immediate family and acquaintances. The universe isn't particularly fair, nor predictable except in general, nor are people in particular. "Fair" is a human concept, and possibly not a valid one. This is probably what most worries most people most of the time. It's why so many of our social mechanisms are attempts to constrain the future actions of the people around us, in order to have a better picture of what "the future" will be like. Personally, I think a predictable future would be 1) boring, and 2) incredibly scary. It's much easier to predict calamity than to accept change that cannot be predicted beyond, even though the second condition is the likeliest to occur.
We are much more likely, or at least more able, to be "fair" to people whose limitations match our own. At least, then, we get to confine ourselves to actions that we see as fair to ourselves.
We look for boxes to put both ourselves and other people in, and we keep doing it no matter how often we're shown that it doesn't work particularly well.
The alternative, of course, would be to have to deal with each person as a mysterious and complex individual at all times. We don't have enough time for that. That's not snark, or exaggeration. We literally do not have enough time, at least most of the time, to consider all the facets of another personality, or even our own.
So people become complicit in developing not-too-uncomfortable boxes, into which we can be placed when there is no time to do it differently. We do it for work, for socialization of many different kinds, for most family interaction.
And this is a good thing. The key, though, to having this work well for the long term, is to make sure we take time to let ourselves and others out of the boxes, to accept the realization that people, including ourselves, are not as constrained as we may need to seem for our everyday interactions. And perhaps making that transition is the hardest part of the process.
Update 5:44 pm CDT: So far it seems as though everyone who commented got hung up on the first couple of points and never went any further. Mea culpa - I probably should have known better. I'll try this again in a few months with a different beginning and see if I can make the riff come off right.