What leads people to do things they know full well are stupid and self-destructive?

(This is a revised version of a column I wrote several years ago. However, I’ve had more time to ponder the subject, so here goes....)

A man commits the perfect murder. It’s totally indetectable, with no evidence whatsoever to connect him to the crime.
For years the man goes on living his life, safe and silently gloating. Nevertheless, he occasionally feels a twinge in his mind. It’s not conscience; that’s never been an issue for him.
It’s an awareness the only thing that could possibly convict him would be an open confession — the one thing he knows he absolutely shouldn’t do.
One day, as the murderer walks down the street, he’s hit with a compulsion to tell someone. He holds it in, but the compulsion grows stronger.
The story ends with the man who committed the perfect, indetectable murder standing on a street corner bellowing his guilt to everyone who will listen, and being led off by the police to face the hangman.
That’s a summary of Edgar Allen Poe’s short story The Imp of the Perverse. If you want to read the whole thing you can find it for free if you search the Internet a bit.
Poe’s point: everybody has a twist in their personality that makes them want to do the wrong thing in any given situation, no matter how stupid or self-destructive.
(In the short story, to give the irony an extra twist, Poe has the murderer’s Imp of the Perverse compel him do the moral thing.)
Everyone has an Imp of the Perverse in their own personalities. I do, you do, the person standing next do you in line at the grocery store does.
Another metaphor for the Imp of the Perverse is that naughty shoulder angel in cartoons who urges you to do stupid or immoral things just because you can. You follow the bad shoulder angel’s advice — and promptly get clobbered for it.
Give yourself three minutes to think about it, and you can probably list half-a-dozen examples in your own life of when you did stupid things that came back to bite you in the butt — knowing full well they were stupid things that would come back to bite you in the butt — but did them anyway.
Most people can see their Imp of the Perverse for what it is, and can resist it — at least in the big ways.
You know you shouldn’t eat that second doughnut, but you do anyway. That’s a minor one.
But if you’re normal you’re able to stop yourself from deliberately crashing your car through the wall of that business, or of shooting up with heroin, or of strolling over and punching that screaming child in the face.
You know they’re bad things, at some level you want to give them a try, and yet you’re able to stop yourself from doing them.
Some people are ruled by their Imp of the Perverse. It’s the dominant part of their personality.
We all know people who shoot themselves in the foot 50 times a day, who consistently set themselves up for failure in situations when they could just as easily succeed. At work, with their families or friends, they constantly sabotage any chances of getting ahead they could have.
I just want someone to explain to me: what do they get out of continuing to behave that way? There must be something — and it must be pretty powerful.