That's not even the case all the time, which is why it seriously pisses me off when people do this.
For example: The Box Turtle. They have a home range of about 100 square yards (think about that, it's only a 30 foot by 30 foot area...very small). When they're taken out of the wild and re-released, their own instincts kill them unless you're careful. They have such a strong homing instinct that they will continue to search for something familiar (all the while ignoring hunger and thirst) until they either find it or die of starvation/dehydration. So if you release something like that, you have to let it go EXACTLY where it came from, otherwise you might as well just kill it yourself because that's basically what you're doing anyway. When we get them in at the rehab center, we pretty much know we have to find them a home or use them for educational purposes unless the person who brought it in can come and get it when it's in better health. They don't eat readily in captivity unless they were CB or can be "taught" by other box turtles...and yes, we do have a "teacher turtle" that we have to put our rehabbing box turtles in with so they'll learn to eat in unfamiliar settings.
Also, another aspect of this comes from non-native animals that are "harmless because they won't survive in this habitat." I live in Pennsylvania. It gets cold here. In April, for example, we had a few 70 degree days followed by a 3-4 foot snowfall. Alligators would never survive a winter here...nonetheless, about 3 weeks ago, I found a 3-4 foot alligator (or croc, I didn't get a close enough look at the head to see for sure) in the lagoons in Presque Isle State Park along Lake Erie. It swam right underneath my raft, and nobody's been able to catch it yet. Just last week my parents told me that the local news (they live 2 hours south of me, but still in PA) at home said that a several foot gator was seen on a highway right outside of my hometown. Sure, they'll die off this winter, but until then, they're going to absolutely destroy the local wildlife.
Every action by irresponsible pet owners has a consequence somewhere along the line. It can be like an example that I mentioned above, or whether it's giving other (responsible) pet owners a bad reputation so that bans are placed on what animals can be kept.