Each person giving money to panhandlers knows that the endgame of their money will never be known. It is not up to the person giving money to expect panhandlers to spend it wisely. They are surviving and how they choose to spend their money is up to them.
It is human nature to give, and there should be nothing wrong with that. A single individual might not be able to cause a systemic change overnight, so it is okay to let them help someone get through the day or even the next couple of hours ahead of them. Human nature demands food and warmth, and the people donating should have that in mind without resorting to immediate judgment and assuming that they would misspend the money.
Both judgments should be trusted and respected equally.
Based on the degree of perceived need, we choose to donate money. Panhandlers understand this, so there is an opportunity on their part to exaggerate their needs, either by lying about their circumstances or by noticeably degrading their appearance rather than seeking support.