>>14 I must agree wholly.
You are not a bad person for getting annoyed at something that could piss anyone off. The child attaches himself to you because between you and his blood father, you are the closest thing he probably has to a real dad. He wants to be around you due to the fact that you love him and give him the attention that he craves from other people. You are, in the simplest terms, his dad. He thinks of you as his father more than his own flesh and blood. You support him, and comfort him, and give him what every child needs in their young lives; love.
As for your brother, I must say, if he won't do anything to pitch in, then you need to address this fact when the boy isn't around. It is imperative that you speak with him on this matter, otherwise it will progressively get worse. He's essentially leaching off of you. You don't have to tolerate that in him, even if he's your brother. He's having you raise his child for him, pay for his shelter, and probably feed him with nothing in return. That is not fair to you. You are his flesh and blood, he should not be doing this to you of all people.
You should start by just asking him to help pay for the house and pitch in with the food money. If he agrees then slowly work your way up to having him help with HIS son. If he can't handle that, then tell him the truth. Sometimes, the first thing someone NEEDS to hear, is the last thing they WANT to hear. So, if he declines this attempt at a deal of sorts, then tell him outright what the problem is. Tell him that if he doesn't help, then he will have to do something else.
No one deserves this from him, but particularly you. You're his brother, he should be trying to do anything in his power to help you, not make your life miserable. If he can't do that, then flickers is right, he doesn't deserve you, your time, or your space.