Hello. I am not a cat pychologist but I have owned many cats. First, I'd like to comment on having your cat declawed at this point. In my opinion he seems somewhat insecure having been moved from different homes and declawing him may make him feel defenseless against whatever it is that he feels threatened by. Also, declawing is not a pleasant experience for a cat. It is very painful and very involved. Not like cutting our nails off. Please educate yourself on this process before you make that decision. I'm told that some vets will not declaw cats anymore.
Do you know how your cat was treated when living with your brother? Having been separated from you and never been around other animals may have stressed him out and saw them as a threat since you mentioned him keeping the dogs away from your little niece. Try to figure out how he became so threatened by men. Do some research.
There are books and information online that may help you to understand his temperment. They do talk to you with body language. You need to really work with this kitty, maybe one on one to get him to trust you and know that you are not a threat. This is not hopless but will take time and effort on your part and will definately be worth it.
Not knowing a lot about conditions where he stayed and if you are gone all day working, does he have toys to keep him occupied and so forth. It's really hard to say why his personality is a little on the rough side. Sounds to me like he has a nice side and could become a very nice pet if you can pin point what some his issues could be.
I hope this helps a little. Just show him lots of love, build his trust in you and most of all show lots of patience.
Again, please rethink the declawing. May do more harm than good.