You mention in one sentence that you two are "trying" to address this issue and then in another you state when you try to talk to him "he gets mad or just walks away".
Both these sentences say exactly the opposite.
Perhaps the reason he does not wish intimacy with you as often as you wish are for reasons that would be painful for him to say to your face and painful for you to hear. If so, a neutral third party (i.e., a therapist) may be beneficial; someone he can speak with that is not so close to the issue as you are.
You can't "fix" anything here. Your partner prefers to masturbate rather than invest the time and energy into sexual contact with you. It would appear you've approached him regarding your concerns and he's blown you off (although, not literally).
There is nothing medically wrong with you that requires a visit to a physician.
Your "illness" can be cured by leaving this relationship and finding someone who is not quite as lazy as this person.
Great...no time to exercise but enough time to drink on the weekends and smoke cigarettes. You don't have a low libido, you're not shy. You're out of shape and run out of steam before reaching a climax.