Just curious, does it sound like this?
https://www.youtube.com/watch?v=dv8x3AfwhOU
That's what mine does/did. Same t-case and shaft as you. It doesn't do it as bad anymore after alternating running some 2 stroke oil and Diesel Kleen every fill up through it for a while, which has smoothed out the idle considerably. I just add Diesel Kleen now.
Like you, everything seems tight and I don't have any vibration (aside from normal!) when driving. Only does it when warm and only in neutral. Put it in gear and it goes away. Pull down or push up on the shaft it goes away.
Since it's lessened considerably, I've just put it on the back burner since at some point down the road I'll be doing a driveline upgrade and other than being annoying it didn't seem to affect anything.