I am working on gravel, too. A few ideas come to mind. You can make a "protector" for the spindle threads out of a short piece of PVC pipe. Cut that piece open lengthwise and remove a strip, leaving you with a 3/4 circle that you can snap over the spindle. Then you can pull the brake drum all the way forward to the end of the spindle without having to lift it.
Maybe you can sling the brake drum/hub assembly with a ratchet cargo strap, a rope or sling attached to the cargo bed and swing it out and lower it that way. Use a dolly to move the assembly around (I used to carry them until I finally asked myself "why???")
Another idea: You can leave the wheels attached, pull the driveshafts, then jack the axle just enough to take the weight off the wheels, slide a sheet of plywood between wheels and gravel and work the wheels off with the hub and drum attached. The weight of the wheel gives counterbalance and it is easier on the back...as long as you keep them upright and do not let them fall over. I have done it that way, albeit only on one axle.
Lastly: Attract young local talent with promises of pizza and beer and have them lift heavy stuff under your direction!