I've been on all sides of the off-roading issue. I've driven 4x4s, motorcycles, and yes an MV, off road. I've also been a land manager as well as a land owner.
I'm not going to point at any one group of off-roaders, I'm just going to say that off-roaders, as a whole, are our own worst enemy. As long as rules like "stay on the trail/road" "keep out" "no trespassing" and "foot travel only" don't apply to us, we will continue to provide everything "they" need to shut us down and kick us out. This problem would be correctable, but I don't know that it ever will be.
Another major problem is one of numbers. When I started riding dirt bikes as a teen, the trails we used might see a dozen or so motorcycles a month. By the time I gave it up to do other things, in my late twenties, they might see a couple dozen a weekend, and maybe a few 3 and 4 wheelers. Eventually, those same trails were seeing a few hundred a weekend. Now they are all closed.
Then there is the problem of voice.
I attended several "Limits of Acceptable Change" meetings for the Daniel Boone National Forest. There were a lot of representatives from several different groups at these public meetings. There were horseback riders, hikers, backpackers, environmentalists, forestry professionals and volunteer trail workers. If there were off-roaders there, I never saw them. These meetings are public. Anyone can attend, and anyone can voice their opinion, as long as they are civil about it.
As you can see, I don't have any answers. Like others, I just point out the problems.