I wouldn't say the M123 series, any variant of it, was rare, in the sense that few were made. However, as most of them were made in the 1950s thru the early 1970s, that would make the youngest ones about 40 years old. They are too big for most to own, certainly not a daily driver. Most, when surplussed out, led lives as logging trucks or hauling very heavy things in workplace environments. Most that survived their service life in the military, including those that came back from Vietnam, came out into a world that would kill them through forced labor and abuse. I would say that the M123 series is rare now, only because most of them are gone, rusting away, and many have already fallen to the scrapper's torch.
There are a few who love the big iron, dirt71 and AMGeneral come to mind when I think of M123 owners. If you can save one, be advised, if it's in good shape it would be a shame to NOT save it for future generations to admire. But be warned, the big iron is much cooler to look at than it is to drive or maintain. If you don't have a shop, your own private one or a real working maintenance facility, you should think twice, as the tools you need will be BIG. Also, you should double-check with your wife. Big green iron ALWAYS puts a strain on marriages.