as others allude to, there are many reasons for the shortage of M54 cargoes, however as Fuzzytoaster mentioned, attrition and age is the factor of the situation you find today .. 15 to 20 years ago I could point you to six surplus truck dealers who each had 6 or more M54's sitting in their yards .. The scrap purge of 2007-08 was the end for many old-school 5-ton and 10-ton trucks that were otherwise decent operational trucks...
I wish my CONUS army inventory report from 1983 was in my hand and not still in storage before I say this, but to answer your main Title Question, I certainly believe that way more M35 cargoes were built than M54 cargoes. (I have to say cargoes at this point because there are readers out there who call any and all variation of a deuce or 5-ton as "M35" or "M54", let alone those who call 5-tons "deuces." ). Not only was the M35 cheaper to procure, they were small and versatile system-wide, a well-suited size for all-around utilitarian use .. Case in point, I have never seen a 5-ton backed up to the loading dock of any mess hall nor zipping around the cantonment areas of any base doing mundane daily tasks. Has the M54 become rare? They are only rare in the sense it is hard to find one that has been put up for sale, as others allude to in other posts.
In my experience, M54's were nowhere to be seen in straight infantry units and most other types. Conversely, M54's were the truck of choice for armor, artillery, and a few other kinds of line and combat support outfits, basically, units whose missions required something much heavier than a deuce but that could still get into tight areas, negotiate densely-wooded areas in support of artillery or armor positions, narrow rough trails, etc... I cannot speak for the use of M54's by navy or air force however I do recall a picture of a (gas) M54 with USAF markings hauling 55gal fuel drums during the Korean War