One thing that may not have been mentioned is parts and service. I'd advise buying something for which you can get the normal wear parts and consumables for locally (if there are any local sources). If at all possible, try one out before making a choice. Most medium sized and larger cities have welding shops that have rental units, lease returns, trade-ins etc. They will generally let you try out a few units in-house if you ask.
When I was thinking about buying a plasma cutter the folks at our local welding mega-mart (Haun Welding) offered to let me take one or more of the used/rental units back to the shop for a while to be sure I got what I wanted.
The are good used units around you all the time. Check Ebay, Craigslist the local papers etc and make everyone you know aware that you are in the market. Welders show up in unexpected places.
About 8 years ago was having a beer with some guys in the NCO Club and mentioned that I was looking for a gas or diesel engine driven welder. One of the guys piped up that he had a Lincoln Ranger 8 in great shape he'd sell for $1200-1300. They were selling for $2600 or so at the time. It was stored in a walk-out basement on a cart and although it was 8 years old it only had 12.5 hours on it. I got it for $1300 with the cart, leads, cover, rods, helmet etc. He'd gone to school for two years for welding and had joined the Army right after that. He had the sales receipt and he had paid $2600 for the welder and accessories 8 years prior. All it needed was a battery and the carb cleaned out.
I bought my MIG welder with cart, gas bottles, several spools of wire etc for a similar savings from a car restoration hobbyist who was down sizing for health reasons. I found him in the local classified ads.
Christmas week the #5 son bought a like new Lincoln AC/DC 225/125 for $230 on Craigslist. It came with cart, leads, four tubes of rods etc. The owner, a logger, had bought a large MIG unit and wasn't using it anymore. The bare welder retails around $600 with tax.
The #1 son, Psycobob on SS, got a Lincoln SA-200 pipeline welder for hauling it away as it hadn't run in years, wouldn't start and was in the way. He tore down and cleaned the entire fuel system and carb and it runs fine after about 3 hours of TLC. Other than fuel for the pick up and gaskets he hasn't got a penny in it.
The point of all this is that there are a lot of solid used units out there for a lot less dough than new ones. If you can be patient for a while and keep your head up you can find some great deals.
The advice to buy a unit for what you really need as opposed to the things you might do sometime in the future is pretty sound, especially if funds are a little tight. Buy a unit for what you wil actually do most of the time and scratch you head a bit over the rest. Consider this your training wheels for welders.
Unless you buy an industrial unit you'll spend a lot of time waiting for the machine to cool (on thick stuff especially) so pay attention to the duty cycle (welding time at a particular power setting versus cooling time).
If you look around online you can find used welding textbooks and one of those can be a good self teaching aid.
Lance