Well you pay for basically 2 things, size, and space.
Maybe it's just me, but I don't want a clunky-ass MP3 player. Most of the cheap ones with big storage will be big and fat, like the size of a discman. Others have much less storage (can only hold like as much as a CD), but are tiny, and actually portable.
Also, consider some things like a radio. Sometimes using the radio is a really nice feature. Some MP3 players also have room for an expansion card, to hold more songs if you wanna pay the cash.
Another thing to consider is what do they run on. Some run on normal batteries, others come with rechargable ones that charge when on the cradle, but can also be replaced with normal ones on the road if they die. Others only have internal batteries and you're screwed if it dies.
Now for my recommendation...
Apple iPod! Comes in 5GB and 10GB form. It's one of the smallest MP3 players on the market. It hooks up via FireWire so it transfers at high speeds, and also charges through the cable. It can be used as both a MP3 player and a portable FireWire HD. Also, it has one of the best (if not
the best) organization metheods out there. The way you set up your playlists and everything is amazing. Also, there is an adapter so you can charge it without a FireWire computer nearby. 10 hour battery. On the 5GB model, you can travel from NYC, to LA, and back without ever hearing the same song twice. The 10GB model is insane, but mainly for peeps who want the portable HD. Also can display your contact list from MS Entourage, Palm Desktop, or Mac OS X address book. Did I mention the nice sized screen and backlight?
Only works on Mac. But I believe there are some programs that allow you to use it on Windows, if you have a fireWire connection. Best damn MP3 player ever made, only thing it needs is a FM radio.