If you do not like the sound of your mixes, limiting/compressing them is the last thing you should worry about. Mastering is a complex process involving a lot more than compressing and limiting. First and foremost, get the mix sounding as good as you can using individual track effects, usually but not limited to EQ, gates, compression, reverb, delay, etc., to target specific problem sounds. Test your mix on diferent speakers and in different rooms to help identify problems you may not be noticing.
A good mix will need very little adjutments made to it in the mastering phase, so if you are not an experienced mastering engineer, your best bet is to poilish up your song by tweaking the mix. If you want to experiment with mastering, keep in mind that there is no one thing that is the magical solution for all songs. Every song is going to need diferent adjustments to get it to where you want it. A typical mastering effects chain might go something like this (remember not all songs will need the same treatment):
EQ - Preferrably at least 10 bands. If you are finding yourself having to make EQ adjustments greater than +/- 5dB on any bands, you might need to go back into the mix and fix your problem there.
Multiband Compression - basically a series of compressors tuned to compress just certain frequency ranges. Allows you to compress certain frequency ranges more or less than others. For example, you might need to compress your low end to be tight and punchy, while allowing the high mids to shine through a little more, or vice versa.
Standard compression - After "evening out" the overall mix with eq and multiband compression this will allow you to compress the overall frequncy range if you so desired.
Sonic Enhancers - This is where you might decide to use something like the BBE Sonic Maximizer (remember it's NOT a limiter) or other effects such as Bass Enhancement and Harmonic Enhancers that will color and enhance certain frequencies by manipulating harmonics and phase.
Reverb - A SLIGHT touch of the right reverb over the whole mix can "pull it together" and give it a nice polish. Typically use shorter decay times here, you should be going more for a feel than an audible reverb. Any reverb that you want to stand out more should be done in the mixing phase.
Limiter - This will allow you to raise te average volume of the song to get it up to that "radio ready" loudness. Make sure that you are happy with your eq and multiband compression befor applying the limiter, as you can get undesired results when limiting an "uneven" frequency range. You dont want the limiter to act more on the low end for example if your low end is a lot louder than the rest of the frequencies.
Visual feedback - This is where you might want to insert a spectrum analyzer or phase meter or other visuals tool to help you identify any problems you might have missed such as phase problems or excessive "bumps" in the frequency spectrum. Particulary helpful if you are not working in a professionally designed mastering room, and/or working with sub par monitors. Keep in mind that you should always trust your ears FIRST. The visual feedback should not be a crutch, it should only be a tool.
As I mentioned, this is by no means set in stone. Different mastering engineers prefer different techniques and tools, but this would be a good place to start for anyone wanting to experiment with mastering.
Remeber that when mastering (and mixing for that matter) LESS IS MORE. Be sure to make small adjustments and listen carefully. If you are having to make adjustments that seem to be large adjustments, you probably need to go back into the mix and fix the problems there. Also, if you are not hearing the difference in what a certain setting is doing, then don't adjust it. Without a professional ear, equipment, and room, it can be VERY hard to hear some of the subltle changes involved in mastering. If in doubt, don't adjust it, you might do more harm than good.
A lot of people will tell you not to attempt mastering without a professional, which IS good advice, but then again, the best way to learn is to experiment, and the more you understand about the process, the more it will help if and when you do decide to take it to a professional.
Some of the best mastering jobs I have done were the ones that were already great mixes when I got them. Remember, you can't shine shite... a little air freshener might cover up the smell a little, but the stink is still there. Make sure your mixes sound great BEFORE you try to mess with any mastering. A good mix might not need much mastering at all.
