So, you've got the latest and greatest DAW, a high end notebook or desktop computer, every plugin you could lay your hands on, some cool looking "studio" monitors and headphones, and you watched every episode of Pensado's Place, but your home studio mixes still suck compared to the pros? What could you possibly be doing wrong?
Let's start at the source and go through all the possible reasons your mixes might suck, and what the pros do differently.
Maybe it's just that you and/or the musicians you are working with simply can't deliver a great sound to begin with. Sometimes you just have to be brutally honest with yourself, and recognize that you (or the musicians you work with) don't have the skills and proficiency yet to make your instruments/parts sound great on their own. Part of this can also be the quality of your instruments. And, by "instrument", I also include vocals, as the voice is usually the most prominent instrument in any song with vocals.
You know why most recordings and mixes that the professional engineers work on sound great? Because they aren't working with beginners! For the most part, they are working with seasoned professionals who have already put in the time to learn how to get great sounds out of the instruments, and who can also afford professional quality instruments and keep them maintained properly.
Back before the home recording boom, when I was just starting out at the major studio I worked at, I often got the all night shifts recording low budget bands that didn't have a lot of experience (or money) yet. Many of these bands would come in and say something like "We want the drums to sound like {insert some famous band}". Similarly, the would usually say the same thing about the guitars and bass. Problem was that none of these bands could play like the musicians in those famous bands they cited. Additionally, their instruments were also rarely up to the quality level that those famous bands could afford. I'm a pretty damn good recording and mixing engineer, but I can't perform miracles!
This is something I often run into with amateur musicians, especially solo artists that do everything themselves. You can't get a great sounding mix if you don't have a great arrangement that isn't over-crowded with too many parts. I have covered this twice already, so I will try not to repeat myself yet again, but it's VERY important. Check out these posts:
Less is More - Bigger sounds through simplification
Better Mixes through Subtraction
Let's assume that you have the source nailed down, your chops and instrument are pro level, and you have written a great song with a great musical arrangement. What about your recording space? If you are recording any instruments, or vocals, with a microphone, then the room you record in plays a major role in the sound that will be captured. I would venture to say that in many cases the room itself is far more important, and has more impact on the sound, than the microphone and the rest of the gear in your recording chain. If you are recording in any kind of room in your home, or even your garage, and you don't have any kind of proper acoustic treatment, then you are more likely than not getting the sound of your room into your recording. In addition to boxy sounding small room reverb and early reflections, the dimensions of your room will cause all sorts of nasty frequency problems if you have parallel surfaces (as almost all homes do).
That small room reverb type of sound will kill the definition of most anything you record with a microphone, and make it extremely hard for you to get anywhere close to that up front and present "in your face" vocal sound. It will always sound exactly like it was recorded in a small room, and will smear the sound.
Pros know better, and they record at major studios that were acoustically designed by professional studio designers, and usually have put a HUGE amount of money into just the construction and acoustics of the studio, to get the best sound possible. Even with many modern engineers working from home based studios these days (myself included), they still know enough to put a large amount of their budget into the acoustics to take the room out of the equation (or at least make it sound good). I'm not talking about throwing up a bunch of acoustic foam products, or egg cartons either. If you don't know what you're doing, hire an acoustic consultant to help you (I did when doing the remodel for my studio in our home). Still, when I need to record a large group, or something where I want a nice natural room sound, I will rent out major studios in the area to do the recording.
However, there are some things you can do to counteract the room sound a bit if you can't invest in a lot of proper acoustical treatment. It's a bit counter-intuitive, but when recording things like vocals in an untreated room, you usually do NOT want to use a large diaphragm condenser microphone! Those are too sensitive, and they will pick up too much of the room sound. Instead, use a dynamic microphone and work it as close as possible so that the signal you want is much louder than any reflections from the surfaces in your room. Similarly, position yourself as far away from any walls or reflective surfaces as possible. It has been said the Bono, from U2, likes to record his vocals in the control room using a lowly handheld Shure SM58, without headphones (listening through the studio monitors). That's the kind of microphone that you need to think of when recording at home. If you have a great voice, and know how to properly work with the microphone, you can get great vocals with just an SM58, and not have to worry about too much bleed into the microphone of room reflections and other sounds.
This works for things other than vocals as well. Guitar amps can be recorded up close with something like an SM57 (still the first choice in many major studios as well) and not worrying too much about room sounds since the amp will usually be MUCH louder. However, even with that, you may need to move the amp around, or even get it up off the floor, to change how the sound of the amp is affected by nearby surfaces and get a better recorded sound.
I'm almost hesitate to put this in here, because even most of the cheap recording gear these days still can sound fairly decent, and the items above are usually much more important than the actual gear that you use. But, there is definitely some crap gear out there that can definitely affect the sound quality of your recorded tracks, thus making it difficult to get a good sounding mix. It's much easier to get a great sounding mix if you have great sounding tracks to begin with! If you're still using adapters to plug your microphone or other gear into the little 1/8 inch audio input on your computer, it's time to at least start looking at a decent audio interface for your computer (USB, Thunderbolt, Firewire, etc.).
Let's face it, even if you have a handle on everything above, it takes a LOT of time, practice/experience, and a certain amount of natural talent to become a really good mixing engineer! There is a reason that most successful band/artists don't do their own mixing -- they simply don't have the time to dedicate to getting great at that, while still having time to be great musicians, songwriters, and performers. It's difficult to be a master at everything, so you have to decide what's important to you, and maybe let some other professionals handle the rest.
If you're more of an engineer than an artist, and your goal is to be a great recording/mixing engineer, then you need to put in the time! Just like you don't pick up a guitar and suddenly turn into a David Gilmour (or whoever your favorite guitar player is), you can't just buy some gear and suddenly become a great mixing engineer. It takes many years of practice and experience to get really good at any artistic endeavor, and mixing is no exception. It's fairly easy to learn the technical side of things, but it's also an art form that is not easily mastered.
In addition to putting in the years of practice it takes to get good at mixing, there is definitely a certain amount of natural talent involved, that not everyone has. Some people pick it up much faster because of their ability to really listen and make the right mixing choices for the song.
Best thing you can do to get better, is to read all the mixing advice you can find from pros that you admire, and then practice mixing every day (just like you would practice an instrument). Solicit bands/artists on the internet to send you their tracks to mix for free while you are learning. Not only will that give you a variety of material to practice with, but you'll also learn how to communicate and work with bands/artists, to try to deliver something that they like (as they will almost always ask for some changes to your mixes).
I already wrote an article on "The Secrets top Mixing Engineers don’t want you to know!", but I'll summarize the main "secret" here.
The main reason that the pros get great sounding mixes is that everything except the last point in this article (practice/experience & natural talent) has already been taken care of for them! Most of the time, the pro mixing engineers get great sounding tracks recorded by other professional engineers in great sounding studios with great sounding gear, along with great musicians who really know how to play (and have great sounding instruments), as well as great songs and arrangements (well, not always). Combine all of that with the years of experience and the natural talent of the professional mixing engineers, and it's hard not to get a great sounding mix!
It's going to be very hard to compete with the pros if you are trying to do everything yourself out of a bedroom studio. It can be done, but it's definitely much more difficult! So, go through the list of points above, and do your best to get everything under control as much as you possibly can, and then just keep practicing! Good luck!