My question regards the additive or non-additive nature of frequencies.
For example, if I have 2 frequencies at 500 Hz each at -6 dbs, and maintaining that same volume, if both frequencies were played simultaneously, would the result be -4 or -3 dbs at 500 Hz?
Keeping that same scenario in mind, what if 2 frequencies at 500 Hz were played simultaneously, but one sound source comes in a -10 dbs and the second one comes in at -6 dbs. If both were played simultaneously, would the results be additive again, or would you only hear the second, or loudest, source? Would the 2 combine?
If these frequencies combine, is there a formula for determining what the end result would be when playing 2 sound sources with similar frequencies? How do you know what you end up with?
With this in mind, when does one frequency "mask" another and when does one "add" with another, if ever?
That's a lot of questions!!
The big simple answer to all these questions is "it depends"!
What it depends on is the phase relationship of the signals that are combining, as well as the type of waveform each signal is.
It's really hard to describe and demonstrate this without using some graphs/drawings, so I searched around the net and found a few articles to help you out. Most of these articles deal more specifically with phase relationships and how it affects audio. But, once you understand phase, then you'll basically know the answers to your questions.
The most helpful and detailed article I found is on the Sound on Sound web site here:
http://www.soundonsound.com/sos/apr08/articles/phasedemystified.htm
And here are a couple shorter articles with some simple illustrations as well:
http://www.audiogeekzine.com/2009/07/techniques-for-dealing-with-phase/
http://www.audiocourses.com/?s=Phase
The best way to show and illustrate these ideas is with simple sine waves, which are the most pure tones and the simplest waveform, which is what most of the articles show. However, nothing you record in the real world is that simple, and the first article on the SoS web site really goes into the most detail on real world examples.
Regarding your last question, masking is a lot more complicated. If you have two pure sine waves of the same frequency, then "masking" doesn't really apply as they will simply combine to make a louder or softer sine wave of the same frequency depending on their phase relationship, as described above. You may have wanted to say canceling instead of masking?? As described in the phase articles, if two identical waveforms are exactly 180 degrees our of phase (or if one waveform is inverted) then they will perfectly cancel each other out when summed. At any other phase relationship, the resulting sine wave may be softer or louder than either one by itself, depending on their exact phase relationship.
The term "Masking" as commonly used in audio is something completely different and is more of a psycho-acoustic effect of the way that we hear things, and has little to do really with the way waveforms add or subtract from each each other. If that's what you were really interested in, then just do a search for audio masking on Google and you'll have lots to read about. This type of Masking is the main principle behind MP3 encoders... in very simple terms they figure out what frequencies are going to be masked out by other frequencies so that they can remove that masked audio information from the source and make the files smaller. Here is the wikipedia page for masking (I haven't read it, and I don't always trust wikipedia, but it's a good start):
http://en.wikipedia.org/wiki/Auditory_masking
Hope that gets you started.