The problem is noise

• What is the signal to noise ratio?
• How many microphones and noise sources do you have?
• Is the noise constant and predictable or changing?
• How speech-like is the noise?