C# SAPI in a web service

var speechEngine = new SpVoiceClass();
SetVoice(speechEngine, job.Voice);

var fileMode = SpeechStreamFileMode.SSFMCreateForWrite;
var fileStream = new SpFileStream();
try
{
    fileStream.Open(filePath, fileMode, false);
    speechEngine.AudioOutputStream = fileStream;
    speechEngine.Speak(job.Script, SpeechVoiceSpeakFlags.SVSFPurgeBeforeSpeak | SpeechVoiceSpeakFlags.SVSFDefault); //TODO: Change to XML
    //Wait for 15 minutes only
    speechEngine.WaitUntilDone((uint)new TimeSpan(0, 15, 0).TotalMilliseconds);
}
finally
{
    fileStream.Close();
}

This exact code works in a WinForm app, but when I run it inside a webservice I get the following

System.Runtime.InteropServices.COMException was unhandled
  Message="Exception from HRESULT: 0x80045003"
  Source="Interop.SpeechLib"
  ErrorCode=-2147201021

Does anyone have any ideas what might be causing this error? The error code means

SPERR_UNSUPPORTED_FORMAT

For completeness here is the SetVoice method

void SetVoice(SpVoiceClass speechEngine, string voiceName)
{
    var voices = speechEngine.GetVoices(null, null);
    for (int index = 0; index < voices.Count; index++)
    {
        var currentToken = (SpObjectToken)voices.Item(index);
        if (currentToken.GetDescription(0) == voiceName)
        {
            speechEngine.SetVoice((ISpObjectToken)currentToken);
            return;
        }
    }
    throw new Exception("Voice not found: " + voiceName);
}

I have given full access to USERS on the folder C:\Temp where the file is to be written. Any help would be appreciated!

I don't think the System.Speech works in windows service. It looks like there is a dependency to Shell, which isn't available to services. Try interop with SAPI's C++ interfaces. Some class in System.Runtime.InteropServices may help on that.