Should the BCL add an IEnumerableWithCount<T>?

NB: I’ve written and thrown away this blog post twice before in the past, as I would convince myself it was in the category of micro-optimization, but I’m finally going to post it this time, if only to get feedback to confirm that it’s a bad idea. Smile

There’s been a few times when it would be nice to have something be IEnumerable<T> but keep the data on how many items will be in the ‘stream’.  Certainly there are many cases where that’s not known ahead of time, and for those our existing IEnumerable<T> works perfectly. Smile

One thing in particular about IEnumerableWithCount<T> is that it could still be covariant like IEnumerable<T> is since we would only need to add a property with a getter.  That is a big difference from ICollection<T>, which is the current ‘lowest-level’ option for a collection interface that includes count.  ICollection<T> means you have to support lots of ‘write’ operations (Add/Clear/Remove) so it’s not what we want if we just want to have a ‘stream’ that has the additional ‘metadata’ of the count of items.

Even more more importantly, IEnumerableWithCount<T> would still support deferred-execution since it inherits from IEnumerable<T>

One key bit is that this interface will be able to ‘go between’ IEnumerable<T> and ICollection<T> – ICollection<T> (and therefore IList<T> and so forth) could also implement IEnumerableWithCount<T> (would just need to declare it, since it already has a Count property that matches).  This means most (if not all) of the collections we’re used to will get this interface ‘for free’.

Somewhat unfortunately, the canonical example of where an IEnumerableWithCount would have benefit is in the Select method of LINQ.

Much of LINQ-to-Objects includes interface checks to optimize performance if the input object supports a higher-level interface than the method requires.

For instance, let’s look at ToArray:

The real work is done in the ctor of Buffer, which is where we can see the first place that would use our new IEnumerableWithCount<T>:

This is where we see one of the places that could change if we added the new interface.

Currently the perf optimization is based on checking for ICollection<TElement> – if that interface is implemented by the source, then we get to skip all the reallocations (since we don’t know the final size) and create the final array directly.

The code also uses ICollection<T>’s CopyTo method to copy the actual elements, and since that seems out of scope for an IEnumerableWithCount, we’d probably just have 2 paths and IEnumerableWithCount would foreach, so that situation would be a little less CPU-efficient (but still better than today’s existing IEnumerable path in the code, since , but saving the reallocations (and generated garbage) is still a big potential perf win.

For the LINQ benefit in particular, we’d likely want a static System.Linq.EnumerableWithCount (similar to Enumerable and Queryable) that implemented the particular methods that can be done ‘better’ with count information.e

What methods would our EnumerableWithCount static class support?  These are the methods being used with LINQ-to-Objects that would benefit from having the compiler using an implementation by our EnumerableWithCount class instead of the current implementation in Enumerable.  If it’s not in this list, then falling back to the Enumerable version is fine (much like some methods aren’t implemented by Queryable), which is what the compiler will do for us.

  • Any method that doesn’t affect the number of items when it acts on the enumerable, of course
    • Select is the method that sticks out, certainly
    • OrderBy / OrderByDescending
    • ThenBy / ThenByDescending
    • Reverse
    • Cast
  • Concat can also benefit, as we’d be able to know the target item count in such a situation without enumerating (so concat’ing multiple inputs and then ToArray would also save all those reallocations)
  • Similarly, Skip and Take would benefit, since we’d know the resulting count due to both – in paging scenarios, for instance, this is very common
  • Count/LongCount is similar in its benefit, as it would be able to just return the Count property without actually enumerating.
  • DefaultIfEmpty would know if it’s already empty or not
  • Zip would know the final count (since it’s Min of the input counts)
  • SequenceEqual could check count mismatch first
  • Various methods that could either throw or return default(T) based on the count instead of having to iterate first, like:
    • ElementAt/ElementAtOrDefault
    • First/FirstOrDefault
    • Last/LastOrDefault
    • Single/SingleOrDefault (which would be able to throw for count > 1 without needing to enumerate twice)
  • ElementAt could throw on “index out of range” immediately without iterating
  • ElementAtOrDefault could similarly know whether to return default immediately
  • First/FirstOrDefault could similarly throw or return default without iterating
  • ToDictionary would know the final count and could preallocate buckets better
  • Range/Repeat/Empty would all be able to create this interface since the final count is known
  • Average would know the count to divide by without incrementing a counter 🙂

Some remaining questions:

  • If the compiler supported IEnumerableWithCount<T> (using the same ‘yield return’ syntax), how would it work?  Would the method declaration need an out param for count?
  • Should we create a matching IEnumeratorWithCount<T> as well?  I don’t think it’s necessary, but not really sure at this point.

Firefox feature I really miss in IE7

Sometimes, it’s the little things.

I love that Firefox shows you the title of the last time you loaded a URL – it makes it much easier to tell which page is the one you really want.  Like the “crosscity.com” entries below – without the title, I’d have to load each before figuring out which one was the one I wanted.

No such luck in IE7, though – all you get from the completion list is the URL itself.

One other point (not a feature I miss, though) – you’ll notice that in firefox the favicon (just to the left of the address you’re typing in) immediately blanks once you start typing – IE7 doesn’t do that, so while I’m about to go to a microsoft site, the favicon is still Google’s (see above).

recent hardware upgrades to the Vista box

So, as you may remember from previous postings, I was using the integrated graphics in this Dell Dimension 4700 box, and while they passed all the non-WDDM (IOW, hardware) specs for Aero (Pixel Shader 2.0, DirectX 9.0, 128MB, 32bpp, etc. etc.), no WDDM driver existed so no Aero for me. 

I was actually fine with that up until Windows Movie Maker wouldn’t run because if it.  Now, I could have gone a different route (use XP in a VPC, use Jessica’s XP machine, use something on the linux box, use a different app on the Vista box, etc.) but since I had a long-term goal of getting dual-monitor on the dell anyway, it gave me an excuse and outlet for some christmas gift cards.

Since I’m still pretty cheap, though, I picked up the Sapphire (ATI) Radeon X1300 card for $47 (newegg) and the second monitor was a Samsung 931B 19″ LCD for $230 (bestbuy.com with free shipping… that’s where my gift card was and it’s an easier return than a web-only shop :).  Oh, and since the Samsung didn’t come with it, a $15 DVI cable from Amazon (I had a Dr. Seuss book I needed to order anyway, so it was free shipping).  Since the card does dual-monitor via 1 analog and 1 digital connection (same as my setup at work), I needed the DVI cable since the Dell E193FP I already had is analog-only.

The only downside is I didn’t expect that the on-board fan for this card would be quite as loud as it is.  It’s not horrific, but the machine’s now definitely louder than it was before this card 🙂

lack of support for Vista / IE 7 on web sites

I wanted to check the local wal-mart’s advertised sales – clicking around, I got here

Unfortunately, clicking on “December Advertised Values” gets me a page saying I’m not using a supported “computer configuration” (both from Firefox 1.5.0.9 and MSIE 7.0)

http://walmart.richfx.com.edgesuite.net/catalog_walmart/december3_2006/error/error.aspx

  1. doing an explicit OS check seems dumb – hopefully they’re not doing that, but it’s certainly possible given the platforms Firefox runs on.
  2. hopefully they’re not checking specifically for IE 5 or 6 (there are published best practices)
  3. I’m curious exactly what the “Microsoft Windows web site” is going to tell me in terms of how to upgrade from Vista to 98, 2000, ME, or XP.
  4. Yes, Vista isn’t widely available yet.  However, a *ton* of sites have managed to update just fine for Vista and IE 7.  Anybody serious enough to be using Akamai (edgesuite.net in the URL) should know better.
  5. While the “User Agent String Utility” exists for IE 7, I was surprised that Firefox didn’t have easier User-Agent modification.  Yes, I know about Opera 🙂

recognize .wav file using System.Speech from .NET 3.0

It’s a tiny little app since they provide you the right methods in their object model (at least for .wav files 🙂

 

using System;
using System.IO;
using System.Speech.Recognition;

namespace Recog
{
    class Program
    {
        static void Main(string[] args)
        {
            if (args.Length != 1)
            {
                Console.Error.WriteLine("Usage: Recog <file.wav>");
                Environment.Exit(1);
            }
            string wavfile = Path.GetFullPath(args[0]);
            if (!File.Exists(wavfile))
            {
                Console.Error.WriteLine("Error: Cannot find file {0}", wavfile);
                Environment.Exit(1);
            }

            DictationGrammar dictationGrammer = new DictationGrammar();
            using (SpeechRecognitionEngine engine = new SpeechRecognitionEngine())
            {
                Console.WriteLine("Using default recognizer: {0} [{1}]", engine.RecognizerInfo.Id, engine.RecognizerInfo.Description);
                engine.LoadGrammar(dictationGrammer);
                engine.SetInputToWaveFile(wavfile);

                while (true)
                {
                    try
                    {
                        Console.WriteLine("Calling engine.Recognize");
                        RecognitionResult result = engine.Recognize();
                        if (result == null)
                        {
                            Console.Error.WriteLine("Exiting by way of Recognize returning null");
                            break;
                        }
                        Console.WriteLine("Recognized text of {0} [confidence {1}]", result.Text, result.Confidence);
                    }
                    catch (InvalidOperationException)
                    {
                        Console.Error.WriteLine("Exiting by way of Recognize throwing InvalidOperationException");
                        break;
                    }
                }
            }
        }
    }
}

quickie notes on Vista speech recognition through System.Speech (.NET 3.0)

Just to play around with it, I wrote a quickie app to call the .NET 3.0 interfaces for Vista’s speech recognition.  In particular, it accepts a .wav file as a param and just has the recognition engine run over that.

Some quick notes:

  • The default recognition engine isn’t necessarily the one you want.
  • On my system, I have 3 installed, one of which is the one from Office 2003 (still installed on the machine).  This appears to be the one I’m getting by default when I don’t specifically specify the recognizer to use.  Calling SpeechRecognitionEngine.InstalledRecognizers() gives, for me:
  • Installed recognizer: MS-1033-61-DESK [Microsoft English (U.S.) v6.1 Recognizer]  <– this is likely the one from Office 2003, and is the one I got by default
  • Installed recognizer: MS-1033-80-DESK [Microsoft Speech Recognizer 8.0 for Windows (English – US)]
  • Installed recognizer: MS-2057-80-DESK [Microsoft Speech Recognizer 8.0 for Windows (English – UK)]
  • When I’m constructing the SpeechRecognitionEngine, using “MS-1033-80-DESK” for the recognizerId string works fine and gets me the recognition I expect.
  • The API seems to assume you want call Recognize or RecognizeAsync multiple times.
  • While looping, there’s no way to check when to end (when to stop calling Recognize) except for checking when you get an InvalidOperationException.  Here’s some quickie PowerShell showing 2 valid calls and then the invalid one once we reach the end of the (short) wav file.  I really hate how end-of-file is an exception since it’s NOT an exceptional condition (at least in the pre-recorded audio sense – I understand why it is for a microphone).
    • C:\> [void][reflection.assembly]::loadwithpartialname(‘system.speech’)
      C:\> $rec = new-object ‘System.Speech.Recognition.SpeechRecognitionEngine’
      C:\> $rec.RecognizerInfo.Description
      Microsoft Speech Recognizer 8.0 for Windows (English – US)
      C:\> $rec.LoadGrammar((new-object ‘System.Speech.Recognition.DictationGrammar’))
      C:\> $rec.SetInputToWaveFile(‘C:\Documents and Settings\admin\Desktop\downloads\test.wav’)
      C:\> $rec.Recognize() | fl text,confidence
    • Text : This is a test 12345678
      Confidence : 0.6058533

      C:\> $rec.Recognize() | fl text,confidence

      Text : 910 this was a test
      Confidence : 0.6938771

      C:\> $rec.Recognize() | fl text,confidence
      Exception calling “Recognize” with “0” argument(s): “No audio input is supplied to this recognizer. Use the method SetInputToDefaultAudioDevice if a microphone is connected to the system, otherwise use SetInputToWaveFile, SetInputToWaveStream or SetInputToAudioStream to perform speech recognition from pre-recorded audio.”
      At line:1 char:15
      + $rec.Recognize( <<<< ) | fl text,confidence
      C:\> $error[0].exception.innerexception.gettype().name
      InvalidOperationException
      C:\>

  • Recognition on “bad” input may actually return null from Recognize (and depending on the size of the file, may block for quite awhile).  Here’s a normal (comes with Vista) wav file.  It doesn’t have any actual speech in it.
    C:\> dir $env:windir\media\start.wav
    Directory: Microsoft.PowerShell.Core\FileSystem::C:\Windows\media
    Mode  LastWriteTime     Length Name
    —-  ————-     —— —-
    -a— 8/23/2001 8:00 AM   1192 start.wav
    • Here’s recognition on that file returning null
      C:\> $rec.SetInputToWaveFile(“$env:windir\media\start.wav”)
      C:\> $output = $rec.Recognize()
      C:\> $output -eq $null
      True

    in Vista, Sound Recorder now forces WMA

    I wanted to make a simple .wav file for some testing.  I decided to use the Sound Recorder that’s been in Windows for awhile.

    Your only option for saving from the built-in Sound Recorder (assuming you’re not using the “N” edition for Europe et al) is WMA now – you can’t save as .wav any more.

    The help spells this out, but their phrasing of “By default” made it sound to me like there would be an option for .wav (and not just that the N editions used .wav still).

    Scanning around for something simple to convert the .wma files to .wav’s, this one seems simplest and least intrusive as a command-line app (that I can therefore most easily script).

    http://www.stud.uni-karlsruhe.de/~ua7i/wma2wav/

    http://www.stud.uni-karlsruhe.de/~ua7i/wma2wav/wmatowav.exe