I was able to get pulseaudio to finally work with Wolfson card.
The persistent problem was that when using either the wolfson distro, or ragnar's pulseaudio insisted on having the on-board bcm2835 as the default sound. I tried various ways, modifying pulse's configs to either remove the bcm2835 or elevate the wolfson to default status, but to no avail. Nor could I simply unload the bcm2835 like a kernel module, since it was compiled into the kernel. Funny thing is, no sound came out of the onboard sound, even though pulse had it as default. Strange...
I just recompiled the 3.10.25 kernel and made bcm2835 into a kernel module, and then commented it out in /etc/modules. Rebooted and now pulseaudio was forced to make the wolfson the default. The 16/44k flacs played fine. But pulse regularly chokes with hi-res files because it tries to resample down to 44k, so to play the sampler provided by wolfson, you should set the alternate-sampling-rate to 96000 in /etc/pulse/daemon.conf, and now everything plays fine. I don't know if I have to up that to 192000 for 192k files, but I don't have any such music to test.
I should have gone this route long ago, but didn't have the patience to compile the kernel on the Pi. Can those of you who compile the kernel a lot tell me what the best way to cross compile it on my Mac would be? I tried a few methods outlined on some blogs, and they didn't work. It would be great if I can speed up the compiling process, so that I could then look into optimizing the kernel and all of Raspbian for working with the wolfson card.