The phonesm project feeds the compressed audio and video data to the operating system; it doesn't do any of the decompression work. If the device doesn't support the format (e.g., if the frame rate or resolution is too high), then it will not work. Supporting
another type of codec would almost certainly best be done with native code (C++ with some NEON thrown in) and--assuming that doesn't run down the battery too fast--that probably would be best as a separate project.
Does the stream meet the requirement according to the Windows Phone
What does the debug output say when you try to play the stream? The H.264 "Configuration" line would be of particular interest. For example, the default NASA stream shows:
Configuration H.264 "Baseline" profile, level 3.1 428x240
Does the stream play with the Win8.1 app?