You have to have cameras at the angle you want to view or enough cameras to cover a continuous FOV/Field of View.
So if a referee could wear this for example, you could look around with a headset but could still only see it from the angle he is standing...just that you could look behind him too:
Moving your head would change the view giving you the feeling you are there though. Might as well stream it straight to your phone as with cardboard or cheap VR headsets you can get the effect now. Some of the VR apps on the play store let you download concerts where you are viewing it as if you are there.
Virtual reality porn appears to the biggest up take of the technology so far.
Doing it for sports is possible and multiple positions by having multiple camera setups is too. Just how long before it jumps from phone to sat or cable box's. They might even all be IPTV by that time anyway, so will end up streaming to a phone with headset over wifi anyway.