I'm one of the few here, who is pro the Olympics*, driving along the A13 it does look impressive. There is a lot of investment going into Stratford, the new Westfield etc, however Tottenham are North London, it makes no sense for them to move East. Let West Ham have it, and allow the stadium to be a monument to the Olympics.
However next to Anish Kapoors sculpture, any stadium will look plain.
Don't forget that Crossrail will make East London a lot more accessible for everyone.
The London Marathon shows that population of East London will get behind major sporting events.
*I'm also getting a bit giddy about the royal wedding