There are two situations where you can move down in the Ranking:
1) You 'Swap' out a photo with votes.
2) Other people get more votes, example:
You have 500 votes and you're 10th place.
Someone else had 499 votes and was in 11th place.
The second person just received votes and now has 505 votes.
They move into 10th place and you "drop" to 11th.