There are two situations where you can move down in the Ranking:
1) You 'Swap' out a photo with votes.
2) Other people get more votes and pass you, for example:
You have 500 votes and you are 10th place.
Someone else has 499 votes and is in 11th place.
The second person just received votes and now has 505 votes.
They move into 10th place and you "dropped" to 11th.