Removing mods would be pretty bad in almost all circumstances, can make all sorts of mess.
A few years back my friend had a modded minecraft server for us to play on, after a few weeks playing one modpack we decided that although fun we wanted to play a different one. He forgot to delete the world first, resulting in lots of mods from the first pack being removed and more added. The result was that my house was turned to air as the modded blocks dissapeared, and a forest made from trees from a mod in the first pack was turned to ore as another mod claimed the now unused block ids. Was quite a sight to see a forest where the leaves were varios kinds of ore. A modded server meant to be long lasting would likely need to choose one minecraft version and stick to it, as not all the mods would be updated and removing them could be disasterous.
Anyhow I havn't played modded for ages, so any examples I give are likely to be very old and possibly not updating anymore, like it turns out Twilight Forest is.