'Wear' is a somewhat misleading term. Permanent magnets gradually decrease in strength, but this has very little to do with usage. The "life" of a permanent magnet depends on many factors. Naturally occuring forces conspire to knock the little domains out of alignment. But this is normally a very slow process. Temperature is a major player in this process. The higher the temperature, the faster this process will occur. Extreme heat [surpassing the curie point] will immediately randomize the domains. A sharp blow can also knock domains out of alignments, as can other nearby magnetic or electrical fields. Radiation can also knock domains out of alignment. But again, under normal conditions, neither your fridge or you will live long enough to watch the little fellow fall to the floor in exhaustion. At the quantum leve [e.g., electrons] magnetism is eternal.