What is the most heavily touched block of information on a FAT file system? The File Allocation Table. When you do a quick format of a FAT file system, it's the File Allocation Table alone that is re-written; that's why you can still recover data from the volume.
The File Allocation Table (FAT) is always written in the same logical location; it has to be. So on a flash media device, the part of the device that is most heavily used and written to is the part that contains the FAT. If you always reformat the card after every use, you are adding yet another read, erase, write cycle to the FAT block.
What causes a flash media device to "wear out"? The number of write cycles to the various physical locations on the device. The usual number of write cycles before a location becomes unusable (aka unreadable) is in the hundreds of thousands ... far more than most uses ever require for the practical use life of the devices thus far since most flash storage devices are replaced in favor of larger/faster/newer design one long before location "wear" has had an impact.
But formatting a flash media device every time it is used is just adding to the normal, expected use model. It is possible that you are increasing the probability of failure by reformatting every time since you are exercising write cycles to the most used portion of the device every time you format it.
Some file system software can relocate the FAT block when it is formatting the device, but most does not. I believe the SDFormatter app does since it analyzes the card and optimizes the creation of its file system, and the SD card format protocol includes a table of write counts and locations external to the file system. Most in-camera formatting is not so sophisticated.
I format cards for all my cameras periodically with SDFormatter. If I need space on a card that's getting too full, I erase (not format) already archived files that are on it. File by file erasure writes far fewer locations in the FAT block than formatting.
