So, for me it is more of a nuisance than a privacy concern.
Either way, changing a couple configs can certainly help.
Open Firefox and hit Tools \ Options and then the Privacy tab.
In the Clear Private Data box check Download History and any other download add-on you might have which appears.
For instance you’ve got the option to see I haveDownload Them Allenabled, now click OK. On the other hand, this is also a great privacy and security tip.
