tldr: This is something of an eternal debate.
IMHO very small imaginary frequencies can be okay, but it depends on your system and needs.
As you might see from the various comments above, there are often different opinions on whether very small imaginary frequencies matter. The truth is, that it depends a bit on the size of the molecule and what you plan to do.
In principal, if you've reached a true local / global minima of the potential energy surface, there should never be imaginary frequencies, because if the second derivative is negative, you're not at a minima with respect to that normal mode.
In practice, for medium to large molecules it can be very hard to completely minimize and find a true minimum. Most geometry optimization methods include various routines to only stop when the forces are very small, the change in energy is very small, etc. But these don't guarantee a true minima, which is why you should calculate frequencies and check (which you mention in the question).
Remember that the potential energy surface has $3N-6$ dimensions for non-linear molecules. For 10 atoms, that's already 24 degrees of freedom. In many molecules, these may be correlated (e.g., think about twisting a dihedral angle in a protein - you might smash into another atom). Many times the potential energy surface can be close to flat, so finding the exact minima is time-consuming.
- You should check for numerical noise, try to push for better optimization tolerances, integration grids, convergence, etc.
(It seems like you've done a lot of this based on your comments.)
- It depends on your properties / needs. If you need rigorous thermochemistry, then there may be some energy error between your current geometry and a true minima. In that case, do what you can to remove the minima.
In your case, in my experience, the property calculations you plan are relatively insensitive to the small energy / geometry difference. Consider if the atomic positions move $0.001Å$ will the polarizability change much? Probably not. Similar story with TDDFT, which usually has a ~0.1-0.2eV error bar.