First, it is not clear which DOS you are referring too? Is it Green function, spectral function or bulk DOS? In this case, try and plot them all!
There are several aspects that determine the smoothness of the DOS and transmission plots:
- The $\eta$ value that determines the imaginary part of the Green function, both for the device part and the electrode part, a larger value will smooth out the DOS, at the cost of precision similarly to the electronic temperature in the fermi-dirac distribution. You should play with both $\eta$ values to see how it influences the DOS/transmission
- The k-point resolution, you'll generally see some van-Hove singularities in the pristine graphene case, again, if you increase the k-point sampling you'll reduce the peaks and smoothen out the functions.
- The transmission is generally less spiky as it involves both incoming and outgoing states, so the numerical influences of the $\eta$ becomes more apparent.
You should play with the $\eta$ and k-point sampling values to see how they are all connected.