maybe, but changing port will lower (almost) to zero all the problems, meaning that all script kiddies and bots won't affect you..
obiously a portscan will reveal your real sshd port, but it's something that can't be done in a silent way.. in this case, fail2ban is a great help
BTW, if only key access is enabled, i.e. no password auth allowed, using a not standard port will let you sleep like a baby
all IMHO