While trouble shooting an IPv6 issue, I discovered that "ping6 <my_ipv6_host>" was getting very delayed responses.
After running tcpdumps and every other debug method I could think of, I finally determined with the help of a friend, that reverse DNS was not setup for <my_ipv6_host>. After fixing this, the ping6 time was like it should have been.
So, do you have a slow ssh or slow ping6 times?
Check for valid reverse DNS.