If you face the same problem, I assume that:

• you are using Ubuntu 14.04 or later version.
• your keys are properly set.

First you need:

sudo apt-get install keychain

Then add this line in your ~/.bashrc, ~/.zshrc, or whatever *sh you used:

eval $(keychain --eval --agents ssh -Q --quiet ~/.ssh/id_rsa)

Sometimes, you need change ~/.ssh/id_rsa to the path of your key.

And enjoy.