If you face the same problem, I assume that:
- you are using Ubuntu 14.04 or later version.
- your keys are properly set.
First you need:
sudo apt-get install keychain
Then add this line in your ~/.bashrc, ~/.zshrc, or whatever *sh you used:
eval $(keychain --eval --agents ssh -Q --quiet ~/.ssh/id_rsa)
Sometimes, you need change
~/.ssh/id_rsa to the path of your key.