Host * UseKeyChain noAs you can see in the radar report, deleting keys using "ssh-add -D" seems to be just as problematic and confusing as it is with gnome-keyring, i.e. "All identities removed" is a lie.
For deleting already saved passwords and re-instating the El-Cap ssh behavior see here.