On OSX
Open up GPG keychain and click through the scary warning to delete the secret keys. If you set it up right these are only stubs, the actual key is on the yubikey. Once you've done that, insert the key you want to use and get the stubs recreated with:
$ gpg --card-status