Wednesday, February 25, 2015

Packaging an existing virtualbox vm for use with vagrant

It's not super-obvious from the vagrant documentation, but you can use the "vagrant package" command to package up an existing VirtualBox VM. This will shut down the VM, create the metadata.json, and zip up the disk. To use it you need to know the name that VirtualBox has for your VM, list them with:
$ VBoxManage list vms
"ubuntu-lucid64" {aaaabbbb-cccc-dddd-1234}
"CentOS build" {aaaabbbb-cccc-dddd-1234}
Package up the vagrant VM with:
$ vagrant package --base "CentOS build" --output centos_5.11_64
$ vagrant box add centos_5.11_64 --name centos_5.11_64

Friday, February 20, 2015

Workaround for broken vagrant up ssh "unsupported encryption type"

Vagrant is still not playing nicely with SSH certificates loaded into ssh-agent. In my case this seemed to only be a problem during provisioning (i.e. "vagrant up"), using "vagrant ssh" after the box was up worked fine. The error is:
The private key you're attempting to use with this Vagrant box uses
an unsupported encryption type. The SSH library Vagrant uses does not support
this key type. Please use `ssh-rsa` or `ssh-dss` instead. Note that
sometimes keys in your ssh-agent can interfere with this as well,
so verify the keys are valid there in addition to standard
file paths.
You can try clearing out some keys from ssh agent with:
$ ssh-add -D
All identities removed.
Except ssh is probably lying if you're running goobuntu, the keys are still there. There's all sorts of confusion about this behaviour, which seems to be the fault of gnome-keyring, which allegedly only allows you to delete manually added keys. If your SSH certs are automatically loaded it seems like you're out of luck.

By far the easiest workaround is to simply temporarily disable the ssh agent:
SSH_AUTH_SOCK="" vagrant up

Python: distinguish strings from other iterable objects

I often want to do a sanity check on a parameter that is supposed to be an iterable. The problem is that strings are also iterable, but iterating over the characters in a string is basically never what I want. This will raise if I'm passed a string or something that isn't iterable.
import collections

def CheckForIterable(someparam):
  if (isinstance(someparam, basestring) or not isinstance(someparam, collections.Iterable)):
    raise ValueError("Expected an iterable, got %s" % str(someparam))
In [50]: CheckForIterable("a")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
 in ()
----> 1 CheckForIterable("a")

 in CheckForIterable(someparam)
      1 def CheckForIterable(someparam):
      2   if (isinstance(someparam, basestring) or not isinstance(someparam, collections.Iterable)):
----> 3     raise ValueError("Expected an iterable, got %s" % str(someparam))

ValueError: Expected an iterable, got a

In [51]: CheckForIterable(["a"])

In [52]: CheckForIterable((1,2))

In [53]: CheckForIterable((1))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
 in ()
----> 1 CheckForIterable((1))

 in CheckForIterable(someparam)
      1 def CheckForIterable(someparam):
      2   if (isinstance(someparam, basestring) or not isinstance(someparam, collections.Iterable)):
----> 3     raise ValueError("Expected an iterable, got %s" % str(someparam))

ValueError: Expected an iterable, got 1

Tuesday, February 10, 2015

Building an OS X vagrant VMware Fusion VM

Install VMWare Fusion, vagrant, pay money to buy the vagrant vmware plugin.  Follow the instructions in the email to install the plugin and download and install the license, something like:

vagrant plugin install vagrant-vmware-fusion
vagrant plugin license vagrant-vmware-fusion ~/license.lic
This blog has great instructions to build the OS X base box. Boiling it down:
  • Install the base OS in VMWare Fusion with user/pass of vagrant/vagrant. This is fairly easy with modern VMware Fusion and an installer dmg.
  • Install VMWare tools from the VMWare menu.
  • Install all updates:
    sudo softwareupdate --install --all
  • If you want to be able to install and use Homebrew, install the relevant Command Line Tools for XCode from apple (developer account required).
  • If you want to be able to make packages, copy PackageMaker.app to /Applications from the Auxiliary Tools for XCode Late - July 2012 from apple (developer account required).
  • Enable remote logon (i.e. SSH) via System Preferences > Sharing
  • Setup passwordless sudo, vagrant ssh, and ssh config as vagrant recommends.
  • If you created any snapshots during this process, delete them
  • Set the free disk space to zero (not sure this is actually necessary, but the original post claims better compression):
    $ diskutil secureErase freespace 0 Macintosh\ HD
    $ sudo halt
    
  • Defrag and compress as vagrant recommends:
    cd ~/Documents/Virtual Machines.localized/OS X 10.8.vmwarevm
    /Applications/VMware\ Fusion.app/Contents/Library/vmware-vdiskmanager -d Virtual\ Disk.vmdk
    /Applications/VMware\ Fusion.app/Contents/Library/vmware-vdiskmanager -k Virtual\ Disk.vmdk
    
  • Copy the relevant files to a new directory:
    mkdir ~/vagrantbox
    cd ~/vagrantbox/
    cp Virtual\ Disk.vmdk OS\ X\ 10.8.nvram OS\ X\ 10.8.vm* .
    
  • Create a basic metadata.json
    echo '{"provider":"vmware_fusion"}' > metadata.json
    
  • Zip it up:
    tar -cvzf OS_X_10.8.5.box ./*
    
  • Add it to vagrant:
    vagrant box add OS_X_10.8.5 OS_X_10.8.5.box
    
  • Add a config to your Vagrantfile:
      config.vm.define "OS_X_10.8.5" do |box|
        box.vm.box = "OS_X_10.8.5"
        # Random keypair generation and insertion doesn't seem to work on OS X
        box.ssh.insert_key = false
        box.vm.provider "vmware_fusion" do |v|
          v.vmx["memsize"] = "4096"
          v.vmx["numvcpus"] = "2"
        end
      end
    
  • Try it out:
    vagrant up OS_X_10.8.5
If you're seeing this error message you didn't install the vmware fusion vagrant plugin:
The provider 'vmware_fusion' could not be found, but was requested to
back the machine 'OS_X_10.8.5'. Please use a provider that exists.

Tuesday, January 13, 2015

Python: Find the first true element (i.e. one that matches a condition) in an array

Say you want to check if any value of one array is in another array, and you only care if there is any match at all. Not what the value is, or how many matches there are. Once you have found a match you want to stop processing. Here's a one-liner that will do that and return bool if there is a single match. In my case both arrays already only held unique values.
In [34]: a=[1,2,3]

In [35]: b=[6, 8, 2, 5, 3]

In [36]: next(ifilter(lambda x: x in b, a), None)
Out[36]: 2

In [37]: b=[6, 8, 5]

In [38]: next(ifilter(lambda x: x in b, a), None)

Friday, December 12, 2014

Skipping a python unittest with a custom decorator

Example custom Python unittest skip decorator:

import functools
import glob
import os

def skipIfNoYamlFiles(f):
  @functools.wraps(f)
  def wrapper(*args, **kwargs):
    yaml_glob = os.path.normpath(os.path.dirname(__file__) + "/../../definitions/*.yaml")
    if glob.glob(yaml_glob):
      return f(*args, **kwargs)
    else:
      return unittest.skip("No YAML found with %s, skipping this test." % yaml_glob)
  return wrapper
Use it like this:
class MyTests(unittest.TestCase):

  @test_lib.skipIfNoYamlFiles
  def testValidation(self):
    """Ensure all Yaml validates."""
    print "test code goes here"

Wednesday, December 10, 2014

Migrating Chrome Profiles to per-channel directories

If you run custom chrome profiles you might see a message like this in the terminal if/when you launch chrome from the commandline:
[9894:9894:1210/140235:WARNING:sxs_linux.cc(139)] User data dir migration is only possible when the profile is used with the same channel.
It seems chrome recently made a change to where user profiles are stored. Previously different tracks (stable, beta etc.) all used the same profile like:
~/.config/google-chrome/myprofilename
Now however, these are split out by track like this:
~/.config/google-chrome/myprofilename
~/.config/google-chrome-beta/myprofilename
and if you run chrome beta with a profile that has also been used with chrome stable then chrome won't use it, and does a half-assed job of warning you about the problem (i.e. it prints a warning to the terminal as above, which you will only ever see if you're running it interactively). My reading of the bug was that this data wouldn't get migrated automatically, but you could migrate it yourself.

I did this for each profile and it seemed to work fine, all my settings and themes look correct:
cp -a ~/.config/google-chrome/myprofilename ~/.config/google-chrome-beta/