Installing a windows MSI on the commandline, detecting MSI parameters (aka properties)

Installing a MSI is fairly straightforward. Here's an example of installing python without any gui prompting to a designated directory.

msiexec /i "python-2.7.9.amd64.msi" /passive TARGETDIR="C:\tools\python2"

But how do I know about the TARGETDIR option I hear you say? Amazingly the msiexec binary can't tell you what parameters (public external properties) the MSI will accept. TARGETDIR appears to be a standard installshield property, but the MSI can expose others. The only way I found that doesn't require additional tools was to enumerate the parameters by dropping the MSI onto this piece of VBScript:

Call GetArguments(ArgArray) 


If IsArray(ArgArray) then 

For Each ArrayElement In ArgArray 
Wscript.Echo GetMSIProperties(ArrayElement) 
Next 

Else 

WScript.Echo "Drag and drop MSI-File over the Script" 

End if 


' ---------------------------------------- 
Private Function GetMSIProperties(strMSIFile) 

Dim oWI : Set oWI = CreateObject("WindowsInstaller.Installer") 
Dim oDB : Set oDB = oWI.OpenDatabase(strMSIFile, 2) 
Dim oView : Set oView = oDB.OpenView("Select * From Property") 
Dim oRecord 
oView.Execute 

Do 
Set oRecord = oView.Fetch 

If oRecord Is Nothing Then Exit Do 

iColumnCount = oRecord.FieldCount 
rowData = Empty 
delim = " " 

For iColumn = 1 To iColumnCount 
If iColumn = iColumnCount Then delim = vbLf 
rowData = rowData & oRecord.StringData(iColumn) & delim 
Next 

strMessage = strMessage & rowData 
Loop 

Set oRecord = Nothing 
Set oView = Nothing 
Set oDB = Nothing 
Set oWI = Nothing 

GetMSIProperties = strMessage 

End Function 



' ---------------------------------------- 
Private Function GetArguments(SourceArray) 

Dim iCount : iCount = 0 

If wscript.arguments.count > 0 then 

ReDim ArgArray(wscript.arguments.count -1) 

For Each Argument in wscript.arguments 

ArgArray(iCount) = Argument 
iCount = iCount +1 
Next 


iCount = Null 
GetArguments = ArgArray 


End if 

End Function 

Which will pop up a GUI window with information about the MSI including the available properties. If you're building MSIs regularly you probably already have the relevant SDK installed and can use Orca to inspect the property table.

Using tox to test code under multiple python versions

Tox is a great project that makes testing against different python versions simple, standing on the shoulders of pip and virtualenv. For our project the tox specification is incredibly simple:

$ cat tox.ini 
[tox]
envlist = py27, py34

[testenv]
commands = nosetests -v
deps =
    nose
    -rrequirements.txt

This tells tox to create two virtualenv's, one for python 2.7 and one for python 3.4, install dependencies listed in requirements.txt, and then run tests with "nosetests -v". You can run the test against all of the configured environments with just

tox

Or a particular subset like this:

tox -e py34

Python requirements.txt: including other files, and installing specific git commits

Some notes on the lesser-known aspects of requirements.txt files.

Recursively including other requirements files

You can recursively include other requirements files like this:

$ cat requirements.txt 
-r client_requirements.txt
-r server_requirements.txt

$ cat client_requirements.txt 
mox==0.5.3
pexpect==3.3

$ cat server_requirements.txt 
pexpect==3.3

$ pip install -r requirements.txt

Unfortunately, this particular configuration won't actually work. You get:

Double requirement given: pexpect==3.3 (from -r server_requirements.txt (line 1)) (already in pexpect==3.3 (from -r client_requirements.txt (line 2)), name='pexpect')

pip can't actually handle complex dependency resolution, which means that if you have the same dependency in more than one file, even if the versions don't conflict, it will refuse to install anything.

Installing specific git commits

If you've ever had to work around a bug in a released version you might find yourself doing something like this:

git clone -b develop https://github.com/pyinstaller/pyinstaller.git
cd pyinstaller
git reset --hard edb5d438d8df5255a5c8f70f42f11f75aa4e08cf
python setup.py install

But pip can actually do all of that for you, this effectively does the same thing:

pip install git+https://github.com/pyinstaller/pyinstaller.git@edb5d438d8df5255a5c8f70f42f11f75aa4e08cf#egg=PyInstaller

and you can put that line into your requirements.txt:

$ cat requirements.txt
-r client_requirements.txt
-r server_requirements.txt
git+https://github.com/pyinstaller/pyinstaller.git@edb5d438d8df5255a5c8f70f42f11f75aa4e08cf#egg=PyInstaller

When this commit goes into a pip release, you'll also be able to specify install options inside requirements.txt.

Applying a diff patch on windows

Applying a diff patch can be done on windows relatively painlessly using git. The catch is that git might try to mess with the line endings. Use this to tell it to leave the line endings alone and apply the patch:

git config --global core.autocrlf false
git apply -p0 mypatch.patch

Writing windows .bat files for cmd.exe: cheatsheet

Here's a cheatsheet of some things I've discovered when writing a .bat script to build things.

For comments you can use "rem" i.e. remark but "::" is nicer IMHO:

rem This is a comment
:: This is too

To set environment variables: (Note that ALLUSERSPROFILE will only get expanded once, so if it changes this will break)

:: Permanently in HKLM which will persist for all shell sessions:
SETX /M PATH "%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"

:: or temporarily, just for this one
SET PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin

A few warning notes here. %PATH% expands to the combination of user and system paths, i.e. the contents of both of these keys:

HKEY_CURRENT_USER\Environment
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SessionManager\Environment

This means that if you're doing a setx as above and there is stuff in HKCU then you are duplicating it in the path every time. This is a problem because of the 1024 character limit on %PATH%. So you may want to clear out the HKCU path:

SETX PATH ""

Download a file with powershell:

powershell -NoProfile -ExecutionPolicy unrestricted -Command "(new-object System.Net.WebClient).DownloadFile('https://downloads.activestate.com/ActivePerl/releases/5.20.2.2001/ActivePerl-5.20.2.2001-MSWin32-x86-64int-298913.msi', 'ActivePerl-5.20.2.2001-MSWin32-x86-64int-298913.msi')"

There seems to be no actual equivalent of the bash "set -e", i.e. exit if any command returns an error, but you can do it per command like this, where the number is the error code:

choco install git -y || echo "git install failed" && exit /b 1

Recursively delete a directory, with no prompting:

rd /s /q openssl-1.0.2a

Get a visual studio environment for building:

:: 64 bit
call "%PROGRAMFILES% (x86)\Microsoft Visual Studio 12.0\VC\bin\amd64\vcvars64.bat"

::32 bit
call "%PROGRAMFILES% (x86)\Microsoft Visual Studio 12.0\VC\bin\vcvars32.bat"

Copy a whole directory recursively. Create the destination directory if necessary:

xcopy C:\Build-OpenSSL-VC-64\include C:\pkg\include\ /s /e /h /y

Calling other .bat files

You basically have two options, either create a completely new process (so changes made to environment variables, CWD etc. will not affect the parent):

cmd /c "mybat.bat"

Or use call, which is effectively like inlining all the commands from that .bat file:

call mybat.bat

Note that if you don't use call, some_other_command in this example will never get executed since you're essentially saying "replace everything from here on with the contents of mybat.bat":

mybat.bat
some_other_command

Mounting a shared folder

This will mount a virtualbox shared folder, clearing any existing drive mapped to X:

net use x: /delete
net use x: /persistent:no \\VBOXSVR\host
x:

Vagrant, Packer, and Boxcutter FTW: Create a windows build environment with one command

A quick intro to vagrant, packer, and boxcutter, which are all projects under the Hashicorp umbrella.  From reading the project websites it's not immediately obvious how all of these relate and compliment each other, they are all vm-related after all.  I'll introduce each and then give an end-to-end example.

Vagrant

Firstly vagrant: it allows you to specify your build environment in a way that is completely repeatable by anyone in the world.  No more "builds fine for me".  You can test in exactly the same environment, with the same versions of dependencies, installed by the same scripts.

So what's special about that? Why not just have a base vm and an install script that sets everything up?  For a long time we essentially did that but a bit worse: we had a server with a set of build VMs on it.  Everyone who wanted to use the VMs would add their ssh key, and copy a giant ssh_config which set up all the port forwards needed to talk to the VMs and build the project.  We never quite got around to automating the dependency installation, so whenever a dependency needed updating we'd need to ssh in and upgrade each VM manually.  The server lived in one timezone, but developers were also on the other side of the world so copying chunks of data over for building was slow and timeouts were fairly common.

With vagrant we saw the following advantages:

• Bringing up a new vm is fast, so there's no need to keep them around and potentially contaminate your new build with previous build products.  If you always provision a new vm to build then maintaining dependencies is as simple as updating the provisioning scripts.
• Port conflicts for multiple VMs are managed automatically, no need for the big ssh_config
• Builds are performed on local VMs, so no waiting on network copies to servers on the other side of the world. Shared folders provide a simple way to move data in and out of the VM.
• Testing additional operating systems and architectures is simple thanks to the atlas catalog of vagrant boxes.
• Updating dependencies across multiple architectures and operating systems is as simple as modifying the provisioning script.  While this isn't technically an advantage of vagrant, it encourages this kind of automation.
• Instead of describing how to set up a build environment in pages of documentation, you can add a Vagrantfile and some scripts to your project and reduce all that documentation to a single command like 'make'.
• We don't need to store the virtual machines ourselves, just upload a base build box to the catalog.  When people build our project vagrant will fetch the VM and check the hash of the downloaded VM matches what we specify in our Vagrantfile (or at least it will soon).

OK that sounds good, but I want to make my own base VMs and it's tedious.  Enter packer.

Packer

So vagrant helps you automate provisioning and running a build environment working from a base VM.  Packer helps you with building the base VM in the first place which opens up a lot of possibilities for testing and automation.  For the purposes of this post, packer is just going to help us avoid the tedious GUI click fest of installing a windows build VM and getting it into a usable state.

Since packer basically eats a JSON config and turns it into a virtual machine, you expect to find an example of a packer config to follow right? You're not the first person to want a Windows 7 VM.  Enter boxcutter.

Boxcutter

Boxcutter is basically a giant library of packer configs that generate almost any VM you can possibly think of.  For this post we're going to use the windows boxcutter repo.

First install vagrant, packer, and whatever virtualisation software you plan to use (e.g. virtualbox or vmware: make sure you have the latest version) then get the boxcutter repo:

$ git clone https://github.com/boxcutter/windows.git boxcutter-windows

Then get a windows VM (this took around 45min for me, but is a spectacularly less work than doing it yourself):

$ make virtualbox/eval-win7x64-enterprise

The boxcutter scripts include all the tricks for minimising disk usage (doing a defrag, zero-ing free disk space etc.), so this is about as small as it can possibly be:

$ ls -lh box/virtualbox/eval-win7x64-enterprise-nocm-1.0.4.box 
-rw-r----- 1 user group 3.2G Apr 13 14:31 box/virtualbox/eval-win7x64-enterprise-nocm-1.0.4.box

Add it into vagrant:

$ vagrant box add box/virtualbox/eval-win7x64-enterprise-nocm-1.0.4.box --name eval-win7x64

Reference it in your Vagrantfile something like this:

  config.vm.define "eval-win7x64" do |box|
    box.vm.box = "eval-win7x64"
    box.vm.guest = :windows
    box.vm.communicator = "winrm"
  end

Fire it up:

$ vagrant up eval-win7x64

Log in:

$ vagrant ssh eval-win7x64
Last login: Mon Apr 13 14:19:31 2015 from 10.0.2.2
Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\vagrant>

Troubleshooting

Note if you see errors like these:

==> virtualbox-iso: Provisioning with shell script: script/vagrant.bat
    virtualbox-iso: 'C:/Windows/Temp/script.bat' is not recognized as an internal or external command,
    virtualbox-iso:
    virtualbox-iso: operable program or batch file.
==> virtualbox-iso: Unregistering and deleting virtual machine...
==> virtualbox-iso: Deleting output directory...
Build 'virtualbox-iso' errored: Script exited with non-zero exit status: 1

or

==> virtualbox-iso: Error detaching ISO: VBoxManage error: VBoxManage: error: Assertion failed: [SUCCEEDED(rc)] at '/build/buildd/virtualbox-4.3.10-dfsg/src/VBox/Main/src-server/MachineImpl.cpp' (10875) in nsresult Machine::saveStorageControllers(settings::Storage&).
==> virtualbox-iso: VBoxManage: error: COM RC = E_ACCESSDENIED (0x80070005).
==> virtualbox-iso: VBoxManage: error: Please contact the product vendor!
==> virtualbox-iso: VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component SessionMachine, interface IMachine, callee nsISupports
==> virtualbox-iso: VBoxManage: error: Context: "SaveSettings()" at line 888 of file VBoxManageStorageController.cpp
==> virtualbox-iso: Unregistering and deleting virtual machine...
==> virtualbox-iso: Deleting output directory...
Build 'virtualbox-iso' errored: Error detaching ISO: VBoxManage error: VBoxManage: error: Assertion failed: [SUCCEEDED(rc)] at '/build/buildd/virtualbox-4.3.10-dfsg/src/VBox/Main/src-server/MachineImpl.cpp' (10875) in nsresult Machine::saveStorageControllers(settings::Storage&).
VBoxManage: error: COM RC = E_ACCESSDENIED (0x80070005).
VBoxManage: error: Please contact the product vendor!
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component SessionMachine, interface IMachine, callee nsISupports
VBoxManage: error: Context: "SaveSettings()" at line 888 of file VBoxManageStorageController.cpp

Then you need to upgrade your virtualbox. I saw these when using the virtualbox shipped with ubuntu.

Creating a new github release (creating tags in git)

To create a new github release you'll want to first create a git tag in your repo. There's a good article on tagging here.

Show tags:

$ git tag
20150409

Tag a particular commit:

git tag -a 20150408 4abcdefg051f382098493f6043482f13437adf05 -m "Pre format change"

Push your new tag:

$ git push origin 20150408

You can now see it in the github web interface and can add notes etc. and turn it into a release.