Thursday, April 23, 2015

Python requirements.txt: including other files, and installing specific git commits

Some notes on the lesser-known aspects of requirements.txt files.

Recursively including other requirements files

You can recursively include other requirements files like this:
$ cat requirements.txt 
-r client_requirements.txt
-r server_requirements.txt

$ cat client_requirements.txt 

$ cat server_requirements.txt 

$ pip install -r requirements.txt
Unfortunately, this particular configuration won't actually work. You get:
Double requirement given: pexpect==3.3 (from -r server_requirements.txt (line 1)) (already in pexpect==3.3 (from -r client_requirements.txt (line 2)), name='pexpect')
pip can't actually handle complex dependency resolution, which means that if you have the same dependency in more than one file, even if the versions don't conflict, it will refuse to install anything.

Installing specific git commits

If you've ever had to work around a bug in a released version you might find yourself doing something like this:
git clone -b develop
cd pyinstaller
git reset --hard edb5d438d8df5255a5c8f70f42f11f75aa4e08cf
python install
But pip can actually do all of that for you, this effectively does the same thing:
pip install git+
and you can put that line into your requirements.txt:
$ cat requirements.txt
-r client_requirements.txt
-r server_requirements.txt
When this commit goes into a pip release, you'll also be able to specify install options inside requirements.txt.

Tuesday, April 21, 2015

Applying a diff patch on windows

Applying a diff patch can be done on windows relatively painlessly using git. The catch is that git might try to mess with the line endings. Use this to tell it to leave the line endings alone and apply the patch:
git config --global core.autocrlf false
git apply -p0 mypatch.patch

Thursday, April 16, 2015

Writing windows .bat files for cmd.exe: cheatsheet

Here's a cheatsheet of some things I've discovered when writing a .bat script to build things.

For comments you can use "rem" i.e. remark but "::" is nicer IMHO:
rem This is a comment
:: This is too
To set environment variables: (Note that ALLUSERSPROFILE will only get expanded once, so if it changes this will break)
:: Permanently in HKLM which will persist for all shell sessions:

:: or temporarily, just for this one
A few warning notes here. %PATH% expands to the combination of user and system paths, i.e. the contents of both of these keys:
This means that if you're doing a setx as above and there is stuff in HKCU then you are duplicating it in the path every time. This is a problem because of the 1024 character limit on %PATH%. So you may want to clear out the HKCU path:
Download a file with powershell:
powershell -NoProfile -ExecutionPolicy unrestricted -Command "(new-object System.Net.WebClient).DownloadFile('', 'ActivePerl-')"
There seems to be no actual equivalent of the bash "set -e", i.e. exit if any command returns an error, but you can do it per command like this, where the number is the error code:
choco install git -y || echo "git install failed" && exit /b 1
Recursively delete a directory, with no prompting:
rd /s /q openssl-1.0.2a
Get a visual studio environment for building:
:: 64 bit
call "%PROGRAMFILES% (x86)\Microsoft Visual Studio 12.0\VC\bin\amd64\vcvars64.bat"

::32 bit
call "%PROGRAMFILES% (x86)\Microsoft Visual Studio 12.0\VC\bin\vcvars32.bat"
Copy a whole directory recursively. Create the destination directory if necessary:
xcopy C:\Build-OpenSSL-VC-64\include C:\pkg\include\ /s /e /h /y
Calling other .bat files

You basically have two options, either create a completely new process (so changes made to environment variables, CWD etc. will not affect the parent):
cmd /c "mybat.bat"
Or use call, which is effectively like inlining all the commands from that .bat file:
call mybat.bat
Note that if you don't use call, some_other_command in this example will never get executed since you're essentially saying "replace everything from here on with the contents of mybat.bat":
Mounting a shared folder

This will mount a virtualbox shared folder, clearing any existing drive mapped to X:
net use x: /delete
net use x: /persistent:no \\VBOXSVR\host

Monday, April 13, 2015

Vagrant, Packer, and Boxcutter FTW: Create a windows build environment with one command

A quick intro to vagrant, packer, and boxcutter, which are all projects under the Hashicorp umbrella.  From reading the project websites it's not immediately obvious how all of these relate and compliment each other, they are all vm-related after all.  I'll introduce each and then give an end-to-end example.


Firstly vagrant: it allows you to specify your build environment in a way that is completely repeatable by anyone in the world.  No more "builds fine for me".  You can test in exactly the same environment, with the same versions of dependencies, installed by the same scripts.

So what's special about that? Why not just have a base vm and an install script that sets everything up?  For a long time we essentially did that but a bit worse: we had a server with a set of build VMs on it.  Everyone who wanted to use the VMs would add their ssh key, and copy a giant ssh_config which set up all the port forwards needed to talk to the VMs and build the project.  We never quite got around to automating the dependency installation, so whenever a dependency needed updating we'd need to ssh in and upgrade each VM manually.  The server lived in one timezone, but developers were also on the other side of the world so copying chunks of data over for building was slow and timeouts were fairly common.

With vagrant we saw the following advantages:

  • Bringing up a new vm is fast, so there's no need to keep them around and potentially contaminate your new build with previous build products.  If you always provision a new vm to build then maintaining dependencies is as simple as updating the provisioning scripts.
  • Port conflicts for multiple VMs are managed automatically, no need for the big ssh_config
  • Builds are performed on local VMs, so no waiting on network copies to servers on the other side of the world. Shared folders provide a simple way to move data in and out of the VM.
  • Testing additional operating systems and architectures is simple thanks to the atlas catalog of vagrant boxes.
  • Updating dependencies across multiple architectures and operating systems is as simple as modifying the provisioning script.  While this isn't technically an advantage of vagrant, it encourages this kind of automation.
  • Instead of describing how to set up a build environment in pages of documentation, you can add a Vagrantfile and some scripts to your project and reduce all that documentation to a single command like 'make'.
  • We don't need to store the virtual machines ourselves, just upload a base build box to the catalog.  When people build our project vagrant will fetch the VM and check the hash of the downloaded VM matches what we specify in our Vagrantfile (or at least it will soon).
OK that sounds good, but I want to make my own base VMs and it's tedious.  Enter packer.


So vagrant helps you automate provisioning and running a build environment working from a base VM.  Packer helps you with building the base VM in the first place which opens up a lot of possibilities for testing and automation.  For the purposes of this post, packer is just going to help us avoid the tedious GUI click fest of installing a windows build VM and getting it into a usable state.

Since packer basically eats a JSON config and turns it into a virtual machine, you expect to find an example of a packer config to follow right? You're not the first person to want a Windows 7 VM.  Enter boxcutter.


Boxcutter is basically a giant library of packer configs that generate almost any VM you can possibly think of.  For this post we're going to use the windows boxcutter repo.

First install vagrant, packer, and whatever virtualisation software you plan to use (e.g. virtualbox or vmware: make sure you have the latest version) then get the boxcutter repo:
$ git clone boxcutter-windows
Then get a windows VM (this took around 45min for me, but is a spectacularly less work than doing it yourself):
$ make virtualbox/eval-win7x64-enterprise
The boxcutter scripts include all the tricks for minimising disk usage (doing a defrag, zero-ing free disk space etc.), so this is about as small as it can possibly be:
$ ls -lh box/virtualbox/ 
-rw-r----- 1 user group 3.2G Apr 13 14:31 box/virtualbox/
Add it into vagrant:
$ vagrant box add box/virtualbox/ --name eval-win7x64
Reference it in your Vagrantfile something like this:
  config.vm.define "eval-win7x64" do |box| = "eval-win7x64"
    box.vm.guest = :windows
    box.vm.communicator = "winrm"
Fire it up:
$ vagrant up eval-win7x64
Log in:
$ vagrant ssh eval-win7x64
Last login: Mon Apr 13 14:19:31 2015 from
Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.


Note if you see errors like these:
==> virtualbox-iso: Provisioning with shell script: script/vagrant.bat
    virtualbox-iso: 'C:/Windows/Temp/script.bat' is not recognized as an internal or external command,
    virtualbox-iso: operable program or batch file.
==> virtualbox-iso: Unregistering and deleting virtual machine...
==> virtualbox-iso: Deleting output directory...
Build 'virtualbox-iso' errored: Script exited with non-zero exit status: 1
==> virtualbox-iso: Error detaching ISO: VBoxManage error: VBoxManage: error: Assertion failed: [SUCCEEDED(rc)] at '/build/buildd/virtualbox-4.3.10-dfsg/src/VBox/Main/src-server/MachineImpl.cpp' (10875) in nsresult Machine::saveStorageControllers(settings::Storage&).
==> virtualbox-iso: VBoxManage: error: COM RC = E_ACCESSDENIED (0x80070005).
==> virtualbox-iso: VBoxManage: error: Please contact the product vendor!
==> virtualbox-iso: VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component SessionMachine, interface IMachine, callee nsISupports
==> virtualbox-iso: VBoxManage: error: Context: "SaveSettings()" at line 888 of file VBoxManageStorageController.cpp
==> virtualbox-iso: Unregistering and deleting virtual machine...
==> virtualbox-iso: Deleting output directory...
Build 'virtualbox-iso' errored: Error detaching ISO: VBoxManage error: VBoxManage: error: Assertion failed: [SUCCEEDED(rc)] at '/build/buildd/virtualbox-4.3.10-dfsg/src/VBox/Main/src-server/MachineImpl.cpp' (10875) in nsresult Machine::saveStorageControllers(settings::Storage&).
VBoxManage: error: COM RC = E_ACCESSDENIED (0x80070005).
VBoxManage: error: Please contact the product vendor!
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component SessionMachine, interface IMachine, callee nsISupports
VBoxManage: error: Context: "SaveSettings()" at line 888 of file VBoxManageStorageController.cpp
Then you need to upgrade your virtualbox. I saw these when using the virtualbox shipped with ubuntu.

Thursday, April 9, 2015

Creating a new github release (creating tags in git)

To create a new github release you'll want to first create a git tag in your repo. There's a good article on tagging here.

Show tags:
$ git tag
Tag a particular commit:
git tag -a 20150408 4abcdefg051f382098493f6043482f13437adf05 -m "Pre format change"
Push your new tag:
$ git push origin 20150408
You can now see it in the github web interface and can add notes etc. and turn it into a release.

Thursday, March 26, 2015

Windows cacls.exe cheatsheet

The windows cacls.exe is a confusing, un-intuitive, and poorly documented tool. I'm recording a few of my more frequent examples here as a cheat sheet.

Give all users full control over all files and subdirs in a directory:
cacls.exe target_dir /t /e /g Users:f

Tuesday, March 24, 2015

[SOLVED] fatal error C1902: Program database manager mismatch; please check your installation

I was running the Visual Studio compiler by ssh'ing into a cygwin SSH server running on windows and running "cmd /c msbuild" which gave me this error:
fatal error C1902: Program database manager mismatch; please check your installation
But strangely, if I connected over RDP and ran exactly the same command in a windows worked. This email led me to the solution so I'm giving it a signalboost. It turns out that if you authenticate to the SSH server with a key you run as a different user and in a login environment that somehow messes up the visual studio build process. You can see the difference by running whoami as below.

If you authenticated with a password you should see a proper username (in this case vagrant):
$ /cygdrive/c/WINDOWS/system32/whoami.exe
But if you authenticate with a key, you'll see something like this:
$ /cygdrive/c/WINDOWS/system32/whoami.exe