Wednesday, December 18, 2013

OpenSSL: verify a windows code signing cert on linux

HOWTO test the signing chain of a windows code signing cert. You'll probably need to convert it to pem first:
openssl pkcs12 -in signing_cert.pfx -out file.pem
openssl verify -verbose -purpose any file.pem
Chances are pretty good that you'll be missing an intermediate signing cert, and the cert issuers don't make it particularly easy to find them. Your best bet is to google the CN from the issuer. Once you have the intermediate you can tell openssl about it using the "untrusted" option, which is helpfully missing from the verify commandline documentation.
openssl x509 -in file.pem -text -noout
openssl verify -verbose -purpose any -untrusted intermediate_untrusted.pem file.pem

Saturday, December 7, 2013

Windows 8 and Ubuntu 13.10 with encrypted root on a UEFI HP Envy 15-j085nr

I recently went through the pain of creating a Windows/Linux dual boot, here are some rough notes for the benefits of others. Since the HP envy is UEFI, some of the existing wisdom on the Internets doesn't apply.

Start by installing windows normally. Since, like most laptops, it ships with an OEM install on a separate partition, this basically just means turning it on and following the prompts.

Create an ubuntu boot USB boot drive (super easy, just download the relevant ISO and use startup-disk-creator)

Reboot to USB in UEFI mode (I followed Settings|General|Advanced Startup, but apparently Shift-Restart also works). This should get you the black UEFI GRUB menu. You can tell you have booted with UEFI if the /sys/firmware/efi directory is present.

Secure Boot is now supported but it will probably cause you problems and require disabling (use F10 to get to the UEFI menu). My install worked fine with it enabled, but when I attempted to boot windows from the ubuntu boot manager the verification failed and it went into some sort of automatic self repair mode. I interrupted this without any negative consequences.

When Ubuntu starts the screen might just go black: use the hardware keys to turn the brightness up. Since UEFI uses GPT rather than MBR partitioning (archlinux has a great writeup of the details), you'll need to use gdisk rather than fdisk to inspect the disk. The syntax is basically the same.

Choose "Try without installing". I found this was handy because if the installer crashes you still have your desktop session. From this point I mostly followed these instructions to create the encrypted container manually. The installer almost works for this setup, but not quite. You can create the boot partition and the encrypted volume in the installer gui but there appears to be no way to tell it to do LVM inside the container to separate swap and root. After following the partitioning instructions my disk ended up looking like this:
$ sudo gdisk -l /dev/sda
GPT fdisk (gdisk) version 0.8.7

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/sda: 1953525168 sectors, 931.5 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 0236CBF5-85E3-4F99-B92F-F33CADBAAAAA
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 1953525134
Partitions will be aligned on 2048-sector boundaries
Total free space is 13677 sectors (6.7 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048          821247   400.0 MiB   2700  Basic data partition
   2          821248         1353727   260.0 MiB   EF00  EFI system partition
   3         1353728         1615871   128.0 MiB   0C01  Microsoft reserved part
   4         1615872       670988287   319.2 GiB   0700  Basic data partition
   5      1899788288      1953513471   25.6 GiB    0700  Basic data partition
   6       670988288       674893823   1.9 GiB     8300  /boot
   7       674893824      1899788287   584.1 GiB   8300  cryptroot
When I rebooted I had a working Ubuntu install with an encrypted root, but no way of accessing windows. I tried a few things at this point, probably all of which you should avoid (you should probably just skip to the next paragraph). I booted the Windows recovery partition (F11), followed the prompts to get a shell and ran bootrec. I also installed EasyBCD, which allegedly supports UEFI as of version 2.2, but the boot manager it installed couldn't boot Ubuntu even though it recognised it. I then installed rEFInd. I had to reorder the boot manager priority with efibootmgr to put it first, but when I did I found it could boot Ubuntu fine, but not Windows :(

I eventually settled on using the Ubuntu boot manager with the compromise of hitting F9 and using the UEFI boot menu to get to Windows (since I'm not intending to use it very often). This are the efibootmgr commands I used to make ubuntu the default (I also ended up deleting rEFInd using efibootmgr).
$ sudo efibootmgr -o 0002,3002,0001,0000,2001,2002,2003
$ sudo efibootmgr
BootCurrent: 0002
Timeout: 0 seconds
BootOrder: 0002,3002,0001,0000,2001,2002,2003
Boot0000* rEFInd Boot Manager
Boot0001* Windows Boot Manager
Boot0002* ubuntu
Boot0003* Internal Hard Disk or Solid State Disk
Boot0005* Internal Hard Disk or Solid State Disk
Boot0009* Internal Hard Disk or Solid State Disk
Boot2001* USB Drive (UEFI)
Boot2002* Internal CD/DVD ROM Drive (UEFI)
Boot3000* Internal Hard Disk or Solid State Disk
Boot3001* Internal Hard Disk or Solid State Disk
Boot3002* Internal Hard Disk or Solid State Disk
Boot3003* Internal Hard Disk or Solid State Disk
Then as I was installing a newer kernel to try and debug a wireless networking issue I saw this after installation of 3.12:
Found Windows Boot Manager on /dev/sda2@/EFI/Microsoft/Boot/bootmgfw.efi
Adding boot menu entry for EFI firmware configuration
and all of a sudden I had a windows option in the GRUB bootloader menu! Hooray! Not sure if the new kernel was required, or perhaps just a grub-update.

For those wondering how well the HP Envy 15-j085nr plays with Ubuntu 13.10 the answer is pretty well. Graphics (1366x768), sound, function keys are all good. There appears to be some sort of packet loss problem with the RTL8188EE wireless driver that I'm still investigating. Here's some details to help pre-purchase investigating:
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor DRAM Controller (rev 06)
00:02.0 VGA compatible controller: Intel Corporation 4th Gen Core Processor Integrated Graphics Controller (rev 06)
00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05)
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5)
00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
00:1c.6 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #7 (rev d5)
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation HM87 Express LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
01:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8188EE Wireless Network Adapter (rev 01)
03:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 5227 (rev 01)
09:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
Update 2013-12-26: I filed a bug for the packet loss problem. Thanks to some forums I realised that the Realtek download page for the RTL8188CE driver actually includes the RTL8188EE and a bunch of others, something that is certainly not obvious from the page itself. After installing the kernel source, the next problem was that the Realtek driver source doesn't actually compile, and is apparently only supported up to 3.9 kernels. Errors on 3.11.0-14 look like this:
make -C /lib/modules/3.11.0-14-generic/build M=/home/g/rtl_92ce_92se_92de_8723ae_88ee_linux_mac80211_0012.0207.2013 modules
make[1]: Entering directory `/usr/src/linux-headers-3.11.0-14-generic'
  CC [M]  /home/g/rtl_92ce_92se_92de_8723ae_88ee_linux_mac80211_0012.0207.2013/base.o
In file included from /home/g/rtl_92ce_92se_92de_8723ae_88ee_linux_mac80211_0012.0207.2013/base.c:39:0:
/home/g/rtl_92ce_92se_92de_8723ae_88ee_linux_mac80211_0012.0207.2013/pci.h:247:15: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘rtl_pci_probe’
 int __devinit rtl_pci_probe(struct pci_dev *pdev,
               ^
/home/g/rtl_92ce_92se_92de_8723ae_88ee_linux_mac80211_0012.0207.2013/base.c: In function ‘rtl_action_proc’:
/home/g/rtl_92ce_92se_92de_8723ae_88ee_linux_mac80211_0012.0207.2013/base.c:885:32: error: ‘struct ieee80211_conf’ has no member named ‘channel’
       rx_status.freq = hw->conf.channel->center_freq;
                                ^
/home/g/rtl_92ce_92se_92de_8723ae_88ee_linux_mac80211_0012.0207.2013/base.c:886:32: error: ‘struct ieee80211_conf’ has no member named ‘channel’
       rx_status.band = hw->conf.channel->band;
                                ^
/home/g/rtl_92ce_92se_92de_8723ae_88ee_linux_mac80211_0012.0207.2013/base.c: In function ‘rtl_send_smps_action’:
/home/g/rtl_92ce_92se_92de_8723ae_88ee_linux_mac80211_0012.0207.2013/base.c:1451:24: error: ‘struct ieee80211_conf’ has no member named ‘channel’
   info->band = hw->conf.channel->band;

Freedom Ben's modified version worked for me however, and seems to have mostly solved the wireless problems I was seeing, and although there's still more packet loss than I would like the laptop is usable now.

Update 2013-12-30: Spoke too soon. Wireless performance was still terrible (30% - 80% packet loss) despite installing the latest proprietary driver and firmware. I eventually gave up on this card and driver and bought a Intel Corporation Centrino Advanced-N 6235 for $24, which works perfectly: 0% loss. Same access point, same location in the room, etc. The conclusion I draw is that the RTL8188EE linux driver and/or firmware/hardware itself is just terrible. I was concerned that the BIOS wouldn't allow the new card since apparently it has a whitelist of wireless card replacements, and this one wasn't on the list in the maintenance manual, but no problems.

Wednesday, November 6, 2013

Disable Find My Mac by Modifiying Nvram Configuration

`When Find My Mac (FMM) is enabled, it writes information into the Mac's nvram. This enables the configuration to persist across OS-level changes, including a full system re-image. This is great if it's been stolen and you want to track it, but not so great if you bought a second hand mac, or your employer gave you a repurposed mac. The previous owner will be able to track the mac's location and even issue a remote wipe.

Apple advises you to disable FMM when handing over ownership. That's fine if you have the owner's cooperation, but it's more problematic if the owner was some guy on craig's list, or last summer's intern who is now building water wells in Kenya. So we need a way to programmatically disable the functionality.

The relevant nvram variables are shown below. fmm-computer-name is just a base64 encoded string of the computer's host name.
$ nvram -x -p
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
[snip]
<key>fmm-computer-name</key>
<data>
bXljb21wdXRlcm5hbWU=
</data>
<key>fmm-mobileme-token-FMM</key>
<data>
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
</data>
The fmm-mobileme-token-FMM data is a base64-encoded binary plist that holds the FMM configuration. The purpose seems pretty obvious, it will beacon to https://p08-fmip.icloud.com with the authToken, AppleID, and other machine metadata. The person ID and username in the plist correspond to the Apple ID account that registered with FMM. Values changed here to protect the innocent :)
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>authToken</key>
<string>AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA~</string>
<key>dataclassProperties</key>
<dict>
<key>com.apple.Dataclass.DeviceLocator</key>
<dict>
<key>apsEnv</key>
<string>Production</string>
<key>authMechanism</key>
<string>token</string>
<key>hostname</key>
<string>p08-fmip.icloud.com</string>
<key>scheme</key>
<string>https</string>
</dict>
</dict>
<key>enabledDataclasses</key>
<array>
<string>com.apple.Dataclass.DeviceLocator</string>
</array>
<key>personID</key>
<string>1111111111</string>
<key>userInfo</key>
<dict>
<key>InUseOwnerDisplayName</key>
<string>A User Name</string>
</dict>
<key>userid</key>
<integer>502</integer>
<key>username</key>
<string>myappleidusername@gmail.com</string>
</dict>
</plist>
The solution is just to kill the nvram variables:
nvram -d fmm-computer-name
nvram -d fmm-mobileme-token-FMM
And in fact, you should probably just clear everything during your re-imaging process:
nvram -c
Probably unnecessary but you could also disable the LaunchDaemon which lives here:
/System/Library/LaunchDaemons/com.apple.findmymac.plist
Or just check that the disable is active in /var/db/launchd.db/com.apple.launchd/overrides.plist:
        <key>com.apple.findmymacd</key>
<dict>
<key>Disabled</key>
<true/>
</dict>
<key>com.apple.findmymacmessenger</key>
<dict>
<key>Disabled</key>
<true/>
</dict>

Tuesday, October 29, 2013

Exceptions.plist: interesting tweaks, including Gatekeeper forced opt-in

The Exceptions.plist is quite interesting:
defaults read /System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/Exceptions.plist
Apple is using it to force opt-in a variety of apps for Gatekeeper protection by setting "LSFileQuarantineEnabled = 1". The list of apps on mavericks as of today is:
com.microsoft.Entourage
com.microsoft.explorer
com.omnigroup.OmniWeb5
com.operasoftware.opera
com.panic.Transmit
jp.hmdt.shiira
org.gudy.azureus2
org.m0k.transmission
org.mozilla.camino
org.mozilla.firefox
org.mozilla.mozilla
org.mozilla.navigator
org.mozilla.seamonkey
org.mozilla.thunderbird
org.pythonmac.unspecified.BitTorrent
org.xlife.Xtorrent
Apple also applies uses Exceptions.plist to apply categories to apps that aren't distributed through the appstore (where categories are presumably applied to all apps) e.g.
        "com.Google.GoogleEarthPlus" =         {
            LSApplicationCategoryType = "public.app-category.reference";
        };
There's also LSMaximumVersionRequiringJava, which I think probably controls whether the OS should prompt the user to install Java for particular versions of various software:
        "com.adobe.Photoshop" =         {
            Java =             {
            };
            LSApplicationCategoryType = "public.app-category.graphics-design";
            LSMaximumVersionRequiringJava = "12.19";
        };
And miscellaneous tweaks, these for power management and display:
    AppNapOverrides =     {
        "com.apple.CrashTestDummy" =         (
                        {
                SoftDisabled = 1;
            }
        );
    };
    HighResolutionOverrides =     {
        "com.apple.AppleQmaster" =         (
                        {
                HighVersion = "3.5.3";
                SoftDisabled = 1;
            }
        );
Daniel's documented VMWare version blacklisting for 10.8 appears to be gone in 10.9.

BIOS malware

Interesting recent developments on the BIOS malware front. MITRE delivered Copernicus and this blackhat talk, and Dragos Ruiu is on the tail of something very interesting if the claims can be believed and it isn't just a long troll...

Update: picked up by arstechnica.

Saturday, October 26, 2013

OS X XProtect

XProtect is Apple’s built-in AV system for OS X. I took at look at it on 10.8 (Mountain Lion). Virus definitions are stored in a simple format here:
/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/XProtect.plist
This is part of the definition for a variant of MacDefender:
           
                <key>Description</key>
                <string>OSX.MacDefender.B</string>
                <key>LaunchServices</key>
                <dict>
                        <key>LSItemContentType</key>
                        <string>com.apple.installer-package</string>
                </dict>
                <key>Matches</key>
                <array>
                        <dict>
                                <key>MatchFile</key>
                                <dict>
                                        <key>NSURLNameKey</key>
                                        <string>Info.plist</string>
                                </dict>
                                <key>MatchType</key>
                                <string>Match</string>
                                <key>Pattern</key>
                                <string>4946506B67466C6167417574686F72697A6174696F6E416374696F6E3C2F6B65793E0A093C737472696E673E4E6F417574686F72697A6174696F6E3C2F737472696E673E0A093C6B65793E4946506B67466C616744656661756C744C6F636174696F6E3C2F6B65793E0A093C737472696E673E2F4170706C69636174696F6E733C2F737472696E673E0A09</string>
                        </dict>
It contains a simple cleartext signature for a binary string in the Info.plist inside the MacDefender installer:
In [1]: a="4946506B67466C6167417574686F72697A6174696F6E416374696F6E3C2F6B65793E0A093C737472696E673E4E6F417574686F72697A6174696F6E3C2F737472696E673E0A093C6B65793E4946506B67466C616744656661756C744C6F636174696F6E3C2F6B65793E0A093C737472696E673E2F4170706C69636174696F6E733C2F737472696E673E0A09"

In [2]: a.decode('hex')
Out[2]:'IFPkgFlagAuthorizationAction\n\t<string>NoAuthorization</string>\n\t<key>IFPkgFlagDefaultLocation</key>\n\t<string>/Applications</string>\n\t'
The XProtect updater (/usr/libexec/XProtectUpdater) runs as a launch daemon (/System/Library/LaunchDaemons/com.apple.xprotectupdater.plist), set to run every 24 hours and at load time by default. XProtectUpdater works by downloading a signed version of the plist from apple.

It (I assume) verifies the signature, and checks the version number in the downloaded version:
 <key>meta</key>
 <dict>
  <key>Version</key>
  <integer>2024</integer>
  <key>PlugInBlacklist</key>
  <dict>
   <key>10</key>
   <dict>
    <key>com.macromedia.Flash Player.plugin</key>
    <dict>
     <key>MinimumPlugInBundleVersion</key>
     <string>11.3.300.271</string>
    </dict>
    <key>com.oracle.java.JavaAppletPlugin</key>
    <dict>
     <key>BlockedPlugInBundleVersions</key>
     <array>
      <string>1.7.06.24</string>
     </array>
    </dict>
   </dict>
  </dict>
 </dict>
against the one recorded in:
/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/XProtect.meta.plist
and updates the metadata and signature plists if the version on disk is older. An sqlite database seems to be used as part of the update process:
/var/root/Library/Caches/XProtectUpdater/Cache.db

Observing detection/cleaning behaviour


When I first started looking at XProtect Apple didn't have signatures for eicar, so seeing it in action was a little tricky, or required real malware. Now they do and it looks like this:
        <dict>
<key>Description</key>
<string>OSX.eicar.com.i</string>
<key>Matches</key>
<array>
<dict>
<key>Identity</key>
<data>
M5WFbOgfK3OC3ucmAveYtkLxQUA=
</data>
<key>MatchFile</key>
<dict>
<key>NSURLNameKey</key>
<string>eicar.com</string>
</dict>
<key>MatchType</key>
<string>Match</string>
</dict>
</array>
</dict>
Breaking that down:
  • Description will show up in the "will damage your computer" warning box as seen below.
  • Matches all criteria must match for alert to be shown
  • Identity a base64 encoded sha1 hash.
  • NSURLNameKey filename
  • MatchType whether to match the filetype. Values are MatchAny, or Match. In this case it relies purely on the extension part of the filename specified in NSURNNameKey (so eicar.com will match, but downloading the same file as eicar.txt won't, and downloading eicar.com and renaming it eicar.txt also will avoid the warning). This makes sense since com has no header and eicar is just a test AV file.
Here's a python snippet to show the identity string is the sha1 hash:
In [12]: identity = "M5WFbOgfK3OC3ucmAveYtkLxQUA="

In [13]: identity.decode("base64").encode("hex")
Out[13]: '3395856ce81f2b7382dee72602f798b642f14140'

In [14]: import hashlib

In [15]: hashlib.sha1(open("eicar.com").read()).hexdigest()
Out[15]: '3395856ce81f2b7382dee72602f798b642f14140'
The signature format also supports various other NSURL filesystem resource keys as seen in this sig:
                                <key>MatchFile</key>
<dict>
<key>NSURLFileSizeKey</key>
<integer>50000</integer>
<key>NSURLNameKey</key>
<string>FileAgent</string>
<key>NSURLTypeIdentifierKey</key>
<string>public.unix-executable</string>
</dict>
<key>MatchType</key>
<string>Match</string>
Lets look at a full signature, OpinionSpy is the simplest, here it is in full:
        <dict>
                <key>Description</key>
                <string>OSX.OpinionSpy</string>
                <key>LaunchServices</key>
                <dict>
                        <key>LSItemContentType</key>
                        <string>{(com.sun.java-archive|com.apple.application-bundle)}</string>
                </dict>
                <key>Matches</key>
                <array>
                        <dict>
                                <key>MatchFile</key>
                                <dict>
                                        <key>NSURLTypeIdentifierKey</key>
                                        <string>com.sun.java-archive</string>
                                </dict>
                                <key>MatchType</key>
                                <string>Match</string>
                                <key>Pattern</key>
                                <string>504B010214000A0000000800547D8B3B9B0231BC{4}502D0700250000000000{12}636F6D2F697A666F7267652F697A7061636B2F70616E656C732F706F696E7374616C6C6572</string>
                        </dict>
                </array>
        </dict>
Note that type match is specified by UTI, and we have a pattern match rather than just a hash match. Replacing {4} and {12} with 4 and 12 bytes of stuff (I chose A's), here is a one-liner that generates a file that matches the signature:
$ python -c "open('eicar.jar','w').write('504B010214000A0000000800547D8B3B9B0231BCAAAAAAAA502D0700250000000000AAAAAAAAAAAAAAAAAAAAAAAA636F6D2F697A666F7267652F697A7061636B2F70616E656C732F706F696E7374616C6C6572'.decode('hex'))"
Which gives you part of a jar file (this signature works against the jar archive itself, not the contents):
$ hexdump -C eicar.jar 
00000000  50 4b 01 02 14 00 0a 00  00 00 08 00 54 7d 8b 3b  |PK..........T}.;|
00000010  9b 02 31 bc aa aa aa aa  50 2d 07 00 25 00 00 00  |..1.....P-..%...|
00000020  00 00 aa aa aa aa aa aa  aa aa aa aa aa aa 63 6f  |..............co|
00000030  6d 2f 69 7a 66 6f 72 67  65 2f 69 7a 70 61 63 6b  |m/izforge/izpack|
00000040  2f 70 61 6e 65 6c 73 2f  70 6f 69 6e 73 74 61 6c  |/panels/poinstal|
00000050  6c 65 72                                          |ler|
00000053
If you double-click the file, you'll see something like this since the jar doesn't actually work:



XProtect doesn't detect it since the quarantine bit isn't present. Upload it somewhere and download it with a browser to get the quarantine bit applied. Double-click and you will see this warning:


If you leave 'report malware' ticked, regardless of whether you click OK or Cancel, the malware report is sent to Apple and you get a nice malware report in:
/Users/me/Library/Logs/DiagnosticReports/XProtect_2012-09-26-102829_my-MacBook-Pro.diag
which looks like:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<array>
        <dict>
                <key>LSQuarantineAgentBundleIdentifier</key>
                <string>com.google.Chrome</string>
                <key>LSQuarantineDataURL</key>
                <string>https://some.download.url</string>
                <key>LSQuarantineOriginURL</key>
                <string>https://some.referrer/url</string>
                <key>LSQuarantineTimeStamp</key>
                <date>2012-09-26T17:09:40Z</date>
                <key>UserAction</key>
                <string>trash</string>
                <key>XProtectSignatureName</key>
                <string>OSX.OpinionSpy</string>
        </dict>
</array>
</plist>
and similar information is written to system.log:
Sep 26 10:28:29 my-MacBook-Pro.local CoreServicesUIAgent[4961]: Adding LSProperties {
            LSQuarantineAgentBundleIdentifier = "com.google.Chrome";
            LSQuarantineAgentName = "Google Chrome";            
            LSQuarantineDataURL = "https://some.download.url";
            LSQuarantineEventIdentifier = "EEEEEEEE-DEAD-DEAD-DEAD-BEEFBEEFBEEF";
            LSQuarantineIsOwnedByCurrentUser = 1;
            LSQuarantineOriginURL = "https://some.referrer/url";
            LSQuarantineTimeStamp = "2012-09-26 17:09:40 +0000";
            LSQuarantineType = LSQuarantineTypeWebDownload;
        } to malware report for file://localhost/Users/me/Downloads/eicar.jar
Sep 26 10:28:29 my-MacBook-Pro.local CoreServicesUIAgent[4961]: Saved diag report for XProtect version ??? to /Users/me/Library/Logs/DiagnosticReports/XProtect_2012-09-26-102829_my-MacBook-Pro.diag
Sep 26 10:28:29 my-MacBook-Pro.local CoreServicesUIAgent[4961]: Successfully submitted CR malware report
Sep 26 10:28:29 my-MacBook-Pro com.apple.launchd.peruser.501[126] ([0x0-0x182182].com.apple.JarLauncher[4960]): Exited: Killed: 9
If you untick 'report malware', no report is written and nothing about the malware is logged, which is terrible for sysadmins who want to know what malware has been found on their machines :(

If you try running the same file (with quarantine) from the commandline, XProtect won't fire because it only applies to files launched through LaunchServices, similar to GateKeeper.
$ java -jar eicar.jar 
Invalid or corrupt jarfile eicar.jar

Modifying signatures


Modifying signatures is as simple as changing the plist:
/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/XProtect.plist
I tested by downloading eicar as above, checking it was detected, then modifying the signature in the XProtect.plist and re-launching: it wasn't detected. When I put the original signature back, it was detected once again.

I didn't test adding signatures, but since the plist seems to be unprotected from modification (i.e. the signature checked at download time isn't stored and checked when it is read from disk), and changes take effect immediately, it should be as simple as modifying the plist. After testing with XprotectUpdater, local modifications seem to persist as long as the version number is higher in the local version.

Summary


  • Signatures are in the clear. An attacker who has been signatured doesn't need to spend any time figuring out what is signatured and testing modifications, they can just release a new version that evades the signature. Not a big deal since this is usually fairly easy without access to the actual signature, and since XProtect only has a small number of infrequently updated signatures the attacker will likely have to adapt to beat 3rd-party AV long before this anyway.
  • Only covers files with the quarantine bit set that are run via LaunchServices. Malware delivered by binaries that haven't opted-in with LSFileQuarantineEnabled won't be detected, and malware that writes files outside the normal APIs (e.g. with a Java browser plugin exploit) probably also won't be detected. Execution via the commandine or POSIX APIs also won't be blocked. Apple is really only targeting the download and double click malware that has been common to-date.
  • XProtect relies on unsigned plists for storing signatures and for storing XProtectUpdater version information that controls updates.  However, while most mac users continue to run as admin malware doesn't need to target these, it can overwrite the updater itself: one variant of Flashback already did this, which is presumably why the MRT was written and distributed via software update.

Update for Mavericks (10.9)


Some things have changed in 10.9. XProtectUpdater seems to have been removed or subsumed into another part of the OS, possibly part of the regular software update. This seems like a good move considering the comment above about malware attacking it. If I find out more I'll write another post.

Friday, October 25, 2013

Python: object container's __str__ calls __repr__ for the objects it contains

This should look familiar:
In [6]: class NewShiny(object):
    def __str__(self):
        return "something readable"
   ...: 

In [7]: a = NewShiny()

In [8]: print a
something readable
But have you ever been surprised by this?
In [10]: print [a]
[<__main__.NewShiny object at 0x4ce1590>]
What's happening is that when you call __str__ on the array it is calling __repr__ on all the objects inside the array. So the solution is to define __repr__:
In [15]: class NewShiny(object):
   ....:     def __str__(self):
   ....:         return "something readable"
   ....:     def __repr__(self):
   ....:         return "use this to recreate with eval()"
   ....: 

In [16]: a = NewShiny()

In [21]: print (a)
something readable

In [22]: print [a]
[use this to recreate with eval()]

In [23]: print (a,a)
(use this to recreate with eval(), use this to recreate with eval())
This stackoverflow answer has an excellent explanation of why this is the standard behaviour, and what the difference is between __str__ and __repr__. The general idea with __repr__ is that it returns a string that can be used to re-create the object if passed to eval().

OS X file types: Uniform Type Identifiers (UTI)

As a follow on from the post about MIME types and default handlers on OS X, here's a bit more about types in general. Apple explains it created the Uniform Type Identifier System in part to allow developers to register for types using a tree hierarchy, so for example if you can handle a text file you can probably handle a lot of text-based filetypes (like XML, HTML etc.). It also abstracts away the underlying complexity of type identification e.g. (multiple extensions like .jpg, .jpeg, and MIME type of image/jpeg).

You declare new UTIs in your application Info.plist. Apple core types can be seen here:
/System/Library/Frameworks/CFNetwork.framework/Versions/A/Resources/CFNetworkCoreTypes-Info.plist
e.g. jpeg looks like this:
            UTTypeConformsTo = "public.image";
            UTTypeDescription = "JPEG image";
            UTTypeIdentifier = "public.jpeg";
            UTTypeTagSpecification =             {
                "com.apple.ostype" = JPEG;
                "public.filename-extension" =                 (
                    jpeg,
                    jpg,
                    jpe
                );
                "public.mime-type" =                 (
                    "image/jpeg",
                    "image/jpg"
                );
            };
and the same file also contains a MIME type to file extension map:
    MIMETypeToExtensionMap =     {
        "application/andrew-inset" =         (
            ez
        );
        "application/mac-compactpro" =         (
            cpt
        );
        "application/msexcel" =         (
            xls
        );
        "application/mspowerpoint" =         (
            ppt
        );
        "application/msword" =         (
            doc
        );
        "application/octet-stream" =         (
            dms,
            lha,
            lzh,
            class,
            so,
            iso,
            fla
        );

Thursday, October 17, 2013

OpenBSM auditd on OS X: these are the logs you are looking for

OpenBSM is an implementation of the now decades old Sun Basic Security Module (BSM) API and file format, written for OS X Panther (10.3) by McAfee Research under contract to Apple. The work was completed to provide OS X with security auditing, a core component required for Common Criteria evaluation, which is itself a necessary step for doing business with the US Federal Government and other governments around the world that respect the standard.

Robert Watson, one of the original authors, gives some background on the project in this interview, including the rationale for choosing BSM:
During the work for Apple, we identified Sun's BSM API and file format as the de facto industry standard for UNIX audit implementations -- it was extensively documented, the foundation of a previously evaluated system, and was extensible to new events and data types.
Apple released the code under a FreeBSD license, enabling inclusion in FreeBSD, currently maintained by Trusted BSD.

Before 10.6 the common criteria audit tools needed to be downloaded and installed separately, but have been installed by default since then. The functionality is a little-known goldmine for security purposes. Der Flounder has a good introduction to the configuration, but I'm going to cover some different aspects.

The /etc/security/audit_control file specifies what to log and retention/rotation policies:
#
# $P4: //depot/projects/trustedbsd/openbsm/etc/audit_control#8 $
#
dir:/var/audit
minfree:5
flags:zz
naflags:zz
policy:cnt,argv
filesz:2M
expire-after:60d AND 1G
superuser-set-sflags-mask:has_authenticated,has_console_access
superuser-clear-sflags-mask:has_authenticated,has_console_access
member-set-sflags-mask:
member-clear-sflags-mask:has_authenticated
  • dir: where to write the logs
  • minfree: minimum % disk free until /etc/security/audit_warn starts getting called. You can customise/replace this script. The default is simply:
    logger -p security.warning "audit warning: $@"
    
  • flags: what user-attributable audit event classes are recorded for all users. You'll want to define a custom class (zz), more on that in a moment.
  • naflags: what non-user-attributable (system daemons etc.) event classes are recorded for all users. This config records the same events as specified by zz.
  • policy: cnt allows processes to still run if they aren't being audited (e.g. if the disk is full), argv records commandline arguments to execve. The other options are mostly either crazy sauce (halting the system if unable to audit), or not implemented.
  • filesz: rotation size threshold
  • expire-after: conditions for expiration of log files (removal)
  • everything else: here be undocumented dragons.
The /etc/security/audit_class file defines some default groups of events, which you can use in flags/naflags above to decide which events to record. I found I needed to create a custom class to log the right types of events. Here's the default file with a custom class added:
#
# $P4: //depot/projects/trustedbsd/openbsm/etc/audit_class#6 $
# $FreeBSD$
#
0x00000000:no:invalid class
0x00000001:fr:file read
0x00000002:fw:file write
0x00000004:fa:file attribute access
0x00000008:fm:file attribute modify
0x00000010:fc:file create
0x00000020:fd:file delete
0x00000040:cl:file close
0x00000080:pc:process
0x00000100:nt:network
0x00000200:ip:ipc
0x00000400:na:non attributable
0x00000800:ad:administrative
0x00001000:lo:login_logout
0x00002000:aa:authentication and authorization
0x00004000:ap:application
0x00008000:zz:mycustomclass
0x20000000:io:ioctl
0x40000000:ex:exec
0x80000000:ot:miscellaneous
0xffffffff:all:all flags set
Then you need to annotate the events you want to log with your custom class in /etc/security/audit_event. An example line to log execve (process execution) is below:
23:AUE_EXECVE:execve(2):pc,ex,zz
There is a huge menu of events to choose from, here are some examples: network socket connect/bind/accept, clock changes, auditctl, login/logout, mount, reboot, passwd, crontab modification, ssh, create/modify/update user/group, execve, fork, chmod, chown and many more.

The final piece of configuration is /etc/security/audit_user, which allows for per-user customisation that is combined with the global settings. This allows you to, for example, treat root differently to other users. The format is
username:alwaysaudit:neveraudit
and the default, below, only contains an customisation for root. alwaysaudit is 'lo' which records login/logout, password changes etc. while neveraudit is 'no' which is no exclusions.
#
# $P4: //depot/projects/trustedbsd/openbsm/etc/audit_user#3 $
# $FreeBSD: head/contrib/openbsm/etc/audit_user 157137 2006-03-26 01:44:35Z rwatson $
#
root:lo:no
The auditd LaunchDaemon (/System/Library/LaunchDaemons/com.apple.auditd.plist which runs /usr/sbin/auditd) is responsible for handling the logfiles, rotation etc. There is a commandline utility /usr/sbin/audit which can be used to start/stop/reconfigure the daemon. "audit -s" is supposed to cause a re-sync of configuration, but I found a reboot was often the only really reliable way of making sure new config was applied.

You can get a live feed of the events being logged using praudit, which is very handy for debugging and checking configuration:
sudo praudit -l /dev/auditpipe
This is also the tool to use to get a textual representation of the binary auditlog files which are in the format of starttime.endtime:
praudit -l /var/audit/20130921124047.20130924235958
There is a GUI app available in the app store for visualising the binary log files, I haven't used it, but it looks reasonable for inspecting small amounts of logs. I wanted to apply existing large-scale log analysis capabilities, which would have been easy had the logs been sent to Apple SysLog (ASL) in plaintext. Sadly they aren't, so if you want to use them off the host you're going to need to either use praudit to convert them to text and ship them, or pump them into ASL yourself, or ship the binary log and convert it to text on the server-side. Running praudit over a set of logs is quite CPU intensive.

auditreduce allows you to read binary audit files and output filtered binary files, but its capabilities are very limited. You can't filter by IP for example (although the functionality was obviously planned since it is partially implemented, it will take a -o sock="" value, but the filter doesn't get applied). This produces events related to any file under /etc:
auditreduce -o file="/etc" /var/audit/20131018010044.20131023043346 | praudit -l
An event output by praudit will look like this (the one I grabbed happened to be an auditd log rotation):
# auditreduce -m AUE_EXECVE /var/audit/20131023061325.20131023062325 | praudit -l
header,159,11,execve(2),0,Tue Oct 22 23:23:25 2013, + 131 msec,exec arg,/usr/sbin/audit,-n,path,/usr/sbin/audit,path,/usr/sbin/audit,attribute,100755,root,wheel,16777220,2190270,0,subject,-1,root,wheel,root,wheel,50716,100000,0,0.0.0.0,return,success,0,trailer,159,
Since that is basically gibberish, you can use the praudit xml output to get a better idea of what each value means:
# auditreduce -m AUE_EXECVE /var/audit/20131023061325.20131023062325 | praudit -x
<record version="11" event="execve(2)" modifier="0" time="Tue Oct 22 23:23:25 2013" msec=" + 131 msec" >
<exec_args><arg>/usr/sbin/audit</arg><arg>-n</arg></exec_args>
<path>/usr/sbin/audit</path>
<path>/usr/sbin/audit</path>
<attribute mode="100755" uid="root" gid="wheel" fsid="16777220" nodeid="2190270" device="0" />
<subject audit-uid="-1" uid="root" gid="wheel" ruid="root" rgid="wheel" pid="50716" sid="100000" tid="0 0.0.0.0" />
<return errval="success" retval="0" />

Tuesday, October 8, 2013

Fix bad timezones in EXIF on JPEG photos

If you're like me you always forget to change the timezone on your camera when you fly somewhere. It's often not until partway through the trip that I remember. Unless you remember to change the timezone before you take any pictures it's probably easiest to leave the camera in your home timezone. In fact, this is better all round since then you don't need to remember to switch it back either.

If you feel compelled to change it, take a photo of your watch (or just your wrist since no-one wears a watch these days) as a marker to remind yourself at what point the times change, since you'll need to work on just the photos with the wrong zone. If you're taking photos in many different timezones that's probably a good idea too.

When you get your photos to a computer there's a quick easy way to fix the time information on your photos on linux. Use exiv2 to modify the EXIF timestamp, this shifts the timestamps back by three hours:
exiv2 -a -03 adjust *.JPG
Then use this to set all the file modification times to the same value as the EXIF:
exiv2 -T rename *.JPG

Friday, September 13, 2013

Django: building a HttpResponse object with a generator results in empty 0-byte responses

Interesting gotcha with Django http responses I hit today. Say you want to stream back content in a http response because gathering it from its source is expensive, and/or the content is huge so you can't hold it all in memory at once. Since Django's HttpResponse method accepts a generator, you might think just yielding content in the generator would work. It might, but the problem is that if something else in your code accesses the HttpResponse before it gets put on the wire (like middleware for example), then you're going to end up delivering an empty response. See example below where the second read comes up empty.
In [42]: from django import http

In [43]: def Generator():
   ...:     yield "aaa"
   ...:     yield "bbb"
   ...:     

In [44]: r=http.HttpResponse(content=Generator())
In [45]: print r
Content-Type: text/html; charset=utf-8

aaabbb

In [46]: print r
Content-Type: text/html; charset=utf-8


Whereas if you send all the content at once, the problem doesn't exist.
In [47]: r=http.HttpResponse(content="aaa")
In [48]: print r
Content-Type: text/html; charset=utf-8

aaa

In [49]: print r
Content-Type: text/html; charset=utf-8

aaa
If marshalling all the content up-front isn't practical, you can try the StreamingHttpResponse object as of Django 1.5.

Thursday, August 15, 2013

Account authentication for pushing to code.google.com: what password?

It's not immediately obvious that you shouldn't just use your gmail account password when submitting code to code.google.com.  While you can change your google code settings to do that, the default is to use the auto-generated password available at https://code.google.com/hosting/settings.  If you're using google two-factor authentication the auto-generated password is probably your only option.

Saturday, August 3, 2013

HOWTO tar, zip, and encrypt a directory tree

Use this command to tar, zip and encrypt a directory:
tar cvz --to-stdout src_dir | gpg --cipher-algo AES256 --symmetric > out.tar.gz.gpg
OR you can use gpg-zip, which is a thin shell wrapper around tar and gpg for exactly this purpose, but it turns out getting the arguments passed correctly to tar and gpg for what I want to do was more complex than the above.

Tuesday, July 16, 2013

Human readable sharing links for files in Google Drive

If you follow the instructions for sharing links to google drive files, you end up with horrible links that are mostly just long random strings, which violate Google's own advice about URLs, but are necessary to avoid namespace conflicts.



To get a more human readable link, right click on a file in drive, click Details, then use the link under 'Hosting'. It still has a random string, but the string is the same for all files within each Drive folder and the end of the URL is the same as the name of the file in Drive, which is a significant improvement.

Friday, July 12, 2013

How not to do incident response

The OIG has a great write-up of an incident response at the Economic Development Administration that cost them $2.7m and included them physically destroying $170k of computer equipment: right down to keyboards and mice because they believed malware had firmware persistence capabilities. There were huge mis-communications with the Department Of Commerce CIRT, and a series of bad assumptions that led to this scenario. Great case study.

Thursday, July 11, 2013

Python: skip a unittest with unittest.skipTest

There's a neat way to skip python tests built into unittest. You can use decorators for the test or whole class of tests that will skip a test unconditionally or based on a condition.

In my case there was significant setup to do before I knew whether or not this test should be skipped, in which case you can call unittest.TestCase.skipTest, which just raises a unittest.SkipTest exception that is handled by the test runner. It looks like this:
In [40]: runner = unittest.TextTestRunner()

In [41]: class testing(unittest.TestCase):
    def testblah(self):
        pass
    def setUp(self):
        self.skipTest('asdfasdf')
    def runTest(self):
        pass
   ....: 

In [42]: thistestclass=testing()

In [43]: runner.run(thistestclass)
s
----------------------------------------------------------------------
Ran 1 test in 0.000s  

OK (skipped=1)
Out[43]: 

Friday, June 28, 2013

Python assertRaises exception inside generator

So I have a generator and I want to make sure it throws an exception after a certain number of calls. So expected behaviour is this:
def generate():
    yield 1
    yield 2
    raise Exception()
How do you check that in a test? You can't do this:
class testExample(unittest.TestCase):
    def test_generatorExample(self):
        self.assertRaises(Exception, a)
since a generator isn't callable. Turns out that assertRaises is a context manager as of python 2.7, which is very cool:
class testExample(unittest.TestCase):
    def test_generatorExample(self):
        with self.assertRaises(Exception):
            list(a)
Or for earlier versions of python you could use a lambda:
self.assertRaises(Exception, lambda: list(a))

Thursday, June 27, 2013

Continuously loop (cycle) through a list in python

Thank you itertools:
In [7]: a=[1,2,3]

In [8]: licycle = itertools.cycle(a)

In [9]: licycle.next()
Out[9]: 1

In [10]: licycle.next()
Out[10]: 2

In [11]: licycle.next()
Out[11]: 3

In [12]: licycle.next()
Out[12]: 1

In [13]: licycle.next()
Out[13]: 2

In [14]: licycle.next()
Out[14]: 3

In [15]: licycle.next()
Out[15]: 1

In [16]: licycle.next()
Out[16]: 2

In [17]: licycle.next()
Out[17]: 3

Wednesday, June 12, 2013

Disable Java web applet in Internet Explorer

Microsoft has joined the good fight to make disabling the vulnerability-ridden Java web plugin easier. They have released a fixit that blocks the Java ActiveX DLLs from loading and disables the JNLP handler, which effectively disables Java in IE.

Sunday, June 2, 2013

SQL join syntax

Jeff Atwood posted a useful memory aid for SQL join syntax, although as many commenters point out there are a number of caveats to using set theory for this task.

Wednesday, May 29, 2013

Puppet: service checking on OS X without using the inbuilt service stanza

Normal service enforcement is easy with puppet, it looks like this for an OS X launchdaemon:
class something::myservice {

  File { owner => 'root', group => 'wheel', mode => '0644' }
  
  file { 'myservice_launchd_plist':
    ensure => file,
    path   => '/Library/LaunchDaemons/com.blah.myservice.plist',
    source => 'puppet:///modules/something/myservice/myservice_launchd.plist',
  }
   
  service { 'com.blah.myservice':
    ensure  => 'running',
    enable  => true,
    require => File['myservice_launchd_plist'],
  }
   
}
Which is great, until you don't want to manage that plist inside of puppet. In my case it was getting installed separately. If all the clients don't get upgraded properly then laying down the new plist with puppet (which points to different paths per version) will break old versions. It also means the plist needs to be updated in two places: inside puppet and inside the package for each release. So it's a hassle. I just want a simple check to restart it if it isn't running.

First attempt:
class something::myservice {

  File { owner => 'root', group => 'wheel', mode => '0644' }
  
  file { 'myservice_launchd_plist':
    path   => '/Library/LaunchDaemons/com.blah.myservice.plist',
  }
   
  service { 'com.blah.myservice':
    ensure  => 'running',
    enable  => true,
    require => File['myservice_launchd_plist'],
  }
   
}
This works fine, until the plist isn't there: i.e. the install failed for some reason, or this machine didn't get myservice installed. In that situation puppet will exit with an error code, so puppet management is effectively broken. No good. So, can we replicate what the service stanza is doing with a couple of simple exec statements? Seems easy...
class something::myservice {

  File { owner => 'root', group => 'wheel', mode => '0644' }

  exec { 'myservice':
    onlyif  => ['! /bin/launchctl list com.blah.myservice &> /dev/null',
                '/bin/test -f /Library/LaunchDaemons/com.blah.myservice.plist'],
    command => '/bin/launchctl load /Library/LaunchDaemons/com.blah.myservice.plist',
  }

}
This will run launchctl load if the service isn't running and the plist is actually there. The problem is puppet is overzealous in its command checking and will fail with this error:
Could not evaluate: Could not find command '!'
OK, what if we do onlyif and unless. Documentation is silent on what happens if you do this. It does appear to work:
class something::myservice {

  File { owner => 'root', group => 'wheel', mode => '0644' }

  exec { 'myservice':
    onlyif  => '/bin/test -f /Library/LaunchDaemons/com.blah.myservice.plist'
    command => '/bin/launchctl load /Library/LaunchDaemons/com.blah.myservice.plist',
    unless  => '/bin/launchctl list com.blah.myservice &> /dev/null',
  }

}
But both conditions always need to be evaluated, and I'm not sure this is actually going to work in the future. In my testing 'onlyif' was run before 'unless' but I wouldn't rely on that either. So lets just work around the broken commandline checking by adding a NOP with true &&:
class something::myservice {

  File { owner => 'root', group => 'wheel', mode => '0644' }

  exec { 'myservice':
    path    => ["/bin", "/usr/bin"],
    onlyif  => ['true && ! /bin/launchctl list com.blah.myservice &> /dev/null',
                '/bin/test -f /Library/LaunchDaemons/com.blah.myservice.plist'],
    command => '/bin/launchctl load /Library/LaunchDaemons/com.blah.myservice.plist',
  }

}
The path is necessary since the commandline checking wants the first part of the command to be an absolute path, or the path specified in 'path'. Since true is an inbuilt part of bash we need to specify path.

Friday, May 17, 2013

pdb and gdb conditional breakpoints

Use a debugger enough and you'll eventually be in a loop looking for a certain condition. And it's too tedious to walk through all iterations of the loop to get to the one you want. Enter conditional breakpoints. In GDB you'd use something like this (borrowed from here):
(gdb) br test.cpp:2
Breakpoint 1 at 0x1234: file test.cpp, line 2.
(gdb) cond 1 i==2147483648
(gdb) run
Or this for strings:
break test2.cpp:10 if strcmp(y,"hello") == 0
In the python debugger the syntax is a little different:
b(reak) ([file:]lineno | function) [, condition]
Like this:
(Pdb) break blah.py:1, somevalue=2

Tuesday, May 14, 2013

python set a variable conditional on another variable

Tiny python snippet. If you're doing this:
if value:
  output = value
else:
  output = 'somethingelse'
Do this instead:
output = value or 'somethingelse'

Monday, May 13, 2013

Basic python chmod for windows

A basic python version of chmod for Windows using pywin32:

import ntsecuritycon
import win32security

DACL_PRESENT = 1
DACL_DEFAULT = 0

def WinChmod(filename, acl_list, user="SYSTEM"):
  """Provide chmod-like functionality for windows.

  Doco links:
    goo.gl/n7YR1
    goo.gl/rDv81
    goo.gl/hDobb

  Args:
    filename: target filename for acl
    acl_list: list of ntsecuritycon acl strings to be applied with bitwise OR.
              e.g. ["FILE_GENERIC_READ", "FILE_GENERIC_WRITE"]
    user: username string
  Raises:
    AttributeError: if a bad permission is passed
    RuntimeError: if filename doesn't exist
  """
  if not os.path.exists(filename):
    raise RuntimeError("filename %s does not exist" % filename)

  acl_bitmask = int()
  for acl in acl_list:
    acl_bitmask |= getattr(ntsecuritycon, acl)

  dacl = win32security.ACL()
  win_user, _, _ = win32security.LookupAccountName("", user)

  dacl.AddAccessAllowedAce(win32security.ACL_REVISION,
                           acl_bitmask, win_user)

  security_descriptor = win32security.GetFileSecurity(
      filename, win32security.DACL_SECURITY_INFORMATION)

  # Tell windows to set the acl and mark it as explicitly set
  security_descriptor.SetSecurityDescriptorDacl(DACL_PRESENT, dacl,
                                                DACL_DEFAULT)
  win32security.SetFileSecurity(filename,
                                win32security.DACL_SECURITY_INFORMATION,
                                security_descriptor)

Tuesday, May 7, 2013

Python: stub out windows libraries for testing on linux

Here's a quick way to stub out windows libraries when you are testing on Linux. This code creates empty modules using imp.
  import imp
  import sys

  def testMethodWithWindowsCall():
    StubOutWindowsLibs()
    import windows_utils
  
  def StubOutWindowsLibs():
    pywintypes = imp.new_module("pywintypes")
    pywintypes.error = Exception
    sys.modules["pywintypes"] = pywintypes

Monday, May 6, 2013

Google coding style guides

I didn't actually realise how important code style guides were until I saw one consistently applied with style-checking tools: it makes a big difference to code quality. If you don't have a style guide, you could do a lot worse than the google style guides, which have been open-sourced and are regularly updated as languages evolve. The Google python guide is here, which is very similar to Guido's python style guide.

Chrome browser commandline switches

Chrome has a prodigious number of command line switches to control all sorts of behaviour. I recently ran into this one, which I think will be handy. From the review log:
Added the flag --host-resolver-rules=XXX in r39342.

Where XXX is a comma separated list of rules that control how hostnames are resolved.

For example:
    "MAP * 127.0.0.1" --> Forces all hostnames to be resolved to 127.0.0.1
    "MAP *.google.com proxy" --> Forces all google.com subdomains to be
                                 resolved to "proxy".
    "MAP test.com [::1]:77 --> Forces "test.com" to resolve to IPv6 loopback.
                               Will also force the port of the resulting
                               socket address to be 77.
    "MAP * baz, EXCLUDE www.google.com" --> Remaps everything to "baz",
                                            except for "www.google.com".

Friday, April 26, 2013

Crontab converts percent signs to newlines

A breadcrumb for the internet: it turns out percent (%) signs get converted to newlines inside crontabs. This is one of those things that I probably learnt in the past but forgot until it bit me: it's useful if you need to send multi-line strings as input to a command in cron. From the manpage:
The entire command portion of the line, up to a newline or % character, will be executed by /bin/sh or by the shell specified in the SHELL variable of the crontab file. Per‐cent-signs (%) in the command, unless escaped with backslash (\), will be changed into newline characters, and all data after the first % will be sent to the command as standard input.
If you have unescaped %'s you will likely see errors like this:
/bin/sh: -c: line 0: unexpected EOF while looking for matching ``'
/bin/sh: -c: line 1: syntax error: unexpected end of file

Monday, April 22, 2013

Pip re-install without download (and pip install matplotlib woe)

For some bizarre reason the default behaviour of pip is to download the package every time. So try:
pip install matplotlib
and see how much fun it is to download a 32MB tarball, have one dependency fail, download again, another dependency etc. You can tell pip to stop being stupid by creating a ~/.pip/pip.conf file with:
[global]
download_cache = ~/.cache/pip
Also, the matplotlib build doesn't die on missing headers at the dependency checking stage, so if you see:
src/_png.cpp:10:20: fatal error: png.h: No such file or directory
src/ft2font.h:16:22: fatal error: ft2build.h: No such file or directory
You need:
pip install numpy
sudo apt-get install libfreetype6-dev
sudo apt-get install libpng-dev

Wednesday, April 10, 2013

mythmote listens on localhost by default

Mythmote is an android app you can install to act as a mythtv remote. No more messing with lirc. To get it working you need to first enable it in the frontend through the 'General' settings menu. Once you have restarted the frontend you should see it listening on the port you specified (6546 is the default).

A bunch of people are having trouble because the backend IP is set as '127.0.0.1', and the frontend control socket doesn't allow you to specify a listen IP. So the frontend control socket just listens on the same address as the frontend: since I have a combined frontend and backend this was localhost. Which means your phone can't reach it. So you'll need to run mythtv-setup and set your IP under 'General Configuration' to the network-local IP of your frontend (probably 192.168.0.something). Don't do the horrible port forwarding or creation of an xinetd forwarder that has been suggested on this page.

You may also want to check your firewall rules since your backend/frontend is now listening on the network interface (i.e. is more accessible to the network), and you might need to open the port for the remote.

Thursday, April 4, 2013

Screen: start multiple sessions/windows automatically

Say you want to regularly start up a screen session with a bunch of different commands running in windows. You can achieve this with a .screenrc file.
# Screen startup
# run with screen -c [filename]

# Import your regular screen settings
source ~/.screenrc

# When a process dies, if it was non-zero exit then keep the window until you
# hit 'q', if you hit 'r', run the same command again.
zombie qr onerror

screen -t vim /usr/bin/vim /code
screen -t tests
screen -t bash
screen -t monitoring /bin/some/montoring
Then just run it with
screen -c myscreenrc

Wednesday, April 3, 2013

Pasting in Vim visual block mode

Say you have a section of text and want to paste something at the start of every line. How do you do that easily in vim?
text line 1
text line 2
text line 3
This blog post led me to the answer, I've added this to my vimrc:
" Use CTRL-V for pasting yank contents in insert mode.  Also useful in visual
" block mode for pasting a line of text in front of all other lines
" http://briancarper.net/blog/341/
inoremap <C-v> <C-r>" <Esc>
So I go:
CTRL-V   (visual block mode)
jj       (select first column of lines)
I        (insert mode)
CTRL-V   (paste yanked content - unnamed register)
To get:
this is new. text line 1
this is new. text line 2
this is new. text line 3
You might want to pick a different mapping if you use the existing CTRL-V functionality to insert funky characters. See the vim help (:h i_CTRL-V)

Wednesday, March 6, 2013

BCM4313 wireless DHCP timing out 'ip-config-unavailable'

I recently had a fight with a Broadcom wireless driver (BCM4313 [14e4:4727]), which was eventually solved by forcing the OS to use the brcmsmac driver. The symptoms were that the WPA-PSK authentication handshake would succeed but DHCP would always timeout, I believe because the underlying 802.11 frames themselves were getting screwed up. Typical logs looked like this:
NetworkManager[948]:  (eth1): DHCPv4 request timed out.
NetworkManager[948]:  (eth1): canceled DHCP transaction, DHCP client pid 2455
NetworkManager[948]:  Activation (eth1) Stage 4 of 5 (IPv4 Configure Timeout) scheduled...
NetworkManager[948]:  Activation (eth1) Stage 4 of 5 (IPv4 Configure Timeout) started...
NetworkManager[948]:  (eth1): device state change: ip-config -> failed (reason 'ip-config-unavailable') [70 120 5]
The state of support for Broadcom wireless chips is quite complicated. Archlinux has a good summary of the mess.

Friday, February 15, 2013

HOWTO disable the GPG GUI dialog prompt

This just annoyed me enough to figure out how to stop it. Ubuntu turns on 'use-agent' by default now:
# For Ubuntu we now use-agent by default to support more automatic
# use of GPG and S/MIME encryption by GUI programs.  Depending on the
# program, users may still have to manually decide to install gnupg-agent.
Which means you see an annoying dialog anytime you need to put in a password to encrypt/decrypt. The solution is to comment out 'use-agent' in
~/.gnupg/gpg.conf
You can also hit escape to get to the regular curses prompt.

Wednesday, February 13, 2013

Python seconds since epoch timestamp

Getting a 'seconds since epoch' timestamp in python is easy, but there are also some wrong ways to do it, a number of which are being recommended on Stack Overflow. time.time() seems to be the best choice:
# System time since epoch for reference (--utc actually makes no difference)
In [30]: !date --utc +%s
1360802215

# Correct!
In [31]: time.time()
Out[31]: 1360802218.887309

# This one is wrong, it mixes local and UTC
In [32]: time.mktime(time.gmtime())
Out[32]: 1360831021.0

# This is also correct, but requires a whole other module
In [33]: calendar.timegm(time.gmtime())
Out[33]: 1360802225

Wednesday, January 9, 2013

Standard time format ISO 8601

I'm often forgetting what this standard is. Here are some code snippets to generate ISO 8601 compliant datetime strings.

Python
In [8]: import time

In [9]: time.strftime('%Y-%m-%dT%H:%M:%S%z')
Out[9]: '2013-01-09T17:58:37-0800'
Python datetime, which has dumb handling of timezones, easiest is to just get a UTC time and hard-code in the Z for Zulu:
In [39]: datetime.datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%SZ')
Out[39]: '2013-01-10T02:09:17Z'
Ruby
$ irb
irb(main):001:0> require 'time'
=> true
irb(main):002:0> Time.now.iso8601
=> "2013-01-09T17:55:01-08:00"
Bash
$ date +%Y-%m-%dT%H:%M:%S%z
2013-01-09T17:59:41-0800
As an aside, python uses the old and incredibly confusing UNIX convention for timezones (man tzset) that is seconds West of UTC (e.g. Pacific US is +28800 currently), whereas basically everything else these days uses hours and minutes East of UTC (Pacific US is -0800 currently).

Here's what it looks like:
In [46]: time.timezone
Out[46]: 28800

In [47]: time.timezone/3600
Out[47]: 8