Mach-O filetype identification

I wanted to write a quick and dirty file-type identifier for Mach-O, turns out this is more tricky than I expected. From /usr/share/file/magic/mach:

# $File: mach,v 1.9 2009/09/19 16:28:10 christos Exp $
# Mach has two magic numbers, 0xcafebabe and 0xfeedface.
# Unfortunately the first, cafebabe, is shared with
# Java ByteCode, so they are both handled in the file "cafebabe".
# The "feedface" ones are handled herein.

and from /usr/share/file/magic/cafebabe:

# Since Java bytecode and Mach-O universal binaries have the same magic number, the test
# must be performed in the same "magic" sequence to get both right.  The long
# at offset 4 in a mach-O universal binary tells the number of architectures; the short at
# offset 4 in a Java bytecode file is the JVM minor version and the
# short at offset 6 is the JVM major version.  Since there are only 
# only 18 labeled Mach-O architectures at current, and the first released 
# Java class format was version 43.0, we can safely choose any number
# between 18 and 39 to test the number of architectures against
# (and use as a hack). Let's not use 18, because the Mach-O people
# might add another one or two as time goes by...

GAAAH! Unsurprisingly more than one engineer wanted to use the cutesy "cafebabe" for their magic string. I ended up using this regex, which will also match Java bytecode, but was good enough for my purpose:

^(cffaedfe|cefaedfe|feedface|feedfacf|cafebabe)

The full Mach-O filetype doco is here. The various magic byte strings are as follows:

• cefaedfe: Mach-O Little Endian (32-bit)
• cffaedfe: Mach-O Little Endian (64-bit)
• feedface: Mach-O Big Endian (32-bit)
• feedfacf: Mach-O Big Endian (64-bit)
• cafebabe: Universal Binary Big Endian. These fat binaries are archives that can include binaries for multiple architectures, but typically contain PowerPC and Intel x86.

Bash man page colours

There are many pages out there describing how to get coloured bash man pages. Tuxarena has one of the better ones that tries to explain what's going on, but unfortunately it is somewhat of a black art due to the obscure colour codes used. Here's the snippet from my bash_aliases that I use:

man() {
    env LESS_TERMCAP_mb=$'\E[01;31m' \
    LESS_TERMCAP_md=$'\E[01;38;5;74m' \
    LESS_TERMCAP_me=$'\E[0m' \
    LESS_TERMCAP_se=$'\E[0m' \
    LESS_TERMCAP_so=$'\E[01;41;33m' \
    LESS_TERMCAP_ue=$'\E[0m' \
    LESS_TERMCAP_us=$'\E[04;38;5;146m' \
    man "$@"
}

I only found one site that actually documented the color options available, and he basically had to reverse engineer it. I'll include the color codes below, since everyone likely has their own personal preference and wants to tweak things slightly.

0   = default colour
1   = bold
4   = underlined
5   = flashing text
7   = reverse field
31  = red
32  = green
33  = orange
34  = blue
35  = purple
36  = cyan
37  = grey
40  = black background
41  = red background
42  = green background
43  = orange background
44  = blue background
45  = purple background
46  = cyan background
47  = grey background
90  = dark grey
91  = light red
92  = light green
93  = yellow
94  = light blue
95  = light purple
96  = turquoise
100 = dark grey background
101 = light red background
102 = light green background
103 = yellow background
104 = light blue background
105 = light purple background
106 = turquoise background

Python Mix-ins

Python mix-ins are a handy way to augment functionality of a class.  LinuxJournal has a good detailed article about them.  To mix in a class dynamically you just need to modify the __bases__ class attribute:

class Base(object):pass
class BaseClass(Base):pass
class MixInClass(object):pass

BaseClass.__bases__ += (MixInClass,)

Note that you seem to only be able to do thsi if the base class doesn't inherit directly from object, hence the extra "Base" class above. This is what the failure looks like:

In [10]: class BaseClass(object):pass

In [11]: class MixInClass(object):pass

In [12]: BaseClass.__bases__ += (MixInClass,)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
 in ()
----> 1 BaseClass.__bases__ += (MixInClass,)

TypeError: Cannot create a consistent method resolution
order (MRO) for bases MixInClass, object

Also note that unless both classes call super in their __init__ you will have problems with initialization. It is also possible to mix in a class to an object instance, instead of a class, as mentioned on Stack Overflow like this:

obj = BaseClass()
obj.__class__ = type('MixInClass',(BaseClass,MixInClass),{})

Don't use == for comparing secrets

TIL: You shouldn't use == to compare HMACs, or anything sensitive really. Doing so creates a timing side channel that can reveal the secret to an attacker. Instead you need to use a comparison function that takes a constant amount of time for all values, not matter how similar they are to the actual HMAC. The python example given in the article is:

def is_equal(a, b):
  if len(a) != len(b):
      return False

  result = 0
  for x, y in zip(a, b):
      result |= x ^ y
  return result == 0

This function is available in python 3.3+ as:

hmac.compare_digest(a, b)

Antennas for free-to-air broadcast TV in the US (i.e. how to watch the olympics for free)

Watching the winter and summer olympics is literally the only reason I have to watch TV these days, and since the content is so locked up in media deals, buying a cable TV package seems to be the only option to see it in the US.  Not so.

NBC (which carries the olympics) is broadcast free to air in HD, along with a few other major channels and a pile of crap that isn't fit for human consumption (like 24hr home shopping).  Getting at this free content is basically a case of picking the right antenna.

1. Check available channels

Put your zipcode into the station direction finder at antennaweb.org to find out what channels are available in your area and where the TV transmitters are.

2. Buy an antenna

For best results you should probably use a high-gain outdoors antenna.  But if you are renting, or want to do it on the cheap, you might want to check out Lifehacker's list of the best indoor antennas. In the SF bay area I can tune 82 channels, including NBC aka KNTV, with the un-amplified Mohu leaf which cost me $40.  The leaf also happens to be the top recommendation from Lifehacker.

I have it mounted indoors about 9ft up the wall and I'm receiving all the channels listed by antennaweb.  I even get KTLN, which antennaweb says is being broadcast from 57 miles away!  The antennaweb recommended antenna to receive at that kind of distance is a large directional antenna with pre-amp, but the Mohu leaf is killing it.  Amazing.

3. Tune channels

Follow your TV's instructions for tuning channels.  You may have to move your antenna around to find the best reception.

OpenSSL: verify a windows code signing cert on linux

HOWTO test the signing chain of a windows code signing cert. You'll probably need to convert it to pem first:

openssl pkcs12 -in signing_cert.pfx -out file.pem
openssl verify -verbose -purpose any file.pem

Chances are pretty good that you'll be missing an intermediate signing cert, and the cert issuers don't make it particularly easy to find them. Your best bet is to google the CN from the issuer. Once you have the intermediate you can tell openssl about it using the "untrusted" option, which is helpfully missing from the verify commandline documentation.

openssl x509 -in file.pem -text -noout
openssl verify -verbose -purpose any -untrusted intermediate_untrusted.pem file.pem

Windows 8 and Ubuntu 13.10 with encrypted root on a UEFI HP Envy 15-j085nr

I recently went through the pain of creating a Windows/Linux dual boot, here are some rough notes for the benefits of others. Since the HP envy is UEFI, some of the existing wisdom on the Internets doesn't apply.

Start by installing windows normally. Since, like most laptops, it ships with an OEM install on a separate partition, this basically just means turning it on and following the prompts.

Create an ubuntu boot USB boot drive (super easy, just download the relevant ISO and use startup-disk-creator)

Reboot to USB in UEFI mode (I followed Settings|General|Advanced Startup, but apparently Shift-Restart also works). This should get you the black UEFI GRUB menu. You can tell you have booted with UEFI if the /sys/firmware/efi directory is present.

Secure Boot is now supported but it will probably cause you problems and require disabling (use F10 to get to the UEFI menu). My install worked fine with it enabled, but when I attempted to boot windows from the ubuntu boot manager the verification failed and it went into some sort of automatic self repair mode. I interrupted this without any negative consequences.

When Ubuntu starts the screen might just go black: use the hardware keys to turn the brightness up. Since UEFI uses GPT rather than MBR partitioning (archlinux has a great writeup of the details), you'll need to use gdisk rather than fdisk to inspect the disk. The syntax is basically the same.

Choose "Try without installing". I found this was handy because if the installer crashes you still have your desktop session. From this point I mostly followed these instructions to create the encrypted container manually. The installer almost works for this setup, but not quite. You can create the boot partition and the encrypted volume in the installer gui but there appears to be no way to tell it to do LVM inside the container to separate swap and root. After following the partitioning instructions my disk ended up looking like this:

$ sudo gdisk -l /dev/sda
GPT fdisk (gdisk) version 0.8.7

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/sda: 1953525168 sectors, 931.5 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 0236CBF5-85E3-4F99-B92F-F33CADBAAAAA
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 1953525134
Partitions will be aligned on 2048-sector boundaries
Total free space is 13677 sectors (6.7 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048          821247   400.0 MiB   2700  Basic data partition
   2          821248         1353727   260.0 MiB   EF00  EFI system partition
   3         1353728         1615871   128.0 MiB   0C01  Microsoft reserved part
   4         1615872       670988287   319.2 GiB   0700  Basic data partition
   5      1899788288      1953513471   25.6 GiB    0700  Basic data partition
   6       670988288       674893823   1.9 GiB     8300  /boot
   7       674893824      1899788287   584.1 GiB   8300  cryptroot

When I rebooted I had a working Ubuntu install with an encrypted root, but no way of accessing windows. I tried a few things at this point, probably all of which you should avoid (you should probably just skip to the next paragraph). I booted the Windows recovery partition (F11), followed the prompts to get a shell and ran bootrec. I also installed EasyBCD, which allegedly supports UEFI as of version 2.2, but the boot manager it installed couldn't boot Ubuntu even though it recognised it. I then installed rEFInd. I had to reorder the boot manager priority with efibootmgr to put it first, but when I did I found it could boot Ubuntu fine, but not Windows :(

I eventually settled on using the Ubuntu boot manager with the compromise of hitting F9 and using the UEFI boot menu to get to Windows (since I'm not intending to use it very often). This are the efibootmgr commands I used to make ubuntu the default (I also ended up deleting rEFInd using efibootmgr).

$ sudo efibootmgr -o 0002,3002,0001,0000,2001,2002,2003
$ sudo efibootmgr
BootCurrent: 0002
Timeout: 0 seconds
BootOrder: 0002,3002,0001,0000,2001,2002,2003
Boot0000* rEFInd Boot Manager
Boot0001* Windows Boot Manager
Boot0002* ubuntu
Boot0003* Internal Hard Disk or Solid State Disk
Boot0005* Internal Hard Disk or Solid State Disk
Boot0009* Internal Hard Disk or Solid State Disk
Boot2001* USB Drive (UEFI)
Boot2002* Internal CD/DVD ROM Drive (UEFI)
Boot3000* Internal Hard Disk or Solid State Disk
Boot3001* Internal Hard Disk or Solid State Disk
Boot3002* Internal Hard Disk or Solid State Disk
Boot3003* Internal Hard Disk or Solid State Disk

Then as I was installing a newer kernel to try and debug a wireless networking issue I saw this after installation of 3.12:

Found Windows Boot Manager on /dev/sda2@/EFI/Microsoft/Boot/bootmgfw.efi
Adding boot menu entry for EFI firmware configuration

and all of a sudden I had a windows option in the GRUB bootloader menu! Hooray! Not sure if the new kernel was required, or perhaps just a grub-update.

For those wondering how well the HP Envy 15-j085nr plays with Ubuntu 13.10 the answer is pretty well. Graphics (1366x768), sound, function keys are all good. There appears to be some sort of packet loss problem with the RTL8188EE wireless driver that I'm still investigating. Here's some details to help pre-purchase investigating:

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor DRAM Controller (rev 06)
00:02.0 VGA compatible controller: Intel Corporation 4th Gen Core Processor Integrated Graphics Controller (rev 06)
00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05)
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5)
00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
00:1c.6 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #7 (rev d5)
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation HM87 Express LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
01:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8188EE Wireless Network Adapter (rev 01)
03:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 5227 (rev 01)
09:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)

Update 2013-12-26: I filed a bug for the packet loss problem. Thanks to some forums I realised that the Realtek download page for the RTL8188CE driver actually includes the RTL8188EE and a bunch of others, something that is certainly not obvious from the page itself. After installing the kernel source, the next problem was that the Realtek driver source doesn't actually compile, and is apparently only supported up to 3.9 kernels. Errors on 3.11.0-14 look like this:

make -C /lib/modules/3.11.0-14-generic/build M=/home/g/rtl_92ce_92se_92de_8723ae_88ee_linux_mac80211_0012.0207.2013 modules
make[1]: Entering directory `/usr/src/linux-headers-3.11.0-14-generic'
  CC [M]  /home/g/rtl_92ce_92se_92de_8723ae_88ee_linux_mac80211_0012.0207.2013/base.o
In file included from /home/g/rtl_92ce_92se_92de_8723ae_88ee_linux_mac80211_0012.0207.2013/base.c:39:0:
/home/g/rtl_92ce_92se_92de_8723ae_88ee_linux_mac80211_0012.0207.2013/pci.h:247:15: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘rtl_pci_probe’
 int __devinit rtl_pci_probe(struct pci_dev *pdev,
               ^
/home/g/rtl_92ce_92se_92de_8723ae_88ee_linux_mac80211_0012.0207.2013/base.c: In function ‘rtl_action_proc’:
/home/g/rtl_92ce_92se_92de_8723ae_88ee_linux_mac80211_0012.0207.2013/base.c:885:32: error: ‘struct ieee80211_conf’ has no member named ‘channel’
       rx_status.freq = hw->conf.channel->center_freq;
                                ^
/home/g/rtl_92ce_92se_92de_8723ae_88ee_linux_mac80211_0012.0207.2013/base.c:886:32: error: ‘struct ieee80211_conf’ has no member named ‘channel’
       rx_status.band = hw->conf.channel->band;
                                ^
/home/g/rtl_92ce_92se_92de_8723ae_88ee_linux_mac80211_0012.0207.2013/base.c: In function ‘rtl_send_smps_action’:
/home/g/rtl_92ce_92se_92de_8723ae_88ee_linux_mac80211_0012.0207.2013/base.c:1451:24: error: ‘struct ieee80211_conf’ has no member named ‘channel’
   info->band = hw->conf.channel->band;

Freedom Ben's modified version worked for me however, and seems to have mostly solved the wireless problems I was seeing, and although there's still more packet loss than I would like the laptop is usable now.

Update 2013-12-30: Spoke too soon. Wireless performance was still terrible (30% - 80% packet loss) despite installing the latest proprietary driver and firmware. I eventually gave up on this card and driver and bought a Intel Corporation Centrino Advanced-N 6235 for $24, which works perfectly: 0% loss. Same access point, same location in the room, etc. The conclusion I draw is that the RTL8188EE linux driver and/or firmware/hardware itself is just terrible. I was concerned that the BIOS wouldn't allow the new card since apparently it has a whitelist of wireless card replacements, and this one wasn't on the list in the maintenance manual, but no problems.