JAVA CA store on OS X

To get the list of CAs trusted by Java on OS X:

keytool -v -list -keystore /System/Library/Java/Support/CoreDeploy.bundle/Contents/Home/lib/security/cacerts

The default password is 'changeit'.

Ruby foo: flatten a hash into a string

Flatten a ruby hash into a string. In this case the comma is the key value delimiter and the semicolon is the entry delimiter.

irb(main):016:0> a={'sdfs'=>['aaa','bbb'], 'ppp'=>['aa','a']}
=> {"ppp"=>["aa", "a"], "sdfs"=>["aaa", "bbb"]}
=> ["ppp:aa,a", "sdfs:aaa,bbb"]
irb(main):018:0> b=a.map { |k, v| "#{k}:#{v.join(',')}" }.join(";")
=> "ppp:aa,a;sdfs:aaa,bbb"

Ruby Dir.glob to provides some of the python os.walk functionality

The ruby Dir.glob command is pretty nice. It provides some of what os.walk provides for python and is a reasonable choice for working with files and directories. Double * gets you recursion. Quick and dirty way to list user home directories and user-level global environment file:

Dir.glob("/Users/**/.MacOSX/environment")

More comprehensive is to use dscl, but it is more work, first get a list of users:

dscl . -list /Users

Then iterate over each of those:

dscl . -read /Users/myuser NFSHomeDirectory

To get a rough list of Applications and their plists (note getting a list of everything installed is more complicated):

Dir.glob("/Applications/**/**app/Contents/Info.plist")

Ruby string manipulation: struct, split, join

Every time I write ruby it seems like I have to re-learn everything. To shortcut the process for next time here is a snippet that demonstrates some string handling by parsing the output of lsof (OS X format). Ruby structs are the equivalent of python namedtuple, very handy for splitting up strings in a sane way.

def parse_lsof(lsof_out)
  net_listener = Struct.new(:command, :pid, :user, :fd, :type, :device, :size, :node, :name, :status)
  output = []

  # strip header line
  lsof_lines = lsof_out.split("\n")[1..-1]

  lsof_lines.each {
    |line|

    line_array = line.split("\s")
    listener_o = net_listener.new(*line_array)
    output.push("#{listener_o.command},#{listener_o.user},#{listener_o.type},#{listener_o.node},#{listener_o.name}")
  }
  return output.join("    ")
end

Adjust log level of python logger in a library (pyactiveresource)

My logging was being polluted by pyactiveresource that was writing a lot of useless logs at the 'INFO' level. I wanted my own logging set at INFO, so how can you change pyactiveresource? As described in the python doco, you can access the logger with getLogger and adjust the level like this:

  noisy_logger = logging.getLogger('pyactiveresource')
  noisy_logger.setLevel(logging.WARN)

Then just set your own logger as normal, something like:

  syslog = logging.handlers.SysLogHandler('/var/run/syslog')
  syslog.setFormatter(logging.Formatter('%(name)s: %(message)s'))
  syslog.setLevel(logging.INFO)
  logging.getLogger().addHandler(syslog)

Mac OS X pkg, bom files, package installation and utilities

List all installed packages:

pkgutil --pkgs --volume /

What files does this package install?

pkgutil --files com.vmware.fusion

Verify file permissions and owner/group is the same as listed in the package BOM in /private/var/db/receipts/ (actually calls repair_packages). See the end of this post for how to verify the file content.

pkgutil --verify com.vmware.fusion

There isn't any uninstall functionality on OS X, this command just drops the metadata out of the receipts directory, which is handy if you want to re-install:

pkgutil --forget com.vmware.fusion

pkgutil has a code signing check feature, but it looks like it is broken or doesn't work how I expected it to:

$ pkgutil --check-signature com.apple.pkg.Safari
Package does not exist: com.apple.pkg.Safari
$ pkgutil --check-signature /Applications/Safari.app/
Package "Safari":
   Status: no signature
$ codesign -d -vvv /Applications/Safari.app/
Executable=/Applications/Safari.app/Contents/MacOS/Safari
Identifier=com.apple.Safari
Format=bundle with Mach-O universal (i386 x86_64)
CodeDirectory v=20100 size=185 flags=0x0(none) hashes=3+3 location=embedded
Hash type=sha1 size=20
CDHash=66615ae53cb89ac254e4efc6d3eb2f93fa6a4a85
Signature size=4064
Authority=Software Signing
Authority=Apple Code Signing Certification Authority
Authority=Apple Root CA
Info.plist entries=37
Sealed Resources rules=11 files=393
Internal requirements count=1 size=108

PKG files

PKG files are just XAR files that contain a bunch of metadata files and the app itself inside a zipped cpio.

To extract the files on OS X, use (note tempdir will be created and can't exist prior to running the command):

pkgutil --expand my.pkg tempdir

Here is a manual unpacking sequence (if you just want metadata use the pkgutil commands to avoid all this unpacking). tar understands xar on OS X but on linux you'll need to install the xar package and use 'xar -xf helloworld.pkg':

$ tar zxvf helloworld.pkg 
x com.totallylegit.helloworld.pkg
x com.totallylegit.helloworld.pkg/Bom
x com.totallylegit.helloworld.pkg/Payload
x com.totallylegit.helloworld.pkg/PackageInfo
x Distribution

$ cd com.totallylegit.helloworld.pkg/
$ ls
Bom  PackageInfo Payload
$ mv Payload Payload.gz
$ gunzip Payload
$ cpio -iv < Payload
.
./helloworld.app
./helloworld.app/Contents
./helloworld.app/Contents/Info.plist
./helloworld.app/Contents/MacOS
./helloworld.app/Contents/MacOS/helloworld
./helloworld.app/Contents/PkgInfo
./helloworld.app/Contents/Resources
./helloworld.app/Contents/Resources/en.lproj
./helloworld.app/Contents/Resources/en.lproj/Credits.rtf
./helloworld.app/Contents/Resources/en.lproj/InfoPlist.strings
./helloworld.app/Contents/Resources/en.lproj/MainMenu.nib
87 blocks

BOM files

Use lsbom to inspect bill of materials files (note apple says "The files and directories where receipts are stored are subject to change. Always use pkgutil to query or modify them"). These options are more readable than the defaults:

$ lsbom -p MUGsf /private/var/db/receipts/com.vmware.fusion.bom
drwxr-xr-x  root admin  .
drwxrwxr-x  root admin  ./Applications
-rwxr-xr-x  root admin 82 ./Applications/._VMware Fusion.app
[snip]

While pkgutil/repair_packages could check file content hasn't changed by calculating the CRC32 and comparing that with the one in the BOM, it doesn't. It would be fairly simple to do. The BOM content for a simple package looks like this (path, octal mode, UID/GID, size, CRC32):

$ lsbom Bom 
. 0 0/0
./helloworld.app 40755 0/0
./helloworld.app/Contents 40755 0/0
./helloworld.app/Contents/Info.plist 100644 0/0 1429 3178638052
./helloworld.app/Contents/MacOS 40755 0/0
./helloworld.app/Contents/MacOS/helloworld 100755 0/0 13488 1649029420
./helloworld.app/Contents/PkgInfo 100644 0/0 8 742937289
./helloworld.app/Contents/Resources 40755 0/0
./helloworld.app/Contents/Resources/en.lproj 40755 0/0
./helloworld.app/Contents/Resources/en.lproj/Credits.rtf 100644 0/0 436 2072274354
./helloworld.app/Contents/Resources/en.lproj/InfoPlist.strings 100644 0/0 92 2769355950
./helloworld.app/Contents/Resources/en.lproj/MainMenu.nib 100644 0/0 27463 4224082203

and the checksum can be calculated with cksum which prints the size in bytes and the CRC32:

$ cksum helloworld 
1649029420 13488 helloworld

Installation

On installation the package app directory is copied to the installation dir (usually /Applications/) and the BOM is written with a plist of installation metadata to the Receipts directory:

$ ls -1 /private/var/db/receipts/com.vmware*
/private/var/db/receipts/com.vmware.fusion.bom
/private/var/db/receipts/com.vmware.fusion.plist

Gatekeeper, XProtect and the Quarantine attribute

Apps can opt-in to Gatekeeper and Xprotect protection by adding LSFileQuarantineEnabled to their Contents/Info.plist. This means that any files created by that app will get tagged with the apple quarantine HFS+ extended attribute. For example, here's what chrome looks like when downloaded with Safari:

$ ls -l@eOd Downloads/googlechrome.dmg 
-rw-r--r--@ 1 user  group  - 39937778 Apr 16 11:31 Downloads/googlechrome.dmg
 com.apple.diskimages.fsck       20 
 com.apple.diskimages.recentcksum       79 
 com.apple.metadata:kMDItemDownloadedDate       53 
 com.apple.metadata:kMDItemWhereFroms      210 
 com.apple.quarantine       57 

and the same file downloaded with curl (which isn't a cocoa app and hence can't opt-in):

$ curl -o chrome_curl.dmg -L https://dl.google.com/chrome/mac/stable/GGRO/googlechrome.dmg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 38.0M  100 38.0M    0     0  15.7M      0  0:00:02  0:00:02 --:--:-- 18.2M
$ ls -l@eOd chrome_curl.dmg 
-rw-r--r--  1 user  group  - 39937627 Apr 20 14:47 chrome_curl.dmg

Looking at the contents of the extended attributes we see a bunch of interesting metadata, including the HTTP referrer in a binary plist inside 'com.apple.metadata:kMDItemWhereFroms' and the quaratine attribute in 'com.apple.quarantine'.

$ xattr -l Downloads/googlechrome.dmg 
com.apple.diskimages.fsck: |?W??\<!*
                                    ?#Z???r??u
com.apple.diskimages.recentcksum: i:325165 on 3A5356B1-31CB-32FB-AFC7-DBE23A5C1547 @ 1334601060 - CRC32:$AE7BD4AF
com.apple.metadata:kMDItemDownloadedDate:
00000000  62 70 6C 69 73 74 30 30 A1 01 33 41 B5 42 0E 78  |bplist00..3A.B.x|
00000010  17 F3 17 08 0A 00 00 00 00 00 00 01 01 00 00 00  |................|
00000020  00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 13                                   |.....|
00000035
com.apple.metadata:kMDItemWhereFroms:
00000000  62 70 6C 69 73 74 30 30 A2 01 02 5F 10 3D 68 74  |bplist00..._.=ht|
00000010  74 70 73 3A 2F 2F 64 6C 2E 67 6F 6F 67 6C 65 2E  |tps://dl.google.|
00000020  63 6F 6D 2F 63 68 72 6F 6D 65 2F 6D 61 63 2F 73  |com/chrome/mac/s|
00000030  74 61 62 6C 65 2F 47 47 52 4D 2F 67 6F 6F 67 6C  |table/GGRM/googl|
00000040  65 63 68 72 6F 6D 65 2E 64 6D 67 5F 10 61 68 74  |echrome.dmg_.aht|
00000050  74 70 73 3A 2F 2F 77 77 77 2E 67 6F 6F 67 6C 65  |tps://www.google|
00000060  2E 63 6F 6D 2F 63 68 72 6F 6D 65 3F 26 62 72 61  |.com/chrome?&bra|
00000070  6E 64 3D 43 48 4D 41 26 75 74 6D 5F 63 61 6D 70  |nd=CHMA&utm_camp|
00000080  61 69 67 6E 3D 65 6E 26 75 74 6D 5F 73 6F 75 72  |aign=en&utm_sour|
00000090  63 65 3D 65 6E 2D 68 61 2D 6E 61 2D 75 73 2D 62  |ce=en-ha-na-us-b|
000000A0  6B 26 75 74 6D 5F 6D 65 64 69 75 6D 3D 68 61 08  |k&utm_medium=ha.|
000000B0  0B 4B 00 00 00 00 00 00 01 01 00 00 00 00 00 00  |.K..............|
000000C0  00 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
000000D0  00 AF                                            |..|
000000d2
com.apple.quarantine: 0002;4f91d6f8;Safari;A89FCF40-0748-46BE-9C5E-1599A280E9D6

Breaking down this string based on the description in:

/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/Headers/LSQuarantine.h

• 0002: kLSQuarantineType (I think), one of:
  • kLSQuarantineTypeWebDownload
  • kLSQuarantineTypeOtherDownload
  • kLSQuarantineTypeEmailAttachment
  • kLSQuarantineTypeInstantMessageAttachment
  • kLSQuarantineTypeCalendarEventAttachment
  • kLSQuarantineTypeOtherAttachment
• 4f91d6f8: LSQuarantineTimeStamp, seconds since epoch timestamp in hex, print with 'date -r 0x4f91d6f8'
• Safari: kLSQuarantineAgentName, the name of the process that wrote the file, or whatever the process sets via the API.
• A89FCF40-0748-46BE-9C5E-1599A280E9D6: a UUID. The key (LSQuarantineEventIdentifier) used for entries in the SQLite DB.

Wait, what SQLite DB? The one here:

$ sqlite3 /Users/auser/Library/Preferences/com.apple.LaunchServices.QuarantineEventsV2
sqlite> .dump
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE LSQuarantineEvent (  LSQuarantineEventIdentifier TEXT PRIMARY KEY NOT NULL,  LSQuarantineTimeStamp REAL,  LSQuarantineAgentBundleIdentifier TEXT,  LSQuarantineAgentName TEXT,  LSQuarantineDataURLString TEXT,  LSQuarantineSenderName TEXT,  LSQuarantineSenderAddress TEXT,  LSQuarantineTypeNumber INTEGER,  LSQuarantineOriginTitle TEXT,  LSQuarantineOriginURLString TEXT,  LSQuarantineOriginAlias BLOB );
INSERT INTO "LSQuarantineEvent" VALUES('A89FCF40-0748-46BE-9C5E-1599A280E9D6',356650616.054473,'com.apple.Safari','Safari','https://dl.google.com/chrome/mac/stable/GGRO/googlechrome.dmg',NULL,NULL,0,NULL,'https://www.google.com/',NULL);
CREATE INDEX LSQuarantineEventIndex  ON LSQuarantineEvent (  LSQuarantineEventIdentifier );
COMMIT;

Note the NULL fields, which are optional fields that can be filled in by the developer who writes the file. The OS writes LSQuarantineAgentNameKey, LSQuarantineAgentBundleIdentifierKey, and LSQuarantineTimeStampKey by default. Chrome adds LSQuarantineDataURLString and LSQuarantineOriginURLString. Apple said in the Leopard Launch Services release notes:

When the Launch Services API is used to open a quarantined file and the file appears to be an application, script, or other executable file type, Launch Services will display an alert to confirm the user understands the file is some kind of application. If the file is opened, the quarantine properties are automatically cleared by Launch Services if the user has write access to the file.

So translating this to the Gatekeeper world, with the default settings Gatekeeper will block execution and fire an alert if:

1. The file is opened with Launch Services (e.g. from the finder via user action, from another cocoa app, or as a mime handler). Excludes executables/installers run from the commandline, or programmatically via POSIX APIs. 'UnsafeExecutable' may play a role here (needs checking).
2. The file has the quarantine xattr set.
3. The file is not signed by apple or an apple developer cert.

Xprotect follows similar logic, and checks files against the attributes in

/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/XProtect.plist

to determine if the file is known-bad.