Make newly added files show up in git diff output

Sometimes it's handy to be able to show a normal diff for all the changes in a patch. The default behaviour is to hide the diffs of new files, i.e. in this case the output of "git diff" is missing the contents of newfile:

$ git init
Initialized empty Git repository in /gittemp/.git/
$ echo "blah" > newfile
$ git add newfile 
$ git diff
$ git status
# On branch master
#
# Initial commit
#
# Changes to be committed:
#   (use "git rm --cached ..." to unstage)
#
# new file:   newfile
#

If you'd like to see the complete content of the new file in the diff output use the "-N" option to git add:

$ echo "blahmore" > newfile2
$ git add -N newfile2
$ git diff
diff --git a/newfile2 b/newfile2
index e69de29..39b5650 100644
--- a/newfile2
+++ b/newfile2
@@ -0,0 +1 @@
+blahmore

Thunderbolt DMA attacks on OS X

The current TL;DR on Thunderbolt DMA attacks is that the VT-d IOMMU is doing its job and Ivy-Bridge (2012 and later hardware) Macs running OS X >= 10.8.2 are not vulnerable to the easy direct-write-to-memory-style attacks we saw popularized with firewire.

While Inception claims to work with Thunderbolt, it's really only a firewire attack, so you need a Thunderbolt to firewire converter and it's subject to the same limitations as normal firewire, as described at the end of this post.

There was a great 2013 Blackhat talk by Russ Sevinsky that covered lots of chip-level reverse engineering for Thunderbolt, but ultimately he didn't come up with an attack (excellent description of the reverse engineering process though). More recently snare described how to set up an attack on thunderbolt with an FPGA board connected to a mac via a Thunderbolt-PCIe device. But the IOMMU foiled his efforts on modern hardware. Snare says he's working on trying to bypass VT-d, so there may be interesting developments in the future.

For now you should probably still be more worried about snare's other work using PCIe option ROMs as a bootkit.

SMTP email conversation threads with python: making gmail associate messages into a thread

I have some python software that sends emails, and I wanted gmail to group messages that were related to the same subject in a conversation. It's not immediately obvious how this works, and there's plenty of bad advice out there, including people stating that you just need to add "RE:" to the subject line, which is just wrong.

The way conversation threads are constructed is by using the SMTP "Message-ID" as a reference to the original email in the "In-Reply-To" and "References" headers. RFC2822 has the details, explains some fairly complex multi-parent cases, and includes some good examples. My use case was very simple, just two messages I wanted associated. The first message is sent with a message-id, the second one references it (you'll need to store this ID somewhere if you want to send more messages in the same thread):

import email.utils
from email.mime.multipart import MIMEMultipart
import smtplib

myid = email.utils.make_msgid()
msg = MIMEMultipart("alternative")
msg["Subject"] = "test"
msg["From"] = "myuser@mycompany.com"
msg["To"] = "myuser@mycompany.com"
msg.add_header("Message-ID", myid)
s = smtplib.SMTP("smtp.mycompany.com")
s.sendmail("myuser@mycompany.com", ["myuser@mycompany.com"], msg.as_string())

msg = MIMEMultipart("alternative")
msg["Subject"] = "test"
msg["From"] = "myuser@mycompany.com"
msg["To"] = "myuser@mycompany.com"
msg.add_header("In-Reply-To", myid)
msg.add_header("References", myid)
s = smtplib.SMTP("smtp.mycompany.com")
s.sendmail("myuser@mycompany.com", ["myuser@mycompany.com"], msg.as_string())

Note that it is up to the host generating the message ID to guarantee it is unique. This function gives you an RFC-compliant message id and uses a datestamp to give you a unique ID. If you are sending lots of mail that may not be enough so you can pass it extra data that will get appended to the ID:

In [3]: import email.utils

In [4]: email.utils.make_msgid()
Out[4]: '<20141110055935 data-blogger-escaped-.21441.10732="" data-blogger-escaped-myhost.mycompany.com="">'

In [5]: email.utils.make_msgid('extrarandomsauce')
Out[5]: '<20141110060140 data-blogger-escaped-.21441.5878.extrarandomsauce="" data-blogger-escaped-myhost.mycompany.com="">'

I wasn't particularly careful with my first test message ID and sent a non-compliant one :) Gmail recognizes this and politely fixes it for you:

Message-ID: <545c12bc data-blogger-escaped-.240ada0a.6dba.5421smtpin_added_broken="" data-blogger-escaped-gmr-mx.google.com="">
X-Google-Original-Message-ID: testafasdfasdfasdfasdf

Splitting an array into fixed-size chunks with python

If you're looking for a way to split data into fixed size chunks with python you're likely to run across this recipe from the itertools documentation:

def grouper(iterable, n, fillvalue=None):
    "Collect data into fixed-length chunks or blocks"
    # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx
    args = [iter(iterable)] * n
    return izip_longest(fillvalue=fillvalue, *args)

which certainly works, but why it works is less than obvious. In my case I was working with a small array where the data length was guaranteed to be a multiple of 4. I ended up using this, which is less sexy but more comprehendable:

[myarray[i:i+4] for i in xrange(0, len(myarray), 4)]