Smörgåsbord

There are times you need to connect to ‘dirty’ networks such as public WiFi hotspots. Hopefully you’re ensuring that sensitive information is encapsulated in transport layer security enabled protocols such as SSL, because anyone on the same link (in the case of WiFi, that’s the air surrounding you. A vacuum will do, too, but that’s less common) can listen in on the traffic you’re sending. With SSL encapsulation such as HTTP over SSL (https://), your traffic can still be read — but for those who do it’s an extremely boring read because they don’t know the session key, only you and the other endpoint do. Hopefully.

One particularly nasty thing that can happen to you is when your machine is subverted into using the attacker’s machine as the router. That is known as ARP poison routing. The attacker can proceed to not only read the traffic coming from your machine (which, on a shared medium, could be done anyway), or read the traffic going into your machine (again: on a shared medium, that could be done anyway), but the attacker can now also modify the traffic between you and the rest of the non-local network, e.g., the internet, in both directions. And that’s when he can really go to town with your traffic. Injecting a javascript keylogger into all the webpages you visit. ‘Sidejacking‘ your sessions, so he does not even need to know your passwords, just your session cookies — which you happen to transmit with every page request.

All possible unless you use transport layer security, which is tamper-proof once properly set up. Once properly set up. But setting up can have problems of itself — there are ways of preventing you ever going from HTTP to HTTPS. If you know a thing or two about HTTP and SSL you’ll be delighted to learn about Moxie’s very evil but very clever ways of doing so.

Anyway, some level of security can be achieved if you tell your machine to ignore any messages sent to you from the other machines on the local network. That includes messages that will make your machine believe that the router has suddenly changed its physical address — which is quite unlikely to happen, but those messages are exactly the type of message an impersonator would send you. Of course we’d need to whitelist the routers of the network, otherwise we can’t get traffic out of it and onto other networks. DNS resolvers will need whitelisting too, unless you’re running one on your own machine (probably not).
Not openly announcing your presence may also be something you wish for. If you have ever been on a network with a Mac user you have probably seen them popping up in your Zeroconf service browser as “Firstname Lastname’s iSomething”. Let’s cut down on that kind of promiscuity, too. But you should understand now that you can not actually hide unless you turn off your WiFi. Shared medium, remember?

I prepared a simple script to accomplish the above. I’ve used ip from the iproute2 package instead of sticking to old-school route, ifconfig, arp & co. And I must say ip neigh flush nud stale has a poetic ring to it, wouldn’t you agree?

Take note: this will only protect you from some kind of attacks, and only partially. An attacker has a window of opportunity between your machine getting assigned a DHCP lease and you running this script, for instance. Or maybe the access point is rigged. Actually all protection other than end-to-end encryption combined with mutual authentication is pretty useless on shared networks ;-)

Here’s the script. Linux-only. If you want to use it, get the latest version from my public repository.

#!/bin/bash
 
# arpshield 0.2
# Protects against ARP poisoning and cloaks your machine for all 
# local link devices but the router(s) and the DNS server(s).
# Whitelisting DHCP servers also works if you use the dhcpcd program
# to obtain DHCP leases.
# This program is of no help if your setup is already poisoned.
# Have a look at ArpON (http://arpon.sourceforge.net/manpage.html) if
# you need more extensive protection.
#
# Needs 'ip', 'awk', 'sed', 'arptables', and 'arping' and expects
# them on $PATH. Needs appropriate privileges (so use sudo).
# Takes a network interface as an argument. The network interface
# should be up and configured. If no argument is given, clear all
# rules. Obviously you should do that before connecting to a new
# network.
#
# Copyright 2010 Wicher Minnaard (wicher@gavagai.eu)
# License: Creative Commons Attribution-Share Alike 3.0
 
# Do you use dhcpcd for aquiring DHCP leases? And is it running?
dhcpcdLEASEFILE="/var/lib/dhcpcd-${1}.info"
dhcpcdPIDFILE="/var/run/dhcpcd-${1}.pid"
test -f ${dhcpcdLEASEFILE} && test -f ${dhcpcdPIDFILE} && source ${dhcpcdLEASEFILE}
 
# In case you lack the luxury of dhcpcd, where is your resolv.conf?
RESOLV="/etc/resolv.conf"
 
# No user-servicable parts below this line.
DEV="${1}"
 
# I know, I know. But if your routing table contains 0.333.456.789 you have bigger problems ;-)
IPREGEX="\([0-9]\{1,3\}\.\)\{3\}[0-9]\{1,3\}"
 
# Register
MACreg=""
 
# If not run as root, bail
[ "$(id -u)" != "0" ] && echo "You need root privileges to modify networking parameters. Exiting." 1>&2 && exit 2
 
getmac(){
# sets MAC register by IP. Sets to nil, if the MAC is not on the local link. 
  getMAC=$(ip neigh show ${1} | awk '{print $5}')
  if [ -z "${getMAC}" ]; then
    arping -c1 -I ${DEV} ${1} > /dev/null 2>&1
    getMAC=$(ip neigh show ${1} | awk '{print $5}')
  fi
  MACreg=${getMAC}
}
 
allow(){
  # Whitelists traffic to and from particular IP+MAC pairings and
  # adds them to static ARP.
  IP=${1}
  MAC=${2}
  if [[ -n "${IP}" && -n "${MAC}" ]]; then
    arptables -A INPUT  -s ${IP} --source-mac      ${MAC} -j ACCEPT
    arptables -A OUTPUT -d ${IP} --destination-mac ${MAC} -j ACCEPT
    ip neigh replace ${IP} lladdr ${MAC} nud permanent dev ${DEV}
  fi
}
 
if [ -n "${DEV}" ]; then
  # whitelist the routers
  test -z ${GATEWAYS} && GATEWAYS=$(ip route show dev ${DEV}| sed -n "s:.* via \(${IPREGEX}\).*:\1:p")
  for GWIP in ${GATEWAYS}; do
    MACreg=""
    getmac ${GWIP}
    allow ${GWIP} ${MACreg}
  done
  # whitelist the DNS servers
  test -z ${DNSSERVERS} && DNSSERVERS=$(sed -n "s:^nameserver \(${IPREGEX}\):\1:p" ${RESOLV})
  for DNS in ${DNSSERVERS}; do
    MACreg=""
    getmac ${DNS}
    allow ${DNS} ${MACreg}
  done
  # if using dhcpcd, we can whitelist the DHCP server too
  test -n ${DHCPSID} && getmac ${DHCPSID} && allow ${DHCPSID} ${MACreg}
  # set default policy to DROP    
  arptables -P INPUT DROP
  arptables -P OUTPUT DROP
  # clear out non-hardcoded ARP cache entries
  ip neigh flush nud reachable
  ip neigh flush nud stale
else
  # No argument given, so clean up.
  arptables -F
  arptables -P INPUT ACCEPT
  arptables -P OUTPUT ACCEPT
  ip neigh flush nud permanent
fi

Access remote X11 servers that have their TCP socket disabled

This happens to me regularly. Someone brings a machine along and I want to display some app, running on my machine, on their display. Networked X11 to the rescue, you say? No, their X11 server is started with ‘-nolisten TCP’ wich is the default on most modern Linux distros. Sadly, the TCP socket can’t be enabled ‘in-flight’ — if you decide you do fancy a TCP socket after all, you’ll have to restart your X server which may be a pain if you’re in the middle of something (besides, restarting is just plain uncool).
But there is a way to expose the Unix domain socket as a TCP socket, with the help of socat. The following examples all use bash, so if you run a different shell (if you don’t know, you probably aren’t) you may need to define environment variables differently.

Braindead Proof of Concept (BPOC)

Situation: You want to display an application running on a machine called w00t on another machine, called bling. There’s an X11 server running on bling, but it’s not configured to listen on any TCP socket. DNS is properly setup, so if you ping w00t from bling, you get replies from bling’s IP, and vice versa.

1. On bling, find the domain socket of bling’s X11 server. Have a look in /tmp/.X11-unix/. The socket’s name usually reflects its X server display number (which you can determine by running echo $DISPLAY in an xterm).
2. On bling, run something along the lines of
   socat TCP-LISTEN:6066 UNIX-CONNECT:/tmp/.X11-unix/X0
   This will open up TCP port 6066 on all of bling’s network interfaces, connecting it to the Unix domain socket of the X server.
3. In an xterm on bling, run xhost +. You have now opened up your X11 server to the whole wide world, a silly thing to do. Anyone with access to the TCP socket can now read your keystrokes, read your window contents, click your mouse buttons…
4. In an xterm on w00t, run DISPLAY="bling:66" xclock. You may have noticed that 66 = 6066 – 6000 and indeed, by convention the TCP port number for a certain display is its display number + 6000. Anyhow…. yay, a clock! It’s displayed on bling, but running on w00t.

Improvements

• You may have noticed that in the BPOC, you can use the display on bling only once. socat will allow only one client, and will exit once that client exits. In some situations, you may consider that a feature (it’s a one-time access grant), but in others you may not. If you want a reusable TCP socket, run something along the lines of
  socat TCP-listen:6066,fork,reuseaddr UNIX-CONNECT:/tmp/.X11-unix/X0 which forks off a socat process for every TCP connection.
• You may not want to expose a TCP socket on all interfaces. Maybe you only want to expose a socket on the LAN interface, or on the localhost interface (and wrap the packets in an SSH tunnel). Well, you can, using the ‘bind’ option:
  socat TCP-LISTEN:6066,bind=localhost UNIX-CONNECT:/tmp/.X11-unix/X0
  Now tunnel it over SSH. On w00t, run ssh -L 6011:localhost:6023 bling. Now localhost:6011 on woot is actually localhost:6023 on bling which is actually /tmp/.X11-unix/X0 on bling. So on w00t you can start an xclock with its display on bling by running DISPLAY="localhost:11" xclock.
• xhost + from the BPOC is braindead indeed. There are a couple things you could have done instead, there are good ways of tightening up your authorization scheme.
  • First off, you don’t really need to run xhost + if you properly set up X11 cookies, which you should. Here are some examples on using the xauth scheme, but take note: xauth generate will probably not work on recent X11 releases since the XSECURITY extension is disabled by default. Just use the same cookies on the client and the server.
  • Run xhost +w00t. That’s host-based authentication, which is stupid, but not as stupid as no authorization at all. Any user on w00t can now connect.
  • Suppose that on bling (of course!) you’d run xhost +SI:localuser:theuser with ‘theuser’ being the userID of the unix-user running the socat instance. Now from the point of view of the X server, any client connecting through socat will be coming from ‘theuser’ and will therefore be allowed access. Entertaining, but not much different from just running xhost +. It is something to keep in mind though! Many distros by default add the unix-user that started the X server to the authorization list. That user does not need a cookie. If you run socat as that user you will have the effect of running xhost + even if you run xhost -.
  • Just run a nested X11 server, such as Xnest or Xephyr. This way you put untrusted users in a sandbox, preventing them from snooping your keyboard and windows. It’s the X11 equivalent of a chroot.

Here’s a trick. Many laptop trackpads lack a middle mouse button. On a regular mouse input device, the middle mouse button is the scroll wheel, and when you press it down it emits a button event. In X11 this button event is used to paste the X selection buffer into the position right beneath the cursor (there lies sublime usability in this simple fact).
You can emulate a middle-mouse-button event by pressing the left and right mouse buttons at the same time. Since I lack the manual dexterity to do this on my tiny netbook trackpad I wanted to be able to do middle-mouse-button-paste with my keyboard. Well, that appeared to be easy to accomplish with the X11 Xtest extension for which the Xautomation collection includes a utility in the form of xte. If you’d enter xte 'mouseclick 2' in a terminal (within an X11 session, of course), you’d get the same effect as if you’d just pressed the middle mouse button. Only thing left is to add a keyboard shortcut to run this command; in my favourite window manager, XFCE, this can be done clickwise via the Settings Manager or simply by running something like xfconf-query -c xfce4-keyboard-shortcuts -p '/commands/custom/<Super>v' -s "xte 'mouseclick 2'". I can now paste my X selection buffer by pressing the funny ‘four-wobbly-squares key’ and ‘v’ simultaneously.

My X11 setup contains some truly ugly fonts. In particular, it contains bitmap variants (unaliased, of course) of some common fonts, donated by Adobe. That must have been swell in the nineties but it isn’t anymore. But they’re still included in X.org X11, spreading uglyness when referenced explicitly:

This very much isn’t how I like my webpages. In this case, drilling down to the font declaration (use the excellent Firebug if you have to) yields:

BODY { font-family: "new century schoolbook", times, serif}
H1   { font-family: "new century schoolbook", times, serif}

Is this ‘new century schoolbook’ font one of those Adobe bitmap fonts? ‘xlsfonts’ yields:

$xlsfonts  | grep -i century
-adobe-new century schoolbook-bold-i-normal--0-0-100-100-p-0-iso10646-1
-adobe-new century schoolbook-bold-i-normal--0-0-100-100-p-0-iso10646-1
-adobe-new century schoolbook-bold-i-normal--0-0-100-100-p-0-iso8859-1
-adobe-new century schoolbook-bold-i-normal--0-0-100-100-p-0-iso8859-1

[... dozens more]

So, let’s get our font subsystem to remap this font declaration to something more appealing. You’ll need to edit your fontconfig configuration which is described rather exhaustively elsewhere and add a stanza like this one:

    <match target="pattern" >    
        <test name="family" qual="any" >   
            <string>new century schoolbook</string> 
        </test>target="font">
        <edit mode="assign" name="family" >
            <string>Nimbus Roman No9 L</string>
        </edit>
    </match>

This is, of course, particular to my wishes and the fonts available on my system. Run fc-cache, reload the webpage with the offending font, and tadaaaa:

Much better.

Update @20091208: With Gentoo, you can use the eselect system to accomplish this. Run eselect fontconfig list and pick the number corresponding with 70-no-bitmaps.conf. Enable it with eselect fontconfig enable $thenumber.
From the 70-no-bitmaps.conf file, it appears there’s a much better way of handling this:

<fontconfig>
<!-- Reject bitmap fonts -->
 <selectfont>
  <rejectfont>
   <pattern>
     <patelt name="scalable"><bool>false</bool></patelt>
   </pattern>
  </rejectfont>
 </selectfont>
</fontconfig>

In this post I’ll show you how to do basic ‘cracking’ of a piece of software and make WebDAV work in Windows Vista. Yes, I’m excited too! I’ll be making snarky remarks about ‘Windows’ in the process, though. If you can’t handle that kind of humour stop reading now.

WebDAV. Filesystems over HTTP. What a fantastic invention. Sadly, every Windows version in the past decade has contained a botched implementation of this protocol.
Fact #1. I run WebDAV servers.
Fact #2. Some people still use ‘Windows’.
Herein, as with any collection of facts containing #2, lies a problem.

So I have been on the hunt for a sane third-party implementation for Vista. And it just so happens that a certain ISP (XS4ALL) offers WebDAV access. They must have run into trouble with Vista’s anti-implementation of WebDAV too as they’re offering a third-party WebDAV client to the Windows hoi polloi.
And it happens to work rather well. You get an extra ‘drive’ so to any userspace program it’s just as if it’s interfacing a regular filesystem. One problem, though:

It seems to be ‘configured’ to disallow access to other servers than XS4ALL’s.
Some explanation for fellow GNU/Linux users: To you, this most probably is a foreign concept. Why anyone wishes to artificially limit the usefullness of their code is difficult to grasp. FLOSS users work together to improve eachother’s code. Well, on proprietary systems such as Windows it’s common to have intentionally limited utility and the users are used to it — remember, they are already artificially limiting themselves by not using FLOSS.
We FLOSS users don’t have artificially limited software on our systems. What we do have, however, is nearly endless configurability. So let’s do it our way. Let’s make this thing do what we want it to do. But how? Tell a clueless end-user to reroute their DNS so we can spoof webdisk.xs4all.nl serverside and pray the software will accept our dodgy SSL certificate? Blank stares all around. No, we have to come up with something better – we have to give them an ‘updated’ version of the .exe !

Start by downloading XS4ALL-webdisk.exe from this page. It says its version number is 5.00.06 and its MD5sum should be 9d008d79099cd1c74abe6e0f1397b0a1. If you get a different checksum don’t worry – you may still be able to crack it, because at the very least you know what to look for when you’re done reading this. I can’t provide you with the version I downloaded because I don’t own the copyright and I haven’t received a license to redistribute.
Go ahead and run the installer. Then, try connecting to a WebDAV server of choice and observe that any attempts will be defeated.

Next, get a hex editor. Here’s the freeware one that I picked. With your editor, open wdfsctl.exe from wherever it is you installed the Webdisk. You should see something resembling this:

To the left, in blue, is the offset. It’s the position in the file. To the right of the offset are bytes in hexadecimal representation. To the far right is the text representation of those bytes.
Now, an .exe can have text mixed in with executable code. A text representation of executable instructions does not make sense which is why you encounter copious amounts of gibberish in the right column. But, as you scroll up and down in the file, you’ll discover lots of proper English sentences in the right column. Select the text and the corresponding bytes will be highlighted. Move over to those bytes in the middle column, change them, and observe that the corresponding text representation also changes. It works the other way around, too. Fun as that may seem we can’t go around changing strings (bits of text are called strings) willy-nilly. Specifically, we cannot change their length or position. Why not? Well, bits and pieces of the program are referenced from other bits and pieces of the program by their offset. Change the offset (position) of some program instruction in the binary (by adding text in front of it or something) and you’ll have to update any and all references to this position. It can be done but we aren’t going to do it. HxD helpfully warns you if you’re trying to do it.
OK, let’s go string hunting. We’re looking for something that is matching xs4all.nl since the restriction most probably works by whitelisting. It took me a quite a while to find it, but it’s at offset 6DAB2. You’ll find the string x.s.4.a.l.l...n.l there. Look over to the hex representation and you’ll find it’s a pattern of characters separated by 00. That’s called null-delimited. Mind you, the 00 you see in the hex representation is not the same as ‘00′ in the text representation:

• Enter a ‘.’ in the text representation and you get 2E in the hex representation.
• Enter a ‘0′ in the text representation and you get 30 in the hex representation.
• Enter a 00 in the hex representation and you get a ‘.’ in the text representation.

It’s the hex representation that counts. Now, after some poking around I established that this string itself is null-terminated, too. So, to end the string, the hex representation has to read 00 00 00 because the characters inside the string are null-terminated as well. That’s two levels of null-termination.
After discovering this, it’s time to dick around with patterns. Change the ‘4′ in x.s.4.a.l.l...n.l to a null by typing ‘00′ in the hex representation at byte 06, offset 0006DAB0. Run the program. Try to connect to https://ha.xs/quux . Chances of success are very slim, but the program doesn’t stop you from trying! You can put anything in front of ‘.xs’ and it will try to connect. Change byte 00 to ‘a’ (in the text representation) and convince yourself that the program will now get out of your way should you attempt to connect to https://fabuloushaxs/quux, but it still stops you from connecting to https://fabuloushaxz/quux.
It appears that the string we’ve just changed has to match the end of the host we’re trying to connect to. So, originally, for https://justconnectmetomyserveralready.net/mydir it would check whether it ends with xs4all.nl which, of course, it doesn’t. However, we can make the string very, very short. In fact, we can make it empty, causing it to always match the end of any host we enter. To do that, just enter 00 in the hex representation at byte 00 at offset 0006DAB0. Connect to your favourite WebDAV server. It works, doesn’t it?

As you can see this approach to configuration requires levels of technical comprehension beyond those we can reasonably expect to find in the average wildtype user of this particular proprietary operating system. The WebDisk-program lacks end-user configurability, there is no such thing as easily redistributable /etc/webdisk/*conf with user overrides stored in a ~/.webdiskrc such as we have come to expect from mature operating systems.
I found hex-editing executable files a refreshing approach to configuration management, but I can’t say it’s user friendly. That is sad, because the platform does show some potential — for instance, the file search agent in ‘Windows Explorer’ nicely compensates for any lack of reasoning ability in the users expected to buy in on this platform.
But until there is out-of-the-box support for bog-standard decade-old networking protocols and usability issues like the hex configuration interface are resolved, we’ll have to conclude that Windows is not ready for the desktop yet.

Joking aside, what we have just done is actually pretty basic. ‘Real’ crackers, the folks that let you bypass registration requirements or serial number checks, use tools to look into the memory area of a running program to see which steps make up its behaviour. They don’t just edit some strings, they add and change instructions (and offsets). That requires a much deeper understanding of what’s going on.

There’s various other stuff to be modified. You can change the window title or the help texts. I disabled the auto-update by replacing the URLs at offset 0006BA60, maybe you should do so too.
Stuck? If you want the exact same binary I’m running, binary-patch the original with this diff. You then have the XS4ALL Webdisk “XS2ALL -OH RLY? YA RLY!” edition.

Now for some legalities. For me, to publish how I edited this .exe, is perfectly legal. Redistributing the original program isn’t. Me or you distributing any modified versions isn’t either. That’s basic copyright law. With some types of binary patches it may be different. However, the particular patch posted above is tiny and contains no ‘original work’ – just pointers to which part of the original to replace with my handiwork. So there.

Samsung does not provide a way to flash the BIOS of NC10 netbooks from within Linux. That does not come as a complete surprise for this type of consumer hardware. What is worse is that you won’t even find DOS utilities (for flashing from a bootdisk) on the NC10 page at samsung.com, for that; you have to somehow guess that you need to go to samsungpc.com (!?!!?). Anyway, with FreeDOS and a bootable thumbdrive I am able to flash my BIOS. I then dump the BIOS from Linux with the Flashrom utility.
These are the dumps so far:

• NC10-04CA.rom sig
MD5: 91ea9ac405953cc43bd1422cb261d805
• NC10-07A.rom sig
MD5: 73acd98b4e73fb08d59e9dbdc6a71dae
• NC10-11CA.rom sig
MD5: cf9efb63223ba051b821048f66440828

You can use flashrom to flash your BIOS with these images, all without leaving Linux. I’ve signed them with GPG (keyID 08B6A4AE), and I also provide an MD5 hash which you should check before flashing. You might even want to verify your current BIOS against a provided dump of the same version; this way you can reassure that you have the exact same hardware:

root >>>flashrom -v NC10-07A.rom 
Calibrating delay loop... OK.
No coreboot table found.
Found chipset "Intel ICH7M", enabling flash write... OK.
Found chip "Macronix MX25L1605" (2048 KB) at physical address 0xffe00000.
Flash image seems to be a legacy BIOS. Disabling checks.
Verifying flash... VERIFIED.

It’s probably wise to use flashrom to make a backup dump of the original BIOS of your NC10.
I have an NP-NC10-KA03UK. KA03 is the revision code, printed on the white sticker on the netbooks’ bottom. The roms have been proven to work on other revisions, too — check the comments, or just try it on yours. Please post your revision code if you successfully flashed a revision that hasn’t been posted yet.

One further note: The purchase time in the BIOS will be set to ‘2009/01′ which is when I got mine. But, should the need arise, you may be able to hex-edit the dump and reflash to reflect your true date of purchase (or any date really – you can may as well edit it to announce you bought it in August, 1982 if you feel like it). In the 11CA release you’ll find the date at offset 18031E. If that doesn’t work, just flash it with the backup dump you made. You did dump your original BIOS, didn’t you?

Needless to say, these BIOS images are unofficial. But then again, even flashing your bios with the utilities provided by on samsungpc.com void your warranty. You and you alone are responsible for any damages arising from the use or inability to use the images I provide. They are provided ‘as-is’.

The other day I compiled Firefox 3.5-beta4, and, apart from many improvements, I noticed that I am now affected by the infamous ‘hiccups’. Firefox will stall for seconds at a time on my poor netbook. Details on how this relates to the many fsync() calls made by the persistance layer (SQLite) can be found all over the net.
But I don’t want Firefox to stall and I don’t want to keep my harddisk awake with all these writes when I’m on battery power.
Luckily, both these problems go away if you put your Firefox profile on a ramdisk. There are numerous guides out there that save you from working out the details; I used this one from the Gentoo forums.
The guide uses cron to sync the (presumably) modified contents of the ramdisk – bookmarks, cookies, whatever – back to permanent storage (harddisk). You and I both know that while it’s often convenient to use cron, it’s not always the Right Way of Doing Things®. Why sync if nothing’s changed? I wrote a script that employs the inotify system to do the syncing only when necessary. You’ll find it in the forum thread I linked to earlier, but I post it here “ter lering ende vermaeck”. Depending on your browser there might be a vertical scrollbar at the bottom which lets you read up to the EOL’s ;-)
You can find the latest version at my public repo.

#!/bin/bash
 
# Packfox, a tool to facilitate running Firefox with its profile stored
# in RAM (tmpfs). Copyright 2008-2009 Wicher Minnaard, wicher@gavagai.eu .
# Distributed under the WTFPL, http://smormedia.gavagai.nl/dist/packfox/COPYING
# Latest version available at http://smormedia.gavagai.nl/dist/packfox/
 
 
# Change this to match your profile
PROFILE=$(hostname)
PFDIR="${HOME}/.mozilla/firefox"
# Tar every .. seconds (regardless of changes)
TMOUT="1800"
# But not more often than every .. seconds (regardless of changes)
TMMIN="60"
# Regex for which files not to act on when they're changed.
# Use inotifywait -m -e modify -e move -e create -e delete --exclude '(/Cache/)' -r your_profile_dir
# and watch the output while browsing to determine which regex will be right for YOU.
IEXCL="(.sqlite-journal$)|(\-log.txt$)|(cookies.sqlite$)|(sessionstore\-[0-9].js$)|(/weave/)|(/Cache/)"
# Have you read everything and have you made the necessary adjustments? Then remove the line below ;-)
echo "I should read the README and adjust the script variables before running this." && exit 2
 
 
# No user servicable parts below this line.
TGT="${PFDIR}/${PROFILE}"
 
# Global vars
INOTYPID=""
SLEEPPID=""
PACKLOCK=""
 
# Cleanup function
terminate(){
  # If we are the daemon and we get SIGINTed/SIGTERMed, kill our children
  # and if not already packing, do one last round of packing.
  if [ "$(basename ${0})" == "packfox-daemon" ]
  then
    if [ -n "${INOTYPID}" ]; then kill ${INOTYPID}; fi
    if [ -n "${SLEEPPID}" ]; then kill ${SLEEPPID}; fi
    if [ -z "${PACKLOCK}" ];then packup; fi
    exit
  fi
}
 
# For cleaning up 
trap terminate SIGINT SIGTERM
 
# Suicide with goodbye note. If gxmessage is installed, use that.
seppuku(){
  echo "${1}" 1>&2
  which gxmessage > /dev/null 2>&1 && gxmessage -nofocus -title "$(basename ${0})" "${1}" || xmessage "${1}"
  exit 2
}
 
# Checks and setup
test -d "${PFDIR}" || seppuku "Profile dir doesn't exist"
if [ -z "$(mount -t tmpfs | grep -F "${TGT}" )" ]
then
    mount "${PFDIR}/${PROFILE}" || seppuku "Mounting of profile's tmpfs failed. Check /etc/fstab and the output of 'dmesg'."
fi
test -f "${TGT}/.unpacked" || tar -xpf "${PFDIR}/${PROFILE}.packed.tar" -C "${PFDIR}" \
&& touch "${TGT}/.unpacked" || seppuku "Error unpacking the profile tarball. You might want to use the backup tarball located in ${PFDIR}."
 
# This tars up the profile
packup(){
  PACKLOCK="locked"  
  cd "${PFDIR}"
  tar --exclude '.unpacked' -cpf "${PFDIR}/${PROFILE}.packed.tmp.tar" "${PROFILE}"
  mv "${PFDIR}/${PROFILE}.packed.tar" "${PFDIR}/${PROFILE}.packed.tar.old"
  mv "${PFDIR}/${PROFILE}.packed.tmp.tar" "${PFDIR}/${PROFILE}.packed.tar"
  PACKLOCK=""
}
 
# No daemon, just packing
if [ "$(basename ${0})" == "packfox" ]; then packup; fi
 
# The daemon loop
if [ "$(basename ${0})" == "packfox-daemon" ]
then
  which inotifywait >/dev/null 2>&1 || seppuku " You'll need the 'inotify-tools' package for this script. Get it at http://inotify-tools.sourceforge.net or from your distro's repos".
  while true
    do inotifywait -q -q -t ${TMOUT} -e modify -e move -e create -e delete --exclude "${IEXCL}" \
    -r "${PFDIR}/${PROFILE}" &
    INOTYPID=${!}
    wait ${INOTYPID}; INOTIFYPID=""
 
    packup
 
    sleep ${TMMIN} &
    SLEEPPID=${!}
    wait ${SLEEPPID}; SLEEPPID=""
    done
  exit
fi

[Update: Het NOS-journaal heeft vodcasts, vrij te downloaden in h.264-formaat. Hoera! Hoe zouden ze dat met rechten geregeld hebben? Hoe dan ook, de algemene principes die hieronder worden uiteengezet zijn helaas nog wel geldig voor de content van de omroepen op bijvoorbeeld uitzendinggemist.nl - waarvoor hier een GreaseMonkey-script dat dumpen van de stream vereenvoudigt.]
Vandaag gaan we illegale software gebruiken om het NOS-journaal te kunnen downloaden. Wat we gaan doen is niet illegaal, maar het gekke is dat de software die we nodig hebben om iets legaals te doen, illegaal is! Dat is raar, heeft met de aard van het internet te maken, dus dat verdient een blogpost.
Eerst wat uitleg over hoe de wereld in elkaar zit. Daarna gaan we aan de slag. Ongeduldig, en niet geïnteresseerd in de wereld? Scrollen!

Fair use

Scenario: Je wilt het journaal kijken.

• maar niet nu, nee, straks in de bus
• of ergens anders waar geen internetverbinding is
• of je wilt er in kunnen spoelen zonder problemen
• of je wilt de aflevering archiveren
• of delen knippen & plakken naar zelfgemaakt videomateriaal (”remixen”)
• of je wilt niet helemaal die NOS-site navigeren, maar gewoon met één druk op de knop klaar zijn…

Downloaden dus. En dat wordt je zo moeilijk mogelijk gemaakt, ondanks het feit dat dat journaal van het door jou afgestane belastinggeld wordt gemaakt. Bovendien wordt het beschikbaar gesteld in een formaat waarvoor licenties van Microsoft of Windows-software nodig is — alsof de overheid alleen nog maar Audi’s zou toestaan op de publieke weg.

Raar, toch? Waarom kun je niet gewoon het journaal downloaden en het ergens op je gemak bekijken, of een fragment laten zien in een presentatie [edit: in privésfeer!] of zo? Dat heet ‘fair use’.

Geen fair use voor jou!

Voor de NOS is het probleem met ‘fair use’ dat ze daarvoor het videomateriaal moet aanbieden op een manier waarop ze verdere controle over het materiaal verliezen. Een voorbeeld: Als jij je gedownloade afleveringen in de bus bekijkt, dan is dat ‘fair use’. Maar ga jij een beeldbank beginnen met al het materiaal dat de NOS van een extern mediabedrijf betrekt (zaken als video’s van die aardbeving in Italië laatst), dan heeft de NOS een probleem omdat dit niet hun materiaal is, ze hebben het slechts in licentie. En die licentie voorziet hoogstwaarschijnlijk niet in ongebreidelde herverspreiding – dan zou de bezittende mediabedrijf het gras voor zijn eigen voeten wegmaaien, of hoe noem je dat.

Het punt is dus dat het heel moeilijk wordt dat laatste (copyrightschending) te verhinderen als je downloaden toestaat. Daarom wordt er gestreamd, en streamen gaat doorgaans in een nogal lullige resolutie. Jammer voor ons!
Maar streamen is toch eigenlijk een soort downloaden? Dat klopt. Als je het journaal bekijkt op de manier zoals de NOS het bedoeld heeft (en 99% van de mensen doet dat zo) komt het materiaal inderdaad langs je computer – maar het zit dan “opgesloten” in een mediaplayer die jou niet toestaat om dat wat er aan materiaal langszoeft op te slaan. Dat heet DRM – officieel Digital Rights Management, in de volksmond Digital Restrictions Management – en de mogelijkheid tot het opleggen van restricties is één van de redenen waarom de omroepen niet een open, licentievrij formaat als Ogg Theora gebruiken – een formaat waarvoor iedereen een mediaspeler kan programmeren, voor een desktopbesturingssysteem naar keuze. Want als iedereen zijn eigen mediaspeler kan schrijven, wie garandeert dan dat er beperkingen zullen worden ingebouwd om het opslaan van streams tegen te gaan?

Dus stop het materiaal in een niet-vrij formaat waarvoor een licentie van Microsoft nodig is, een licentie die Microsoft niet geeft aan makers van programma’s die opslaan van streams toestaan, en klaar! Toch?
Het is een handhavingsprobleem dat wordt afgewikkeld op de gebruikers, en wel door deze gebruikers sterk in hun vrijheden te beperken. Het is niet de fout van de NOS dat de wereld op deze manier in elkaar zit. Er is, voor zover ik weet, nog geen oplossing voor dit probleem zonder ingrijpende veranderingen in bedrijfsmodellen.

Ondertussen willen we nog steeds het journaal downloaden en daar fair-use dingen mee doen.

MPlayer to the rescue

Er is een relatief kleine, maar bovengemiddeld technisch onderlegde gemeenschap van gebruikers van en ontwikkelaars voor besturingssystemen uit de Linux/BSD-hoek. Via reverse engineering en ongetwijfeld wat hacks zijn er mediaspelers gemaakt die het ‘geheime’ video- en streamingformaat kunnen lezen, zonder daarvoor een licentie te kopen. Ik weet niet hoe het in Europa zit – wij hebben gelukkig geen softwarepatenten en geen DMCA – maar onder de Amerikaanse patentwetgeving is dat illegaal.

Download dus MPlayer van http://www.mplayerhq.hu, of installeer ‘m via je packagemanager. Gebruik van deze software is niet illegaal. De makers zijn mogelijk niet legaal bezig, maar wij blijven vandaag legaal.

De syntax is ongeveer zo (in een shelletje natuurlijk):

mplayer -dumpstream -user-agent browserid URL

waarbij browserid de browseridentificatiestring is zoals deze in de logs van de webserver van de publieke omroep zal gaan verschijnen, en URL de URL van de stream is.
Voor de browseridentificatie kun je alles invullen wat je maar wilt. Je kunt ‘m “Sjoernaaldownlooier 2.03 Beta” noemen als je wilt. Maar het punt is dat we niet willen opvallen, want we willen dit graag kunnen blijven doen. Een zeer geschikte browseridentificatiestring is daarom

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Dat is die van een niet al te beste browser op een niet al te chique besturingssysteem
[edit: we moeten natuurlijk de UA van de player hebben, niet die van de browser!]
Windows-Media-Player/11.0.6001.7000
Dat is die van een mediaspeler met DRM, maar dat is dus wat de hoi polloi gebruiken.

Vervolgens moeten we MPlayer nog een URL geven. Uitvissen welke dat is is nontriviaal – een simpele ‘view source’ op de afspeelpagina van site van de NOS is niet voldoende omdat het streamobject met javascript geïnjecteerd wordt. Met de Firebug-extensie voor Firefox kun je wel de URL terugvinden, dus als je ook willekeurige afleveringen van uitzendinggemist.nl wilt downloaden kun je die gebruiken. De URL die ik gevonden heb wijst altijd naar het laatste journaal, hartstikke handig:

http://cgi.omroep.nl/cgi-bin/streams?/nos/journaal/laatstejournaalBB.wmv

Dus dan wordt het:

mplayer -dumpstream -user-agent 'Windows-Media-Player/11.0.6001.7000' 'http://cgi.omroep.nl/cgi-bin/streams?/nos/journaal/laatstejournaalBB.wmv'

(maar dan op één regel). Downloaden duurt net zolang als de aflevering zelf, even geduld dus. De output laat geen voortgang zien. Maar als je dit ziet is het goed gegaan:

Everything done. Thank you for downloading a media file containing proprietary and patented technology.
Core dumped ;)

Exiting... (End of file)

Voor de niet-nerds: Dit is humor, en ja, die “;)” is een knipoogsmiley. Hoe dan ook – we hebben nu een bestand ’stream.dump’. Die kun je afspelen met

mplayer stream.dump
Presto!

En verder

Als je echt wilt reltrappen kun je de boel nog met bijvoorbeeld ffmpeg2theora in een vrij en open formaat omzetten en verspreiden via BitTorrent. Dat is niet legaal. Het is ook niet legaal om dit (mbv wat scripts) met alle afleveringen van alles dat op uitzendinggemist.nl te verkrijgen is te doen. Niet legaal, maar wel technisch haalbaar, en zo’n onuitwisbare middelvinger van een terabyte is een statement zonder weerga dat wij het handhavingsprobleem van iemand anders niet op ons bordje willen krijgen, onderwijl vreugde verspreidend onder iedereen die ook in de bus een aflevering van ‘t een of ander wil bekijken.
Klinkt dat als imagine-all-the-people-dromerij? De Noorse publieke omroep stelt zelf programma’s beschikbaar in een open formaat, in hoge resolutie, via BitTorrent, zonder DRM.