Streaming TV

… some quick notes on getting streaming digital TV working under Linux.

Kernel and driver

According to Hal, you need at least 2.6.15. On boot, a whole load of modules are loaded:

$ lsmod | grep ^cx
cx88_blackbird         21692  0
cx88_dvb               10660  0
cx8802                 12740  2 cx88_blackbird,cx88_dvb
cx8800                 34668  1 cx88_blackbird
cx88xx                 64384  4 cx88_blackbird,cx88_dvb,cx8802,cx8800

Sometimes, when the machine boots it doesn’t seem to detect the card correctly.

cx88[0]: Your board isn't known (yet) to the driver.  You can
cx88[0]: try to pick one of the existing card configs via
cx88[0]: card=n insmod option.

… however, after a reboot it was happy again.

Install stuff

We use the Debian packages dvb-utils, mplayer, and vlc.

Scanning for channels

You need a channels.conf file, which tells various apps what frequencies and tuning parameters to use for all the different logical channels. This is generated thusly:

$ scan /usr/share/doc/dvb-utils/examples/scan/dvb-t/au-sydney_north_shore | tee  channels.conf

… however for some reason at KEG we can’t get SBS on its normal frequency, only on the Wollongong one. So we have to scan that manually:

$ cat wollongong.scan
# T freq bw fec_hi fec_lo mod transmission-mode guard-interval hierarchy
T 711625000 7MHz 2/3 NONE QAM64 8k 1/8 NONE
$ scan wollongong.scan | tee  channels.conf

Now put channels.conf in the right place for different apps:

$ mkdir -p ~/.tzap ~/.mplayer
$ mv channels.conf ~/.tzap
$ ln -s ~/.tzap/channels.conf ~/.mplayer/channels.conf.ter

Here is the final channels.conf file, also including the (mostly unused) D44 channels.

Testing reception

$ mplayer dvb://SBS HD

You can edit the channel names in the channels.conf file to make them easier to use (it’s the first field on each line).

Streaming

With dvbstream

The dvbstream package contains a simple app that can dump a stream to the network. To do unicast streaming to a specific host, we used netcat. The dvbstream program has its own options for generating multicast streams, but we didn’t try this. On the host you want to view the stream do something like:

nc -u -l -p portnum | mplayer -cache 8000 -vo xv -

And then on the TV box, we first used tzap to tune the channel in, and then ran dvbstream:

$ tzap SBS HD
^C to interrupt it once it acquires a signal (FE_HAS_LOCK)
$ dvbstream -v 102 -a 103 -o | nc -u target portnum

102 and 103 are the video and audio program IDs respectively. You can find them in the relevant line of channels.conf. dvbstream also has options to specify the frequency and tuning parameters, so it may not be necessary to use tzap, however it doesn’t use channels.conf like tzap does, so you may want this script (by Bodo Bauer) to parse the channels file and invoke dvbstream for you.

With VLC

vlc is an all-singing all-dancing media streamer, but in the end we don’t use it to do much more than dvbstream does (the only real advantage it offers for multicast is generation of the SAP announcements). See the VLC streaming HOWTO for more detailed docs. Here is a script to invoke vlc with all the right arguments, you will probably want to edit it to suit your setup:

#!/bin/bash

# machines to stream to (unicast)
# each additional client multiplies the bandwidth required
# leave unset to use multicast
#clients="zarquon"

# channels.conf file to use
conf="$HOME/.tzap/channels.conf"

# work out frequency/program ID
# default to SBS, it has the soccer :)
[ $# == "0" ] && ch='SBS HD' || ch=$1
if grep -q ^"$ch" $conf; then
        chan=$(grep -i ^"$ch" $conf | head -1 | cut -d: -f1)
        freq=$(grep -i ^"$ch" $conf | head -1 | cut -d: -f2)
        prog=$(grep -i ^"$ch" $conf | head -1 | cut -d: -f13)
        echo "$chan, frequency $freq, program ID $prog"
else
        echo "Unknown channel $ch" > /dev/stderr
        exit 1
fi

# construct arguments to vlc
if [ ! -z "$clients" ]; then # unicast
  dsts=""
  for host in $clients; do
    dsts="${dsts}dst=standard{access=udp,mux=ts,url=${host}},"
  done
  sout="#duplicate{${dsts}}"
else # multicast
  dst="access=udp,mux=ts,url=239.255.42.42,sap,name=\"KEG TV: $chan\""
  sout="#standard{$dst}"
fi

# run the damn thing!
exec vlc -I dummy --ttl 1 --program=$prog dvb: \
--dvb-frequency=$freq --dvb-bandwidth=7 --sout "$sout"

Notes:

  • $prog is the program ID from channels.conf
  • all Australian DVB channels have 7MHz bandwidth (without this option, vlc can’t tune)
  • a TTL of 1 prevents the stream from crossing any routers; increase this only if you need to
  • 239.255.x.x is in the private space for multicast IPs, the .42.42 is arbitrary

To view this stream on a client, you could do:

$ vlc udp:@239.255.42.42

… or alternatively, enable the SAP Announcements discovery service (Settings-Preferences-Playlist-Services Discovery), and select KEG TV in the playlist when it pops up automatically.

Mumudvb

I haven’t tried it, but its author claims that:

  • Mumudvb is a program that can redistribute streams from DVB on a network using multicasting. It is able to multicast a whole DVB transponder by assigning each channel to a different multicast IP. It has be written to use as few CPU in order to stream a lot of channels with one small server.

… it might be worth a try.

Common problems

Stream doesn’t arrive

Check that the kernel supports multicast (CONFIG_IP_MULTICAST). Apparently the default Gentoo kernels have it disabled. Check for any firewall rules that may be blocking the UDP multicast traffic. With iptables:

iptables -A INPUT -p udp --dst 224.0.0.0/4 --dport 1025: -j ACCEPT

Stream starts playing but stops/pauses after a few minutes

This one was tricky. It turned out that the firewall rules were allowing the UDP data, but blocking the IGMP packets. After a while the switch timed out and stopped sending the stream out that port. The solution was:

iptables -A INPUT -p igmp -j ACCEPT
iptables -A OUTPUT -p igmp -j ACCEPT

Multicast across networks

To get streaming working across to other networks:

  1. Enable IGMP on the switch (not really necessary, but it did fix a problem with 10MBit devices getting saturated with unwanted traffic).
  2. Add a route on the gateway box for multicast (put this in startup scripts):
    • route add -net 224.0.0.0 netmask 240.0.0.0 dev eth1
  3. Make sure any firewall rules allow multicast out, and don’t attempt to NAT it.
  4. Install and run pimd on the gateway box.
  5. Increase the TTL in the vlc arguments from 1 to 2 (or higher if needed).

Rack power control script

This one started off quite simply and turned into a monster by the gradual addition of features. We use it to reboot and power up or down our various development machines. Here’s what it does:

  • uses native SNMP to talk to APC networked power bars, turning machines on/off or rebooting them
  • alternatively/also uses ipmitool to use the IPMI management controller to do the same thing directly at the machine
  • finally, also talks to a conserver to make sure that it doesn’t do something nasty to a machine that someone is currently using

It probably doesn’t make much sense outside ETH in its current form, but there are bits of code in there that might prove useful. An obvious first step might be factoring them out into modules (and moving the configuration out of the program!).

   1 #!/usr/bin/env python
   2 # -*- coding: utf-8 -*-
   3 
   4 # Andrew Baumann andrewb@inf.ethz.ch, 2008/02/27 2008/11/17 2009/07/02
   5 
   6 import socket, popen2, sys, os, time
   7 from pysnmp.entity.rfc3413.oneliner import cmdgen
   8 from pysnmp.proto import rfc1902
   9 from optparse import OptionParser
  10 
  11 # delay in seconds to wait before querying IPMI status after sending a command
  12 IPMI_DELAY = 4
  13 
  14 # enable verbose/debug output
  15 debug_enable = False
  16 
  17 # command constants
  18 class commands:
  19     ON    = 1
  20     OFF   = 2
  21     RESET = 3
  22 
  23 class config:
  24     community = cmdgen.CommunityData('my-agent', 'private', 0)
  25     
  26     snmp_port = socket.getservbyname('snmp', 'udp')
  27     power1 = ('power1', snmp_port)
  28     power2 = ('power2', snmp_port)
  29     
  30     ports = {
  31         # host/console: [(powerbar, outlet)]
  32         'nos1':         [(power1, 16)],
  33         'nos2':         [(power2, 16)],
  34         'nos3':         [(power1, 15)],
  35         'nos4':         [(power2, 15)],
  36         'nos5':         [(power1, 14)],
  37         'nos6':         [(power2, 14)],
  38         'gruyere':      [(power1,10), (power1,11), (power1,12), (power1,13)],
  39         'sbrinz1':      [(power2, 7), (power2, 8)],
  40         'sbrinz2':      [(power2, 9), (power2, 10)],
  41     }
  42     
  43     ipmi = {
  44         # host/console: (ipmi-host, user, password)
  45         'gruyere':      ('gruyere-mgmt', 'foo', 'bar'),
  46         'sbrinz1':      ('sbrinz1-mgmt', 'foo', 'bar'),
  47         'sbrinz2':      ('sbrinz2-mgmt', 'foo', 'bar'),
  48     }
  49 
  50 # print a message if debug is enabled
  51 def debug(msg):
  52     if debug_enable:
  53         print os.path.basename(sys.argv[0]) + :  + msg
  54 
  55 # run a process and capture its output
  56 def runcmd(cmdline):
  57     child = popen2.Popen3(cmdline, True)
  58     out = child.fromchild.readlines()
  59     err = child.childerr.readlines()
  60     
  61     ret = child.wait()
  62     if ret != 0 or err != []:
  63         msg = '%s' exited %d % (cmdline, ret)
  64         if err != []:
  65             msg = msg + ', stderr follows:\n' + '\n'.join(err)
  66         debug(msg)
  67 
  68         # only raise an exception if they returned non-zero
  69         if ret != 0:
  70             raise Exception(msg)
  71     
  72     return out
  73 
  74 class conserver_client:
  75     def __init__(self):
  76         self.consoles = self.__getstate()
  77     
  78     def who_owns(self, consolename):
  79         return self.consoles.get(consolename)
  80     
  81     def __getstate(self):
  82         ret = {}
  83         for line in runcmd('console -i'):
  84             parts = line.strip().split(':')
  85             conname, child, contype, details, users, state = parts[:6]
  86             ret[conname] = None
  87             if users:
  88                 for userinfo in users.split(','):
  89                     mode, username, host, port = userinfo.split('@')[:4]
  90                     if 'w' in mode:
  91                         ret[conname] = username
  92         return ret
  93 
  94 class ipmi_client:
  95     command_map = {
  96         commands.ON:    'on',
  97         commands.OFF:   'off',
  98         commands.RESET: 'reset',
  99     }
 100 
 101     def _mkcmd(self, (host, user, password), cmd):
 102         return 'ipmitool -H %s -U %s -P %s power %s' % (host, user, password, cmd)
 103     
 104     def get(self, controller):
 105         return runcmd(self._mkcmd(controller, 'status'))[0].split()[-1]
 106     
 107     def set(self, controller, cmd):
 108        runcmd(self._mkcmd(controller, self.command_map[cmd]))
 109 
 110 class apc_control:
 111     def __init__(self):
 112         self.cg = cmdgen.CommandGenerator()
 113     
 114     port_control_oid = (1,3,6,1,4,1,318,1,1,12,3,3,1,1,4)
 115     
 116     def state_to_string(self, state):
 117         try:
 118             return ['on', 'off', 'rebooting'][state - 1]
 119         except:
 120             raise Exception('Invalid state %d' % state)
 121     
 122     command_map = {
 123         commands.ON:            1, # immediateOn
 124         commands.OFF:           2, # immediateOff
 125         commands.RESET:         3, # immediateReboot
 126         #:                      4, # delayedOn
 127         #:                      5, # delayedOff
 128         #:                      6, # delayedReboot
 129         #:                      7, # cancelPendingCommand
 130     }
 131     
 132     def get(self, (dst, portnum)):
 133         # construct a get request
 134         target = cmdgen.UdpTransportTarget(dst)
 135         oid = self.port_control_oid + (portnum,)
 136         
 137         ret = self.cg.getCmd(config.community, target, oid)
 138         errorIndication, errorStatus, errorIndex, varBinds = ret
 139         
 140         assert(not (errorIndication or errorStatus))
 141         
 142         try:
 143             [(obj, retval)] = varBinds
 144             assert(obj == oid)
 145             assert(retval is not None)
 146         except:
 147             raise Exception(unexpected data returned from SNMP command)
 148         return self.state_to_string(int(retval))
 149     
 150     def set(self, (dst, portnum), cmd):
 151         # construct a set request
 152         target = cmdgen.UdpTransportTarget(dst)
 153         oid = self.port_control_oid + (portnum,)
 154         val = rfc1902.Integer32(self.command_map[cmd])
 155         
 156         ret = self.cg.setCmd(config.community, target, (oid, val))
 157         errorIndication, errorStatus, errorIndex, varBinds = ret
 158         
 159         assert(not (errorIndication or errorStatus))
 160         
 161         try:
 162             [(obj, retval)] = varBinds
 163             assert(retval == val)
 164         except:
 165             raise Exception(unexpected data returned from SNMP command)
 166 
 167 def parse_args():
 168     p = OptionParser(usage='%prog [options] [victim]',
 169                     description='APC powerbar / IPMI control utility')
 170     
 171     p.add_option('-u', action='store_const', dest='cmd', const=commands.ON,
 172                  help='switch outlet on')
 173     p.add_option('-d', action='store_const', dest='cmd', const=commands.OFF,
 174                  help='switch outlet off')
 175     p.add_option('-r', action='store_const', dest='cmd', const=commands.RESET,
 176                  help='power cycle (reboot) if already on, switch outlet on if off')
 177     p.add_option('-i', action='store_false', dest='ipmi', default=True,
 178                  help=don't use IPMI, force use of the power bar)
 179     p.add_option('-v', action='store_true', dest='verbose', default=False,
 180                  help=verbose output)
 181     p.set_defaults(cmd=None)
 182     
 183     options, args = p.parse_args()
 184     if len(args) == 0:
 185         victim = None
 186     elif len(args) == 1:
 187         victim = args[0]
 188         if not (config.ports.has_key(victim) or config.ipmi.has_key(victim)):
 189             p.error('unknown victim %s' % victim)
 190     else:
 191         p.error('more than one victim specified')
 192     if options.cmd is not None and victim is None:
 193         p.error('no victim specified for command')
 194     return victim, options
 195 
 196 def main():
 197     victim, options = parse_args()
 198     global debug_enable
 199     debug_enable = options.verbose
 200 
 201     apc = apc_control()
 202     c = conserver_client()
 203     i = ipmi_client()
 204     
 205     if options.cmd:
 206         # check for console ownership
 207         owner = c.who_owns(victim)
 208         if owner and owner != os.environ['LOGNAME']:
 209             sys.stderr.write(
 210              Error: according to conserver %s currently owns this console\n
 211              If you really need to do this, force them off first\n % owner)
 212             return 1
 213         
 214         apccfg = config.ports.get(victim)
 215         ipmicfg = config.ipmi.get(victim)
 216         
 217         # find status of powerbar
 218         if apccfg:
 219             apcstate = map(apc.get, apccfg)
 220         else:
 221             apcstate = None
 222         
 223         if apcstate:
 224             debug(current APC status:  +  .join(apcstate))
 225         
 226         # try to use IPMI if enabled and the port is switched on
 227         if options.ipmi and ipmicfg and (apcstate is None or 'on' in apcstate):
 228             # get current status
 229             status = i.get(ipmicfg)
 230             debug(using IPMI: current status is %s % status)
 231             
 232             # if they asked for a reset but the outlet is off, turn it on
 233             if status == 'off' and options.cmd == commands.RESET:
 234                 options.cmd = commands.ON
 235             
 236             # do it
 237             debug(sending IPMI %s command... % ipmi_client.command_map[options.cmd])
 238             i.set(ipmicfg, options.cmd)
 239 
 240             # make sure it really happened
 241             debug(waiting for %d seconds to check status % IPMI_DELAY)
 242             time.sleep(IPMI_DELAY)
 243             status = i.get(ipmicfg)
 244             debug(IPMI status is now %s % status)
 245             if ((options.cmd == commands.OFF and status == 'on')
 246                 or (options.cmd in [commands.ON, commands.RESET] and status == 'off')):
 247                 print Warning: IPMI status is still %s, trying again % status
 248                 if options.cmd == commands.RESET:
 249                     i.set(ipmicfg, commands.ON)
 250                 else:
 251                     i.set(ipmicfg, options.cmd)
 252 
 253         else:
 254             # use APC on every configured port
 255             for p in apccfg:
 256                 debug(APC: port %d on %s % (p[1], p[0][0]))
 257                 apc.set(p, options.cmd)
 258     
 259     else:
 260         # print current status
 261         if victim:
 262             victims = [victim]
 263         else:
 264             victims = list(set(config.ports.keys() + config.ipmi.keys()))
 265             victims.sort()
 266         
 267         formatstr = %-10s %-15s %-4s %s
 268         print formatstr % ('VICTIM', 'POWER', 'IPMI', 'OWNER')
 269         for victim in victims:
 270             owner = c.who_owns(victim) or 
 271             
 272             apccfg = config.ports.get(victim)
 273             if apccfg:
 274                 apcstate = map(apc.get, apccfg)
 275             else:
 276                 apcstate = []
 277             
 278             ipmicfg = config.ipmi.get(victim)
 279             if options.ipmi and ipmicfg and ('on' in apcstate):
 280                 try:
 281                     ipmistate = i.get(ipmicfg)
 282                 except:
 283                     ipmistate = ERR
 284             else:
 285                 ipmistate = 
 286             
 287             print formatstr % (victim,  .join(apcstate), ipmistate, owner)
 288 
 289     return 0
 290 
 291 if __name__ == '__main__':
 292     sys.exit(main())

rackpower.py

How to use POSIX capabilities to allow non-root users to perform network packet capturing on Linux

… because it seemed a sensible thing to do, but I couldn’t find a single concise document saying how to do it! These instructions worked for me on a recent Debian testing installation, YMMV.

For specific users (or, the right way)

  1. Install the tools:
    sudo aptitude install libcap2-bin
  2. Grant the necessary programs the ability to inherit the CAP_NET_RAW capability when execed:
    sudo /sbin/setcap cap_net_raw+ei /usr/bin/dumpcap
    sudo /sbin/setcap cap_net_raw+ei /usr/sbin/tcpdump

    (dumpcap is the backend used by Wireshark).

  3. Add the pam_cap module to your authentication process, eg. add something like:
    auth    required        pam_cap.so

    … to the top of /etc/pam.d/common-auth.

  4. Create /etc/security/capability.conf (using, for example, /usr/share/doc/libcap2-bin/examples/capability.conf), and grant the desired users the CAP_NET_RAW privilege. For example:
    # needed to run packet sniffers
    cap_net_raw             andrewb

For all users (or, the lazy/easy way)

Replace step 2 above with:

sudo /sbin/setcap cap_net_raw+ep /usr/bin/dumpcap
sudo /sbin/setcap cap_net_raw+ep /usr/sbin/tcpdump

… and skip the remaining steps. This means that every time these processes are execed, they will automatically acquire the CAP_NET_RAW capability.

EXIF timestamp adjustment

This is a script to modify the timestamps in one or more EXIF tags on JPEG images by a fixed offset. It is useful if you’ve gone overseas, but forgotten to change the clock on your camera, so all the timestamps are incorrectly in the middle of the night (or the day, as the case may be). It requires Benno’s pexif library, version 0.12 or later (it’s also now included in the pexif distribution).

   1 #!/usr/bin/env python
   2 
   3 
   4 Utility to adjust the EXIF timestamps in JPEG files by a constant offset.
   5 
   6 Requires Benno's pexif library: http://code.google.com/p/pexif/
   7 
   8 -- Andrew Baumann andrewb@inf.ethz.ch, 20080716
   9 
  10 
  11 import sys
  12 from pexif import JpegFile, EXIF_OFFSET
  13 from datetime import timedelta, datetime
  14 from optparse import OptionParser
  15 
  16 DATETIME_EMBEDDED_TAGS = [DateTimeOriginal, DateTimeDigitized]
  17 TIME_FORMAT = '%Y:%m:%d %H:%M:%S'
  18 
  19 def parse_args():
  20     p = OptionParser(usage='%prog hours file.jpg...',
  21            description='adjusts timestamps in EXIF metadata by given offset')
  22     options, args = p.parse_args()
  23     if len(args)  2:
  24         p.error('not enough arguments')
  25     try:
  26         hours = int(args[0])
  27     except:
  28         p.error('invalid time offset, must be an integral number of hours')
  29     return hours, args[1:]
  30 
  31 def adjust_time(primary, delta):
  32     def adjust_tag(timetag, delta):
  33         dt = datetime.strptime(timetag, TIME_FORMAT)
  34         dt += delta
  35         return dt.strftime(TIME_FORMAT)
  36 
  37     if primary.DateTime:
  38         primary.DateTime = adjust_tag(primary.DateTime, delta)
  39 
  40     embedded = primary[EXIF_OFFSET]
  41     if embedded:
  42         for tag in DATETIME_EMBEDDED_TAGS:
  43             if embedded[tag]:
  44                 embedded[tag] = adjust_tag(embedded[tag], delta)
  45 
  46 def main():
  47     hours, files = parse_args()
  48     delta = timedelta(hours=hours)
  49 
  50     for fname in files:
  51         try:
  52             jf = JpegFile.fromFile(fname)
  53         except (IOError, JpegFile.InvalidFile):
  54             type, value, traceback = sys.exc_info()
  55             print  sys.stderr, Error reading %s: % fname, value
  56             return 1
  57 
  58         exif = jf.get_exif()
  59         if exif:
  60             primary = exif.get_primary()
  61         if exif is None or primary is None:
  62             print  sys.stderr, %s has no EXIF tag, skipping % fname
  63             continue
  64 
  65         adjust_time(primary, delta)
  66 
  67         try:
  68             jf.writeFile(fname)
  69         except IOError:
  70             type, value, traceback = sys.exc_info()
  71             print  sys.stderr, Error saving %s: % fname, value
  72             return 1
  73 
  74     return 0
  75 
  76 if __name__ == __main__:
  77     sys.exit(main())

exif_timezone.py

Scripts to fetch weather data from the Australian BoM

Contents

  1. Scripts to fetch weather data from the Australian BoM

    1. Current observations
    2. Forecast
    3. Slimserver Weather Display

The Australian Bureau of Meteorology provides a lot of free useful weather data, as well as regular forecasts, but it is in HTML or plain text format. These scripts try to parse relevant parts of that data out for use elsewhere, with varying levels of success.

Current observations

This script and Python module fetches current weather observations by parsing specially-encoded comments in the automatically-updated report; it has been very reliable. To get the data for a station other than Sydney, just change the STATION_NAME variable (and possibly also REPORT_URL).

   1 #!/usr/bin/env python
   2 
   3 # $Id: bom_weather.py,v 1.7 2004/10/26 06:45:04 andrewb Exp $
   4 # script to pull out raw data from Bureau of Meteorology weather observations
   5 # andrewb@cse.unsw.edu.au, 2004/10
   6 
   7 import sys, os, urllib, time, getopt
   8 from HTMLParser import HTMLParser
   9 
  10 REPORT_URL=ftp://ftp2.bom.gov.au/anon/gen/fwo/IDN65066.html
  11 STATION_NAME=Sydney - Observatory Hill
  12 MAX_REPORT_AGE=2*60*60 # 2 hours
  13 
  14 
  15 class BomCommentParser(HTMLParser):
  16     def __init__(self):
  17         HTMLParser.__init__(self)
  18         self.head = None
  19         self.unit = None
  20         self.data = []
  21 
  22     def handle_comment(self, data):
  23         fields = data.strip().split(',')
  24         fields = map(lambda x: x.strip(), fields)
  25 
  26         entrytype = fields[0]
  27         if entrytype not in [HEAD, UNIT, DATA]:
  28             return # some other comment, we ignore it
  29         fields = fields[1:]
  30 
  31         if self.head is None:
  32             assert(entrytype == HEAD)
  33             self.head = map(lambda x: x.lower(), fields)
  34         elif self.unit is None:
  35             assert(entrytype == UNIT)
  36             assert(len(fields) == len(self.head))
  37             self.unit = fields
  38         else: # DATA
  39             assert(len(fields) == len(self.head))
  40             dict = {}
  41             for i in range(len(fields)):
  42                 dict[self.head[i]] = fields[i]
  43             self.data.append(dict)
  44 
  45 
  46 class ScreenLog:
  47     def __init__(self, fh):
  48         self.fh = fh
  49     def log(self, msg):
  50         self.fh.write(Error:  + msg + \n)
  51 
  52 
  53 class FileLog:
  54     def __init__(self, filename):
  55         self.fh = open(filename, 'a')
  56     def log(self, msg):
  57         self.fh.write(time.strftime(%b %d %H:%M:%S ) + msg + \n)
  58 
  59 
  60 def main(argv):
  61     logfilename = None
  62     outfilename = None
  63     try:
  64         opts, args = getopt.getopt(argv[1:], l:)
  65         for (opt, arg) in opts:
  66             if opt == -l:
  67                 logfilename = arg
  68         assert(len(args) = 1)
  69         if len(args) == 1:
  70             outfilename = args[0]
  71     except:
  72         sys.stderr.write(Usage: %s [-l LOGFILE] [FILE]\n % argv[0])
  73         return 1
  74 
  75     try:
  76         if (logfilename is not None):
  77             log = FileLog(logfilename)
  78         else:
  79             log = ScreenLog(sys.stderr)
  80     except IOError:
  81         sys.stderr.write(Error: couldn't append to %s\n % logfilename)
  82         return 2
  83 
  84     try:
  85         parser = BomCommentParser()
  86         fh = urllib.urlopen(REPORT_URL)
  87         parser.feed(fh.read())
  88         fh.close()
  89         parser.close()
  90     except IOError:
  91         log.log(failed to fetch report from %s % REPORT_URL)
  92         return 2
  93     except:
  94         log.log(failed to parse report data)
  95         return 3
  96 
  97     try:
  98         data = None
  99         for entry in parser.data:
 100             if entry[name] == STATION_NAME:
 101                 data = entry
 102                 break
 103         assert(data)
 104     except:
 105         log.log(failed to find data for %s % STATION_NAME)
 106         return 3
 107 
 108     try:
 109         temperature = round(float(data[temperature]))
 110         humidity = round(float(data[relative humidity]))
 111         pressure = round(float(data[pressure]))
 112     except:
 113         log.log(failed to unpack report data)
 114         return 3
 115 
 116     try:
 117         timestamp = time.mktime(time.strptime(data[time], %Y%m%d:%H%M%S))
 118         now = time.mktime(time.localtime())
 119         if (now  timestamp or now - timestamp  MAX_REPORT_AGE):
 120             log.log(timestamp %s out of range, data[time])
 121             return 4
 122     except:
 123         log.log(failed to parse timestamp)
 124         return 3
 125 
 126     try:
 127         if (outfilename is not None):
 128             fh = open(outfilename, 'w')
 129         else:
 130             fh = sys.stdout
 131         fh.write(%d\xb0C %d%% %dhPa\n % (temperature, humidity, pressure))
 132         if (outfilename is not None):
 133             fh.close()
 134     except IOError:
 135         log.log(couldn't write to %s % outfilename)
 136         return 2
 137 
 138     return 0
 139 
 140 
 141 if __name__ == __main__:
 142     sys.exit(main(sys.argv))

bom_weather.py

Forecast

This script and Python module attempts to grab the current long or short forecast as well as the minimum and maximum temperatures for Sydney from the published text file. Unfortunately the format of the file seems to be a little variable, so I have had to change the script a few times to handle different formats and try to make it more robust. It could probably be hacked to handle other state forecasts, or return different min/max temperatures.

document.write(‘Toggle line numbers<\/a>‘);

   1 #!/usr/bin/env python
   2 
   3 # $Id: bom_forecast.py,v 1.9 2006-12-05 22:11:41 andrewb Exp $
   4 # script to grab bureau of meteorology forecast for sydney
   5 # andrewb@cse.unsw.edu.au, 2004/10
   6 
   7 import sys, os, urllib, time, getopt
   8 from bom_weather import FileLog, ScreenLog
   9 
  10 FORECAST_URL=ftp://ftp2.bom.gov.au/anon/gen/fwo/IDN10064.txt
  11 FORECAST_LOCN=City
  12 
  13 
  14 def long_forecast(lines, minmax):
  15     day = time.strftime(%A).lower()
  16     para = None
  17     for line in lines:
  18         if para == None:
  19             if line.strip().lower().startswith(forecast for  + day):
  20                 para = []
  21         else:
  22             if line ==  or line.startswith(Precis:):
  23                 break
  24             else:
  25                 para.append(line)
  26 
  27     forecast =  .join(para)
  28     try:
  29         if minmax:
  30             forecast +=  Max %d % minmax[1]
  31     except ValueError:
  32         pass # no max temperature
  33     return forecast
  34 
  35 def short_forecast(lines):
  36     for line in lines:
  37         if line.startswith(Precis:):
  38             return  .join(line.split(':',1)[1].split()).strip(.)
  39     raise RuntimeError(Couldn't parse short forecast)
  40 
  41 def get_minmax(lines):
  42     found = False
  43     seen_forecast = False
  44     min = max = None
  45     prevword = None
  46     for line in lines:
  47         if line.startswith(Forecast for):
  48             # if we see this twice, it means that the min/max temps are
  49             # for tomorrow's forecast and not today's, so don't return them
  50             if seen_forecast:
  51                 return None
  52             seen_forecast = True
  53         words = line.split()
  54         for word in words:
  55             if found:
  56                 try:
  57                     n = int(word)
  58                     if prevword.lower().startswith('min'):
  59                         min = n
  60                     elif prevword.lower().startswith('max'):
  61                         max = n
  62                 except ValueError:
  63                     pass
  64                 if max:
  65                     return min, max
  66                 prevword = word
  67             elif word == FORECAST_LOCN + ::
  68                 found = True
  69                 prevword = None
  70     return None, None
  71 
  72 def fetch_forecast(do_long_forecast):
  73     fh = urllib.urlopen(FORECAST_URL)
  74     lines = map(str.strip, fh.readlines())
  75     minmax = get_minmax(lines)
  76     if (do_long_forecast):
  77         forecast = long_forecast(lines, minmax)
  78     else:
  79         forecast = short_forecast(lines)
  80         if minmax:
  81             forecast = forecast + ( %d % minmax[1])
  82     fh.close()
  83     return forecast
  84 
  85 def main(argv):
  86     logfilename = None
  87     outfilename = None
  88     do_long_forecast = False
  89     try:
  90         opts, args = getopt.getopt(argv[1:], fl:)
  91         for (opt, arg) in opts:
  92             if opt == -l:
  93                 logfilename = arg
  94             elif opt == -f:
  95                 do_long_forecast = True
  96         assert(len(args) = 1)
  97         if len(args) == 1:
  98             outfilename = args[0]
  99     except:
 100         sys.stderr.write(Usage: %s [-l LOGFILE] [-f] [FILE]\n % argv[0])
 101         return 1
 102 
 103     try:
 104         if (logfilename is not None):
 105             log = FileLog(logfilename)
 106         else:
 107             log = ScreenLog(sys.stderr)
 108     except IOError:
 109         sys.stderr.write(Error: couldn't append to %s\n % logfilename)
 110         return 2
 111 
 112     try:
 113         forecast = fetch_forecast(do_long_forecast)
 114     except IOError:
 115         log.log(failed to fetch forecast from %s % FORECAST_URL)
 116         return 2
 117     except:
 118         log.log(failed to parse forecast data)
 119         return 3
 120 
 121     # fridge hack for very short forecasts
 122     if (len(forecast) = 6):
 123         forecast = Forecast:  + forecast
 124 
 125     try:
 126         if (outfilename is not None):
 127             fh = open(outfilename, 'w')
 128         else:
 129             fh = sys.stdout
 130         fh.write(forecast + \n)
 131         if (outfilename is not None):
 132             fh.close()
 133     except IOError:
 134         log.log(couldn't write to %s % outfilename)
 135         return 2
 136 
 137     return 0
 138 
 139 
 140 if __name__ == __main__:
 141     sys.exit(main(sys.argv))

bom_forecast.py

Slimserver Weather Display

This is unlikely to be useful for many people, but it shows how to use the bom_forecast module in another script. It connects to a running SlimServer (on which the CLI must be enabled) and displays the latest weather forecast for a specified duration. I run it from a cron job to have the weather forecast displayed on my Squeezebox when I wake up.

document.write(‘Toggle line numbers<\/a>‘);

   1 #!/usr/bin/python
   2 
   3 import sys, socket, urllib
   4 from optparse import OptionParser
   5 import bom_forecast
   6 
   7 class SlimCLI:
   8     def __init__(self, host, port=9090):
   9         self.sock = socket.socket()
  10         self.sock.connect((host, port))
  11 
  12     def cmd(self, command, args = [], player = None):
  13         if player != None:
  14             line = player +  
  15         else:
  16             line = 
  17         line += command
  18         for arg in args:
  19             line +=   + urllib.quote(str(arg))
  20         self.sock.sendall(line + \n)
  21         buf = self.sock.recv(4096)
  22         line = buf.splitlines()[0]
  23         return map(urllib.unquote, line.split())
  24 
  25     def close(self):
  26         self.sock.sendall(exit\n)
  27         self.sock.close()
  28 
  29 def parse_args():
  30     p = OptionParser()
  31     p.add_option('-m', type='int', dest='mins', default=30,
  32                  help='time (in minutes) to display weather, default: 30')
  33     return p.parse_args()
  34 
  35 def main():
  36     opts, args = parse_args()
  37     forecast = bom_forecast.fetch_forecast(True)
  38     cli = SlimCLI(localhost)
  39     cli.cmd(display, [Today's weather forecast, forecast, opts.mins * 60])
  40     cli.close()
  41 
  42 if __name__ == __main__:
  43     main()

slimweather.py

SPA3000 syslog parser and logwatch module

This is a script suitable for use with logwatch that parses the syslog debug output from a Sipura SPA3000 VoIP ATA and prints a list of call made and received calls and their times. It might also work with other Sipura models.

   1 #!/usr/bin/env python2.4
   2 
   3 # logwatch script for parsing syslog output from Sipura SPA3000 devices
   4 # andrewb@cse.unsw.edu.au, 2007/01/12
   5 
   6 import sys, re, time, datetime
   7 
   8 # list of all IP addresses for the Sipura; used to detect PSTN calls
   9 # FIXME: make this automatic, or configurable elsewhere
  10 LOCAL_IPS = ['127.0.0.1', '192.168.0.6']
  11 
  12 SYSLOG_RE = re.compile(r'^ *(?Ptime... .. ..:..:..) (?Phost\S*) (?Pdata.*)')
  13 NEWMSG_RE = re.compile(r'^\[\d+:\d+\](?Pdir|-)(?Ppeer\d+\.\d+\.\d+\.\d+:\d+)$')
  14 SIP_URI_RE = re.compile(r'^sip:(?Puser.*)@(?Phost[^:]+)(:(?Pport\d+))?$')
  15 SYSLOG_TIME = '%b %d %H:%M:%S'
  16 DISPLAY_TIME = '%b %d %H:%M'
  17 
  18 STATE_SETUP = 1
  19 STATE_RINGING = 2
  20 STATE_INPROGRESS = 3
  21 STATE_TERMINATED = 4
  22 
  23 class SIPCallState:
  24     def __init__(self):
  25         self.originator = None
  26         self.state = None
  27         self.uri = None
  28         self.starttime = None
  29         self.duration = None
  30 
  31     def got_cmd(self, ts, cmd, arg):
  32         if cmd == 'INVITE':
  33             self.originator = False
  34             self.state = STATE_SETUP
  35         elif cmd == 'BYE' or cmd == 'CANCEL':
  36             self.state = STATE_TERMINATED
  37             if not self.uri:
  38                 self.uri = arg
  39             if self.starttime and not self.duration:
  40                 self.duration = ts - self.starttime
  41         elif cmd == 'ACK':
  42             if self.state == STATE_SETUP or self.state == STATE_RINGING:
  43                 assert(self.uri == arg)
  44                 self.state = STATE_INPROGRESS
  45                 self.starttime = ts
  46 
  47     def got_reply(self, ts, status):
  48         if self.state == STATE_SETUP and status / 10 == 18:
  49             self.state = STATE_RINGING
  50         elif self.state in [STATE_SETUP, STATE_RINGING] and status / 100 == 2:
  51             self.state = STATE_INPROGRESS
  52             self.starttime = ts
  53         elif status / 100 = 4:
  54             self.state = STATE_TERMINATED
  55 
  56     def sent_cmd(self, ts, cmd, arg):
  57         if cmd == 'INVITE':
  58             self.originator = True
  59             self.state = STATE_SETUP
  60             self.uri = arg
  61         else:
  62             self.got_cmd(ts, cmd, arg)
  63 
  64     def sent_reply(self, ts, status):
  65         return self.got_reply(ts, status)
  66 
  67 class StatsGatherer:
  68     def __init__(self):
  69         self.last_peer = self.last_inout = None
  70         self.peers = {}
  71         self.calls_made = []
  72         self.calls_received = []
  73 
  74     def process_entry(self, timestamp, line):
  75         match = NEWMSG_RE.match(line)
  76         if match:
  77             self.last_inout = (match.group('dir') == '-')
  78             ip, port = match.group('peer').split(':', 1)
  79             if ip in LOCAL_IPS:
  80                 ip = LOCAL_IPS[0]
  81             self.last_peer = ip + ':' + port
  82         else:
  83             words = line.split()
  84             is_reply = words[0].startswith('SIP/')
  85             is_command = words[-1].startswith('SIP/')
  86             if is_reply or is_command:
  87                 assert(len(words)  2)
  88                 callstate = self.peers.setdefault(self.last_peer, SIPCallState())
  89                 if is_reply:
  90                     status = int(words[1])
  91                     if self.last_inout:
  92                         callstate.sent_reply(timestamp, status)
  93                     else:
  94                         callstate.got_reply(timestamp, status)
  95                 else:
  96                     if self.last_inout:
  97                         callstate.sent_cmd(timestamp, words[0], words[1])
  98                     else:
  99                         callstate.got_cmd(timestamp, words[0], words[1])
 100                 if callstate.state == STATE_TERMINATED:
 101                     self.__process_call(callstate)
 102                     del self.peers[self.last_peer]
 103 
 104     def __process_call(self, callstate):
 105         if not callstate.duration:
 106             return
 107         if callstate.originator:
 108             self.calls_made.append((callstate.uri, callstate.starttime, callstate.duration))
 109         else:
 110             self.calls_received.append((callstate.uri, callstate.starttime, callstate.duration))
 111 
 112     def print_stats(self, fh):
 113         for (calls, desc, received) in [(self.calls_made, 'made', False),
 114                                         (self.calls_received, 'received', True)]:
 115             if calls != []:
 116                 fh.write('Calls %s:\n' % desc)
 117             for (uri, start, duration) in calls:
 118                 if uri is None:
 119                     uri = 'unknown'
 120                 else:
 121                     match = SIP_URI_RE.match(uri)
 122                     if match:
 123                         host = match.group('host')
 124                         if host in LOCAL_IPS:
 125                             if received:
 126                                 uri = '(PSTN)'
 127                             else:
 128                                 uri = match.group('user')
 129                         else:
 130                             uri = match.group('user') + '@' + host
 131                 fh.write(' %s  %-30s %s\n' % (start.strftime(DISPLAY_TIME), uri, duration))
 132 
 133 def process_syslog(fh, statobj):
 134     prevline = None
 135     for line in fh.readlines():
 136         if line == prevline:
 137             continue
 138         match = SYSLOG_RE.match(line)
 139         timestr = match.group('time')
 140         timestamp = datetime.datetime(*(time.strptime(timestr, SYSLOG_TIME)[:6]))
 141         contents = match.group('data').strip()
 142         if contents != '':
 143             statobj.process_entry(timestamp, contents)
 144         prevline = line
 145 
 146 if __name__ == '__main__':
 147     statobj = StatsGatherer()
 148     process_syslog(sys.stdin, statobj)
 149     statobj.print_stats(sys.stdout)

spa3k.py

ETRN client utility

This is a small utility that uses Python’s SMTP module to send RFC1985 ETRN requests to one or more mail servers. This would normally be used to tell the backup mail servers for a domain to attempt delivery after the primary MX comes back online. I wrote it, because after much searching, the best I could find were scripts using telnet or nc, and with no error handling. This program should speak SMTP correctly, and will return error codes and messages in the case of failure.

   1 #!/usr/bin/python
   2 
   3 # Andrew Baumann andrewb@cse.unsw.edu.au, 2007/04/29
   4 
   5 import sys, smtplib, socket
   6 from optparse import OptionParser
   7 
   8 def parse_args():
   9     mydomain = socket.getfqdn()
  10     p = OptionParser(usage='%prog [option] host...',
  11                      description=sends an ETRN request (RFC1985) to one or more mail servers)
  12     p.add_option('-d', dest='domain',
  13                  help='domain to request mail for, default: %s' % mydomain)
  14     p.add_option('-v', action='store_const', dest='verbose', const=True,
  15                  help='verbose/debug output')
  16     p.set_defaults(domain=mydomain, verbose=False)
  17     options, args = p.parse_args()
  18     if len(args) == 0:
  19         p.error('no host specified')
  20     return options, args
  21 
  22 def do_etrn(host, domain, smtpobj=None):
  23     if smtpobj is None:
  24         smtpobj = smtplib.SMTP()
  25     smtpobj.connect(host)
  26     smtpobj.ehlo()
  27     (retcode, retmsg) = smtpobj.docmd('ETRN', domain)
  28     if retcode / 10 != 25:
  29         raise smtplib.SMTPResponseException(retcode, retmsg)
  30     smtpobj.quit()
  31 
  32 def main():
  33     options, hosts = parse_args()
  34     errorflag = False
  35     smtpobj = smtplib.SMTP()
  36     smtpobj.set_debuglevel(options.verbose)
  37 
  38     for host in hosts:
  39         def error(msg):
  40             sys.stderr.write('Error: %s: %s\n' % (host, msg))
  41 
  42         try:
  43             do_etrn(host, options.domain, smtpobj)
  44         except socket.error, (errnum, errmsg):
  45             error(errmsg)
  46             errorflag = True
  47         except smtplib.SMTPResponseException, e:
  48             error('%d %s' % (e.smtp_code, e.smtp_error))
  49             errorflag = True
  50         except smtplib.SMTPException:
  51             error('Generic SMTP error')
  52             errorflag = True
  53 
  54     return errorflag
  55 
  56 if __name__ == __main__:
  57     if main():
  58         sys.exit(1)

etrn.py

OzTiVo to DVD conversion

Here’s how I convert programs on the TiVo to DVDs for archival.

Things needed

  • A TiVo, with the video resolution set to 720×576 which avoids the need to resize and thus transcode the video
  • Playitsam with the following patches applied:
    1. This patch fixes large file support so the program doesn’t crash when writing a stream larger than 2G.
      --- hdr/defines.h       19 Oct 2002 03:56:07 -0000      1.1
      +++ hdr/defines.h       1 Jan 2007 02:05:50 -0000       1.2
      @@ -3,4 +3,8 @@
       #ifdef __linux__
       #define __USE_FILE_OFFSET64
       #define __USE_LARGEFILE64
      +
      +#define _LARGEFILE_SOURCE
      +#define _LARGEFILE64_SOURCE
      +#define _FILE_OFFSET_BITS 64
       #endif
    2. This patch changes it to use madplay and mplayer instead of the ancient mpg123 and mpeg2dec (NB: if you want to use gplayitsam you will need to make a similar change there):
      --- unix/unixcli.c      19 Oct 2002 03:55:44 -0000      1.6
      +++ unix/unixcli.c      31 Dec 2006 22:59:59 -0000      1.7
      @@ -18,8 +18,8 @@
       static struct termios oldtios, newtios;         /* Console settings for raw */
                                                       /* and cooked modes */
      
      -#define AUDIO_PROG mpg123 - 2 /dev/null
      -#define VIDEO_PROG mpeg2dec -o x11 2 /dev/null
      +#define AUDIO_PROG madplay -Q -
      +#define VIDEO_PROG mplayer -ao null -vo xv -  /dev/null 2 /dev/null
      
       /* Open the output devices used for playback. Returns
        * 1 if ok, 0 if an error.
  • The ffmpeg, mjpegtools, dvdwizard, dvdauthor and dvd+rw-tools packages and their dependencies.

Procedure

  1. Use TivoWeb to get the list of FSIDs for the program; for example: 18022,18024,18025
  2. Run Playitsam, giving it the TiVo’s hostname, the FSIDs, and the output file names:
    playitsam tivo:18022,18024,18025 program.mpa program.mpv
  3. Perform editing operations in Playitsam (see the documentation for details). Then, either:
    • If you are going to do the heavy lifting on the same machine, press R to record the stream to the output files.
    • If you want to do the processing on another machine (maybe with more disk space or CPU power), follow these steps:
      1. In Playitsam, press L to get the cut list, for example:
        Here is the list of cuts:
          cut from chunk 0 to 496
          cut from chunk 28273 to infinity
      2. On the other machine, use the nvcut tool (also part of Playitsam) to download the actual stream. The cut list becomes the final arguments. Eg:
        nvcut tivo:18022,18024,18025 program.mpa program.mpv 0 496 28273
  4. Transcode the audio stream to AC3 with ffmpeg:
    ffmpeg -i program.mpa -ar 48000 -ab 192 -acodec ac3 program.ac3
  5. Multiplex the audio and video streams, using the mplex program from the mjpegtools package.
    mplex -f 8 -o program.mpg program.mpv program.ac3

    NB: on one recording, I found that the audio was out of sync about half a second ahead of the video. I don’t know what caused it, but if you have the same problem, you can use the -O option to mplex to correct it. You will need to determine the correct offset experimentally, in my case I used -O-500ms.

  6. Create a DVD filesystem and menus with the dvdwizard program. At this point you can also put other programs on the DVD or make nicer menus. See the dvdwizard documentation for details. An example command-line is:
    dvdwizard -T 'disc title' -N PAL -A en -t 'program title' -c 900 program.mpg
  7. If the program is in anamorphic widescreen (for example, if it was recorded from a set-top box in 16:9 mode), the generated DVD will have the wrong aspect ratio of 4:3. If this is the case, you need to edit the generated dvdwizard.xml file, changing aspect=4:3 to aspect=16:9 where appropriate, then reauthor the DVD:
    rm -r dvd  dvdauthor -x dvdwizard.xml
  8. Test the result in your favourite software DVD player, eg. Xine or mplayer:
    xine dvd:$PWD/dvd
    mplayer -dvd-device $PWD/dvd dvd://
  9. If you’re happy, burn it to a DVD:
    growisofs -dvd-compat -Z /dev/dvdrw -dvd-video dvd/