Scripts to fetch weather data from the Australian BoM

Contents

  1. Scripts to fetch weather data from the Australian BoM

    1. Current observations
    2. Forecast
    3. Slimserver Weather Display

The Australian Bureau of Meteorology provides a lot of free useful weather data, as well as regular forecasts, but it is in HTML or plain text format. These scripts try to parse relevant parts of that data out for use elsewhere, with varying levels of success.

Current observations

This script and Python module fetches current weather observations by parsing specially-encoded comments in the automatically-updated report; it has been very reliable. To get the data for a station other than Sydney, just change the STATION_NAME variable (and possibly also REPORT_URL).

   1 #!/usr/bin/env python
   2 
   3 # $Id: bom_weather.py,v 1.7 2004/10/26 06:45:04 andrewb Exp $
   4 # script to pull out raw data from Bureau of Meteorology weather observations
   5 # andrewb@cse.unsw.edu.au, 2004/10
   6 
   7 import sys, os, urllib, time, getopt
   8 from HTMLParser import HTMLParser
   9 
  10 REPORT_URL=ftp://ftp2.bom.gov.au/anon/gen/fwo/IDN65066.html
  11 STATION_NAME=Sydney - Observatory Hill
  12 MAX_REPORT_AGE=2*60*60 # 2 hours
  13 
  14 
  15 class BomCommentParser(HTMLParser):
  16     def __init__(self):
  17         HTMLParser.__init__(self)
  18         self.head = None
  19         self.unit = None
  20         self.data = []
  21 
  22     def handle_comment(self, data):
  23         fields = data.strip().split(',')
  24         fields = map(lambda x: x.strip(), fields)
  25 
  26         entrytype = fields[0]
  27         if entrytype not in [HEAD, UNIT, DATA]:
  28             return # some other comment, we ignore it
  29         fields = fields[1:]
  30 
  31         if self.head is None:
  32             assert(entrytype == HEAD)
  33             self.head = map(lambda x: x.lower(), fields)
  34         elif self.unit is None:
  35             assert(entrytype == UNIT)
  36             assert(len(fields) == len(self.head))
  37             self.unit = fields
  38         else: # DATA
  39             assert(len(fields) == len(self.head))
  40             dict = {}
  41             for i in range(len(fields)):
  42                 dict[self.head[i]] = fields[i]
  43             self.data.append(dict)
  44 
  45 
  46 class ScreenLog:
  47     def __init__(self, fh):
  48         self.fh = fh
  49     def log(self, msg):
  50         self.fh.write(Error:  + msg + \n)
  51 
  52 
  53 class FileLog:
  54     def __init__(self, filename):
  55         self.fh = open(filename, 'a')
  56     def log(self, msg):
  57         self.fh.write(time.strftime(%b %d %H:%M:%S ) + msg + \n)
  58 
  59 
  60 def main(argv):
  61     logfilename = None
  62     outfilename = None
  63     try:
  64         opts, args = getopt.getopt(argv[1:], l:)
  65         for (opt, arg) in opts:
  66             if opt == -l:
  67                 logfilename = arg
  68         assert(len(args) = 1)
  69         if len(args) == 1:
  70             outfilename = args[0]
  71     except:
  72         sys.stderr.write(Usage: %s [-l LOGFILE] [FILE]\n % argv[0])
  73         return 1
  74 
  75     try:
  76         if (logfilename is not None):
  77             log = FileLog(logfilename)
  78         else:
  79             log = ScreenLog(sys.stderr)
  80     except IOError:
  81         sys.stderr.write(Error: couldn't append to %s\n % logfilename)
  82         return 2
  83 
  84     try:
  85         parser = BomCommentParser()
  86         fh = urllib.urlopen(REPORT_URL)
  87         parser.feed(fh.read())
  88         fh.close()
  89         parser.close()
  90     except IOError:
  91         log.log(failed to fetch report from %s % REPORT_URL)
  92         return 2
  93     except:
  94         log.log(failed to parse report data)
  95         return 3
  96 
  97     try:
  98         data = None
  99         for entry in parser.data:
 100             if entry[name] == STATION_NAME:
 101                 data = entry
 102                 break
 103         assert(data)
 104     except:
 105         log.log(failed to find data for %s % STATION_NAME)
 106         return 3
 107 
 108     try:
 109         temperature = round(float(data[temperature]))
 110         humidity = round(float(data[relative humidity]))
 111         pressure = round(float(data[pressure]))
 112     except:
 113         log.log(failed to unpack report data)
 114         return 3
 115 
 116     try:
 117         timestamp = time.mktime(time.strptime(data[time], %Y%m%d:%H%M%S))
 118         now = time.mktime(time.localtime())
 119         if (now  timestamp or now - timestamp  MAX_REPORT_AGE):
 120             log.log(timestamp %s out of range, data[time])
 121             return 4
 122     except:
 123         log.log(failed to parse timestamp)
 124         return 3
 125 
 126     try:
 127         if (outfilename is not None):
 128             fh = open(outfilename, 'w')
 129         else:
 130             fh = sys.stdout
 131         fh.write(%d\xb0C %d%% %dhPa\n % (temperature, humidity, pressure))
 132         if (outfilename is not None):
 133             fh.close()
 134     except IOError:
 135         log.log(couldn't write to %s % outfilename)
 136         return 2
 137 
 138     return 0
 139 
 140 
 141 if __name__ == __main__:
 142     sys.exit(main(sys.argv))

bom_weather.py

Forecast

This script and Python module attempts to grab the current long or short forecast as well as the minimum and maximum temperatures for Sydney from the published text file. Unfortunately the format of the file seems to be a little variable, so I have had to change the script a few times to handle different formats and try to make it more robust. It could probably be hacked to handle other state forecasts, or return different min/max temperatures.

document.write(‘Toggle line numbers<\/a>‘);

   1 #!/usr/bin/env python
   2 
   3 # $Id: bom_forecast.py,v 1.9 2006-12-05 22:11:41 andrewb Exp $
   4 # script to grab bureau of meteorology forecast for sydney
   5 # andrewb@cse.unsw.edu.au, 2004/10
   6 
   7 import sys, os, urllib, time, getopt
   8 from bom_weather import FileLog, ScreenLog
   9 
  10 FORECAST_URL=ftp://ftp2.bom.gov.au/anon/gen/fwo/IDN10064.txt
  11 FORECAST_LOCN=City
  12 
  13 
  14 def long_forecast(lines, minmax):
  15     day = time.strftime(%A).lower()
  16     para = None
  17     for line in lines:
  18         if para == None:
  19             if line.strip().lower().startswith(forecast for  + day):
  20                 para = []
  21         else:
  22             if line ==  or line.startswith(Precis:):
  23                 break
  24             else:
  25                 para.append(line)
  26 
  27     forecast =  .join(para)
  28     try:
  29         if minmax:
  30             forecast +=  Max %d % minmax[1]
  31     except ValueError:
  32         pass # no max temperature
  33     return forecast
  34 
  35 def short_forecast(lines):
  36     for line in lines:
  37         if line.startswith(Precis:):
  38             return  .join(line.split(':',1)[1].split()).strip(.)
  39     raise RuntimeError(Couldn't parse short forecast)
  40 
  41 def get_minmax(lines):
  42     found = False
  43     seen_forecast = False
  44     min = max = None
  45     prevword = None
  46     for line in lines:
  47         if line.startswith(Forecast for):
  48             # if we see this twice, it means that the min/max temps are
  49             # for tomorrow's forecast and not today's, so don't return them
  50             if seen_forecast:
  51                 return None
  52             seen_forecast = True
  53         words = line.split()
  54         for word in words:
  55             if found:
  56                 try:
  57                     n = int(word)
  58                     if prevword.lower().startswith('min'):
  59                         min = n
  60                     elif prevword.lower().startswith('max'):
  61                         max = n
  62                 except ValueError:
  63                     pass
  64                 if max:
  65                     return min, max
  66                 prevword = word
  67             elif word == FORECAST_LOCN + ::
  68                 found = True
  69                 prevword = None
  70     return None, None
  71 
  72 def fetch_forecast(do_long_forecast):
  73     fh = urllib.urlopen(FORECAST_URL)
  74     lines = map(str.strip, fh.readlines())
  75     minmax = get_minmax(lines)
  76     if (do_long_forecast):
  77         forecast = long_forecast(lines, minmax)
  78     else:
  79         forecast = short_forecast(lines)
  80         if minmax:
  81             forecast = forecast + ( %d % minmax[1])
  82     fh.close()
  83     return forecast
  84 
  85 def main(argv):
  86     logfilename = None
  87     outfilename = None
  88     do_long_forecast = False
  89     try:
  90         opts, args = getopt.getopt(argv[1:], fl:)
  91         for (opt, arg) in opts:
  92             if opt == -l:
  93                 logfilename = arg
  94             elif opt == -f:
  95                 do_long_forecast = True
  96         assert(len(args) = 1)
  97         if len(args) == 1:
  98             outfilename = args[0]
  99     except:
 100         sys.stderr.write(Usage: %s [-l LOGFILE] [-f] [FILE]\n % argv[0])
 101         return 1
 102 
 103     try:
 104         if (logfilename is not None):
 105             log = FileLog(logfilename)
 106         else:
 107             log = ScreenLog(sys.stderr)
 108     except IOError:
 109         sys.stderr.write(Error: couldn't append to %s\n % logfilename)
 110         return 2
 111 
 112     try:
 113         forecast = fetch_forecast(do_long_forecast)
 114     except IOError:
 115         log.log(failed to fetch forecast from %s % FORECAST_URL)
 116         return 2
 117     except:
 118         log.log(failed to parse forecast data)
 119         return 3
 120 
 121     # fridge hack for very short forecasts
 122     if (len(forecast) = 6):
 123         forecast = Forecast:  + forecast
 124 
 125     try:
 126         if (outfilename is not None):
 127             fh = open(outfilename, 'w')
 128         else:
 129             fh = sys.stdout
 130         fh.write(forecast + \n)
 131         if (outfilename is not None):
 132             fh.close()
 133     except IOError:
 134         log.log(couldn't write to %s % outfilename)
 135         return 2
 136 
 137     return 0
 138 
 139 
 140 if __name__ == __main__:
 141     sys.exit(main(sys.argv))

bom_forecast.py

Slimserver Weather Display

This is unlikely to be useful for many people, but it shows how to use the bom_forecast module in another script. It connects to a running SlimServer (on which the CLI must be enabled) and displays the latest weather forecast for a specified duration. I run it from a cron job to have the weather forecast displayed on my Squeezebox when I wake up.

document.write(‘Toggle line numbers<\/a>‘);

   1 #!/usr/bin/python
   2 
   3 import sys, socket, urllib
   4 from optparse import OptionParser
   5 import bom_forecast
   6 
   7 class SlimCLI:
   8     def __init__(self, host, port=9090):
   9         self.sock = socket.socket()
  10         self.sock.connect((host, port))
  11 
  12     def cmd(self, command, args = [], player = None):
  13         if player != None:
  14             line = player +  
  15         else:
  16             line = 
  17         line += command
  18         for arg in args:
  19             line +=   + urllib.quote(str(arg))
  20         self.sock.sendall(line + \n)
  21         buf = self.sock.recv(4096)
  22         line = buf.splitlines()[0]
  23         return map(urllib.unquote, line.split())
  24 
  25     def close(self):
  26         self.sock.sendall(exit\n)
  27         self.sock.close()
  28 
  29 def parse_args():
  30     p = OptionParser()
  31     p.add_option('-m', type='int', dest='mins', default=30,
  32                  help='time (in minutes) to display weather, default: 30')
  33     return p.parse_args()
  34 
  35 def main():
  36     opts, args = parse_args()
  37     forecast = bom_forecast.fetch_forecast(True)
  38     cli = SlimCLI(localhost)
  39     cli.cmd(display, [Today's weather forecast, forecast, opts.mins * 60])
  40     cli.close()
  41 
  42 if __name__ == __main__:
  43     main()

slimweather.py

Published by

ab

Well, AB's just this guy, you know?