Re: [luv-talk] rain radar w/o 2GB of GUI browser [was: Interesting times, where is the beareu of meterology gone]
8 Sep
2020
8 Sep
'20
2:35 a.m.
On Mon, 17 Aug 2020, Tim Connors wrote:
On Mon, 17 Aug 2020, Tim Connors wrote:
And finally, the BoM has just transitioned to a 5 minute scan strategy, so
this seems to be the current schedule:
if (then.minute+1) % 5 == 0], # was 6, should now be 5
--
Tim Connors
On Fri, 14 Aug 2020, Trent W. Buck wrote:
That was easy. Turns out I do still learn stuff:
- *tempdir_path.glob('*.T.*.4.png'),
+ *sorted(tempdir_path.glob('*.T.*.4.png')),
Thanks for reminding me to finish my
proof-of-concept script to undo
some of BOM's recent monetization changes; I just did so.
Can you chuck a sort around that glob? I'd do it myself, but it's not
perl and I'm an old fart. 
9 Sep
9 Sep
2:38 p.m.
New subject: rain radar w/o 2GB of GUI browser [was: Interesting times, where is the beareu of meterology gone]
Tim Connors wrote:
On Mon, 17 Aug 2020, Tim Connors wrote:
One of the good folk of Debian Australia (I think it was Paul Wise)
spotted that they now have a rain radar using OSM:
https://weather.bom.gov.au/location/r1r0fsn-melbourne
You need to enable first-party javascript to get ANY of content, not
just the map, but it's a lot closer to the old m.bom.gov.au than
anything else.
Also doesn't work in PrinceXML :-(
$ prince-browser --javascript https://weather.bom.gov.au/
prince: https://weather.bom.gov.au/:1: error: unexpected token
reserved("const")
prince: https://weather.bom.gov.au/static/js/vendors~main.4319003d.chunk.js:1: error:
unexpected token identifier("patches")
prince: https://weather.bom.gov.au/static/js/main.9384b51b.chunk.js:1: error:
unexpected token identifier("fetchParameters")
Honestly it hasn't been a priority much lately, what with not leaving the house :-P
On Mon, 17 Aug 2020, Tim Connors wrote:
And finally, the BoM has just transitioned to a 5 minute scan strategy, so
this seems to be the current schedule:
if (then.minute+1) % 5 == 0], # was 6, should now be 5 On Fri, 14 Aug 2020, Trent W. Buck wrote:
That was easy. Turns out I do still learn stuff:
- *tempdir_path.glob('*.T.*.4.png'),
+ *sorted(tempdir_path.glob('*.T.*.4.png')),
Thanks for reminding me to finish my
proof-of-concept script to undo
some of BOM's recent monetization changes; I just did so.
Can you chuck a sort around that glob? I'd do it myself, but it's not
perl and I'm an old fart.
12 Sep
12 Sep
7:14 a.m.
New subject: rain radar w/o 2GB of GUI browser [was: Interesting times, where is the beareu of meterology gone]
On Thu, 10 Sep 2020, Trent W. Buck wrote:
One of the good folk of Debian Australia (I think it
was Paul Wise)
spotted that they now have a rain radar using OSM:
https://weather.bom.gov.au/location/r1r0fsn-melbourne
You need to enable first-party javascript to get ANY of content, not
just the map, but it's a lot closer to the old m.bom.gov.au than
anything else.
Also doesn't work in PrinceXML :-(
$ prince-browser --javascript https://weather.bom.gov.au/
prince: https://weather.bom.gov.au/:1: error: unexpected token
reserved("const")
prince: https://weather.bom.gov.au/static/js/vendors~main.4319003d.chunk.js:1: error:
unexpected token identifier("patches")
prince: https://weather.bom.gov.au/static/js/main.9384b51b.chunk.js:1: error:
unexpected token identifier("fetchParameters")
Management are pretty keen to get rid of everything that works on the old
website too.
I was slightly annoyed how it took 1 second to download all the gifs on my
50mbit/s connection, so I discovered that debian's `puf` has a largely
wget compatible syntax (and I didn't have to work out how to fork off
dozens of python subprocesses and wait for them all). Now gm is by far
the slowest invocation:
diff -u -b -r1.3 radar
--- radar 12 Sep 2020 06:55:46 -0000 1.3
+++ radar 12 Sep 2020 07:06:14 -0000
@@ -28,15 +28,14 @@
underlays = ('background', 'locations', 'range',
'topography')
-with tempfile.TemporaryDirectory() as tempdir_str:
+with tempfile.TemporaryDirectory(prefix="radar.") as tempdir_str:
tempdir_path = pathlib.Path(tempdir_str)
now = datetime.datetime.utcnow() # NOTE: TZ=UTC
# Truncate to start of hour to kludge over with propagation delays on the bom.gov.au
servers
# This means the data will be up to 1h delayed, so FIXME.
# now = now.replace(minute=0)
subprocess.call([
- 'wget', '-nv', '-P', tempdir_path,
- '--',
+ 'puf', '-P', tempdir_path,
# Try to get a rain image for every minute in the last hour.
# 1 in every 6 will work. For now, just ignore that some fail.
# FIXME: this is VERY VERY VERY slow to download.
Funny thing about today's 95% chance 20-40mm forecast: 95% refers to the
chance of it raining more than 0.2mm (from memory), with 50% (from memory)
chance of it falling between 20-40mm. Those of us north of the wet bit
got our 1mm. Those of you in the south appear to have gotten your 20mm.
And the Bureau were technically correct - the best form of correct!
--
Tim Connors
11:58 a.m.
New subject: rain radar w/o 2GB of GUI browser [was: Interesting times, where is the beareu of meterology gone]
Tim Connors wrote:
On Thu, 10 Sep 2020, Trent W. Buck wrote:
Because "monetize or GTFO"?
One of the good folk of Debian Australia (I think
it was Paul Wise)
spotted that they now have a rain radar using OSM:
https://weather.bom.gov.au/location/r1r0fsn-melbourne
You need to enable first-party javascript to get ANY of content, not
just the map, but it's a lot closer to the old m.bom.gov.au than
anything else.
Also doesn't work in PrinceXML :-(
$ prince-browser --javascript https://weather.bom.gov.au/
prince: https://weather.bom.gov.au/:1: error: unexpected token
reserved("const")
prince: https://weather.bom.gov.au/static/js/vendors~main.4319003d.chunk.js:1: error:
unexpected token identifier("patches")
prince: https://weather.bom.gov.au/static/js/main.9384b51b.chunk.js:1: error:
unexpected token identifier("fetchParameters")
Management are pretty keen to get rid of everything that works on the old
website too. I was slightly annoyed how it took 1 second to
download all the gifs on my
50mbit/s connection, so I discovered that debian's `puf` has a largely
wget compatible syntax (and I didn't have to work out how to fork off
dozens of python subprocesses and wait for them all). Now gm is by far
the slowest invocation:
You could just do C in python:
for i in range(60):
if 0 == os.fork():
os.execvp('wget', f'...{i}.webp')
os.wait()
This makes the interpreter slightly pissy, but for something this simple, will work.
The next layer up is to do basically the same thing with green threads (which makes all
the non-fluids bindings cry).
The full-blown Stockholm Syndrome version is the twisted framework, with all the class
shit.
Or if you're on 3.something, something something asyncio.
subprocess.call([
- 'wget', '-nv', '-P', tempdir_path,
- '--',
+ 'puf', '-P', tempdir_path,
# Try to get a rain image for every minute in the last hour.
# 1 in every 6 will work. For now, just ignore that some fail.
# FIXME: this is VERY VERY VERY slow to download.
I did consider wget2 --http2-request-window:
Set maximum number of parallel streams per HTTP/2 connection (default: 30).
except of course bom's main site is not even TLS, let alone SPDY.
Mandatory javascript w/o even TLS was what finally made me annoyed
enough to write the first version of that script.
Remember kids: "javascript" and "postscript" are just fancy brand
names for "arbitrary code execution"!
16 Oct
16 Oct
6:45 a.m.
New subject: rain radar w/o 2GB of GUI browser [was: Interesting times, where is the beareu of meterology gone]
On Sat, 12 Sep 2020, Trent W. Buck wrote:
Tim Connors wrote:
And finally, lets do this open source thing.
Uploaded to here so I don't have to keep mailing updates around to people
who don't care about it:
https://github.com/spacelama/bomradar
Who cares about speed of executing when it's just always running out of
cron and displaying on your permanently-on Pi? Still, good to use '-nc'
to not kill the bureau's bandwidth every minute.
It satisfies to see that mass of blue run down the screen right now since
we missed out on yesterday's red.
--
Tim Connors
On Thu, 10 Sep 2020, Trent W. Buck wrote:
subprocess.call([
- 'wget', '-nv', '-P', tempdir_path,
- '--',
+ 'puf', '-P', tempdir_path,
# Try to get a rain image for every minute in the last hour.
# 1 in every 6 will work. For now, just ignore that some fail.
# FIXME: this is VERY VERY VERY slow to download.
I did consider wget2 --http2-request-window:
888
days inactive
926
days old
4 comments
2 participants
participants (2)
-
Tim Connors -
Trent W. Buck