Re: [luv-talk] rain radar w/o 2GB of GUI browser [was: Interesting times, where is the beareu of meterology gone]
On Mon, 17 Aug 2020, Tim Connors wrote:
On Mon, 17 Aug 2020, Tim Connors wrote:
On Fri, 14 Aug 2020, Trent W. Buck wrote:
Thanks for reminding me to finish my proof-of-concept script to undo some of BOM's recent monetization changes; I just did so.
Can you chuck a sort around that glob? I'd do it myself, but it's not perl and I'm an old fart.
That was easy. Turns out I do still learn stuff:
- *tempdir_path.glob('*.T.*.4.png'), + *sorted(tempdir_path.glob('*.T.*.4.png')),
And finally, the BoM has just transitioned to a 5 minute scan strategy, so this seems to be the current schedule: if (then.minute+1) % 5 == 0], # was 6, should now be 5 -- Tim Connors
Tim Connors wrote:
On Mon, 17 Aug 2020, Tim Connors wrote:
On Mon, 17 Aug 2020, Tim Connors wrote:
On Fri, 14 Aug 2020, Trent W. Buck wrote:
Thanks for reminding me to finish my proof-of-concept script to undo some of BOM's recent monetization changes; I just did so.
Can you chuck a sort around that glob? I'd do it myself, but it's not perl and I'm an old fart.
That was easy. Turns out I do still learn stuff:
- *tempdir_path.glob('*.T.*.4.png'), + *sorted(tempdir_path.glob('*.T.*.4.png')),
And finally, the BoM has just transitioned to a 5 minute scan strategy, so this seems to be the current schedule:
if (then.minute+1) % 5 == 0], # was 6, should now be 5
One of the good folk of Debian Australia (I think it was Paul Wise) spotted that they now have a rain radar using OSM: https://weather.bom.gov.au/location/r1r0fsn-melbourne You need to enable first-party javascript to get ANY of content, not just the map, but it's a lot closer to the old m.bom.gov.au than anything else. Also doesn't work in PrinceXML :-( $ prince-browser --javascript https://weather.bom.gov.au/ prince: https://weather.bom.gov.au/:1: error: unexpected token reserved("const") prince: https://weather.bom.gov.au/static/js/vendors~main.4319003d.chunk.js:1: error: unexpected token identifier("patches") prince: https://weather.bom.gov.au/static/js/main.9384b51b.chunk.js:1: error: unexpected token identifier("fetchParameters") Honestly it hasn't been a priority much lately, what with not leaving the house :-P
On Thu, 10 Sep 2020, Trent W. Buck wrote:
One of the good folk of Debian Australia (I think it was Paul Wise) spotted that they now have a rain radar using OSM:
https://weather.bom.gov.au/location/r1r0fsn-melbourne
You need to enable first-party javascript to get ANY of content, not just the map, but it's a lot closer to the old m.bom.gov.au than anything else.
Also doesn't work in PrinceXML :-(
$ prince-browser --javascript https://weather.bom.gov.au/ prince: https://weather.bom.gov.au/:1: error: unexpected token reserved("const") prince: https://weather.bom.gov.au/static/js/vendors~main.4319003d.chunk.js:1: error: unexpected token identifier("patches") prince: https://weather.bom.gov.au/static/js/main.9384b51b.chunk.js:1: error: unexpected token identifier("fetchParameters")
Management are pretty keen to get rid of everything that works on the old website too. I was slightly annoyed how it took 1 second to download all the gifs on my 50mbit/s connection, so I discovered that debian's `puf` has a largely wget compatible syntax (and I didn't have to work out how to fork off dozens of python subprocesses and wait for them all). Now gm is by far the slowest invocation: diff -u -b -r1.3 radar --- radar 12 Sep 2020 06:55:46 -0000 1.3 +++ radar 12 Sep 2020 07:06:14 -0000 @@ -28,15 +28,14 @@ underlays = ('background', 'locations', 'range', 'topography') -with tempfile.TemporaryDirectory() as tempdir_str: +with tempfile.TemporaryDirectory(prefix="radar.") as tempdir_str: tempdir_path = pathlib.Path(tempdir_str) now = datetime.datetime.utcnow() # NOTE: TZ=UTC # Truncate to start of hour to kludge over with propagation delays on the bom.gov.au servers # This means the data will be up to 1h delayed, so FIXME. # now = now.replace(minute=0) subprocess.call([ - 'wget', '-nv', '-P', tempdir_path, - '--', + 'puf', '-P', tempdir_path, # Try to get a rain image for every minute in the last hour. # 1 in every 6 will work. For now, just ignore that some fail. # FIXME: this is VERY VERY VERY slow to download. Funny thing about today's 95% chance 20-40mm forecast: 95% refers to the chance of it raining more than 0.2mm (from memory), with 50% (from memory) chance of it falling between 20-40mm. Those of us north of the wet bit got our 1mm. Those of you in the south appear to have gotten your 20mm. And the Bureau were technically correct - the best form of correct! -- Tim Connors
Tim Connors wrote:
On Thu, 10 Sep 2020, Trent W. Buck wrote:
One of the good folk of Debian Australia (I think it was Paul Wise) spotted that they now have a rain radar using OSM:
https://weather.bom.gov.au/location/r1r0fsn-melbourne
You need to enable first-party javascript to get ANY of content, not just the map, but it's a lot closer to the old m.bom.gov.au than anything else.
Also doesn't work in PrinceXML :-(
$ prince-browser --javascript https://weather.bom.gov.au/ prince: https://weather.bom.gov.au/:1: error: unexpected token reserved("const") prince: https://weather.bom.gov.au/static/js/vendors~main.4319003d.chunk.js:1: error: unexpected token identifier("patches") prince: https://weather.bom.gov.au/static/js/main.9384b51b.chunk.js:1: error: unexpected token identifier("fetchParameters")
Management are pretty keen to get rid of everything that works on the old website too.
Because "monetize or GTFO"?
I was slightly annoyed how it took 1 second to download all the gifs on my 50mbit/s connection, so I discovered that debian's `puf` has a largely wget compatible syntax (and I didn't have to work out how to fork off dozens of python subprocesses and wait for them all). Now gm is by far the slowest invocation:
You could just do C in python: for i in range(60): if 0 == os.fork(): os.execvp('wget', f'...{i}.webp') os.wait() This makes the interpreter slightly pissy, but for something this simple, will work. The next layer up is to do basically the same thing with green threads (which makes all the non-fluids bindings cry). The full-blown Stockholm Syndrome version is the twisted framework, with all the class shit. Or if you're on 3.something, something something asyncio.
subprocess.call([ - 'wget', '-nv', '-P', tempdir_path, - '--', + 'puf', '-P', tempdir_path, # Try to get a rain image for every minute in the last hour. # 1 in every 6 will work. For now, just ignore that some fail. # FIXME: this is VERY VERY VERY slow to download.
I did consider wget2 --http2-request-window: Set maximum number of parallel streams per HTTP/2 connection (default: 30). except of course bom's main site is not even TLS, let alone SPDY. Mandatory javascript w/o even TLS was what finally made me annoyed enough to write the first version of that script. Remember kids: "javascript" and "postscript" are just fancy brand names for "arbitrary code execution"!
On Sat, 12 Sep 2020, Trent W. Buck wrote:
Tim Connors wrote:
On Thu, 10 Sep 2020, Trent W. Buck wrote:
subprocess.call([ - 'wget', '-nv', '-P', tempdir_path, - '--', + 'puf', '-P', tempdir_path, # Try to get a rain image for every minute in the last hour. # 1 in every 6 will work. For now, just ignore that some fail. # FIXME: this is VERY VERY VERY slow to download.
I did consider wget2 --http2-request-window:
And finally, lets do this open source thing. Uploaded to here so I don't have to keep mailing updates around to people who don't care about it: https://github.com/spacelama/bomradar Who cares about speed of executing when it's just always running out of cron and displaying on your permanently-on Pi? Still, good to use '-nc' to not kill the bureau's bandwidth every minute. It satisfies to see that mass of blue run down the screen right now since we missed out on yesterday's red. -- Tim Connors
participants (2)
-
Tim Connors -
Trent W. Buck