[Colin Ure]

Hi

I'm currently in the process of moving my Excel based Weather Records which stores my AWS data to an Access database solution.
As well as storing the data from my own personal station I was looking to import data from local weather stations for comparison.

Looking to extract daily weather information which has the following data:
Temp (Max, Min, Avg); Humidity; Windspeed (Avg; Gust); Pressure; Rainfall; Sunshine; Weather Conditions

I am aware of Ogimet (daily summaries page) which holds this sort of data but if anyone here knows of similar sites I would be interested to compare before making a decision on where to get data from.
Always a good idea to look at all options at this early planning stage.

Ideally this site would be able to supply the data in CSV or Text format so that I can automatically load the data into my database - I don't think Ogimet has a feature for this.
Any suggestions greatly appreciated - don't necessarily mind if the site supplies the data in METAR format - although it would mean I'd need to write a fair bit of additional code.
Also, does METAR supply daily Max / Min / Avg weather information in addition to the timed report data?

In relation to Ogimet - Am I correct in saying that the Sun figures are for the previous day (daily summaries, column Sun (D-1))
Also, why are the weather summary icons (if you hover your mouse over the icon) out by one day from the date column? - are the summary icons also for the day before?

Look forward to hearing your suggestions - Thanks in advance for your help

Cheers
Colin

[Dave K]

 Colin Ure, on 25 February 2012 - 15:10, said:

Hi

I'm currently in the process of moving my Excel based Weather Records which stores my AWS data to an Access database solution.
As well as storing the data from my own personal station I was looking to import data from local weather stations for comparison.

Looking to extract daily weather information which has the following data:
Temp (Max, Min, Avg); Humidity; Windspeed (Avg; Gust); Pressure; Rainfall; Sunshine; Weather Conditions

I am aware of Ogimet (daily summaries page) which holds this sort of data but if anyone here knows of similar sites I would be interested to compare before making a decision on where to get data from.
Always a good idea to look at all options at this early planning stage.

Ideally this site would be able to supply the data in CSV or Text format so that I can automatically load the data into my database - I don't think Ogimet has a feature for this.
Any suggestions greatly appreciated - don't necessarily mind if the site supplies the data in METAR format - although it would mean I'd need to write a fair bit of additional code.
Also, does METAR supply daily Max / Min / Avg weather information in addition to the timed report data?

In relation to Ogimet - Am I correct in saying that the Sun figures are for the previous day (daily summaries, column Sun (D-1))
Also, why are the weather summary icons (if you hover your mouse over the icon) out by one day from the date column? - are the summary icons also for the day before?

Look forward to hearing your suggestions - Thanks in advance for your help

Cheers
Colin

To answer that question, yes it seems that a total of the sunshine hours is only given on the 06Z synop return, and for the previous day 00z to 00z. Each hourly synop report has a section for sunshine minutes in the last hour but these aren't decoded in Ogimet.

If you look at Kinloss for example, the 06z report today gives 7.1 hours and if you look at this page with the breakdown of hourly totals for yesterday

http://www.weatherca...obsid/3066.html

it adds up to 420 minutes or 7.0 hours, same goes for the 23rd with 1.8 on Ogimet against 1.7 on Weathercast ( not sure where the 0.1 extra comes from though!).

[Colin Ure]

Thanks Dave, I thought it looked like it was for the previous day.

The site you linked to (weathercast) also looks interesting as a source of station data - still looking to see if I can extract in some kind of automated way as I'm looking at about 8 - 10 local stations to pull daily information from.
Even contemplating processing the html webcode as a method of extracting the data!

[Dave K]

Having said that, I'm not too sure about what happens to the sunshine total on Ogimet once the sun is up before 06Z - if the minutes count to the previous day or the current one!

This post has been edited by Big Dave's Gusset: 25 February 2012 - 18:47

[Colin Ure]

i've been looking around Ogimet and there is a function called 'getsynop' which can return the unencoded synop reports in CSV format.

Great for extracting data automatically the down side is I would have to write code to decoded the reports - I nice challenge (perhaps!)
I noticed that the 06Z report (as I think you mentioned Dave) contains additional information like minimum temperature for the past 24 hours etc
But, if this comes at 06Z for the previous 24hrs who can you find out the minimum temperature for a particular day 00Z to 23Z ?

I'm going to investigate how you decode synop reports to see if this is a challenge I want to take on but if there are any synop experts out there that can help answer both my initial question (and Dave's question above) that would help me get my head around it a bit more.

Cheers
Colin

[Dave K]

 Colin Ure, on 26 February 2012 - 10:58, said:

i've been looking around Ogimet and there is a function called 'getsynop' which can return the unencoded synop reports in CSV format.

Great for extracting data automatically the down side is I would have to write code to decoded the reports - I nice challenge (perhaps!)
I noticed that the 06Z report (as I think you mentioned Dave) contains additional information like minimum temperature for the past 24 hours etc
But, if this comes at 06Z for the previous 24hrs who can you find out the minimum temperature for a particular day 00Z to 23Z ?

I'm going to investigate how you decode synop reports to see if this is a challenge I want to take on but if there are any synop experts out there that can help answer both my initial question (and Dave's question above) that would help me get my head around it a bit more.

Cheers
Colin

If you Google around you might find that someone has already provided something in the public domain using java or XML to parse synop codes such as this for example (depends how you want it output of course)

http://sourceforge.n...ects/metaf2xml/

[Colin Ure]

 Big Dave, on 26 February 2012 - 11:05, said:

If you Google around you might find that someone has already provided something in the public domain using java or XML to parse synop codes such as this for example (depends how you want it output of course)

http://sourceforge.n...ects/metaf2xml/

Excellent, thanks Dave - I've downloaded this - I just need to figure out how to run it ! Which in the five minutes I've looked at it is not completely obvious - missing something simple I bet.
The plan is ultimately to import the data into an Access database so if I can extract the information - decode it and then load it into Access that would work ok.

[Dave K]

I found this on weatherfaqs

Quote

Sunshine groups carried in SYNOP reports from automatic stations

Where Weather Stations are suitably equipped, and are reporting in the SYNOP code for inclusion in international bulletins, then "sunshine" duration (mainly using radiation sensors) is carried in the following groups:

At 0600Z ONLY .... In Section 333: the group 55SSS shows the duration of sunshine, in tenths of an hour, for the 24 hours of the previous day.
At ALL hours .... In Section 333, the group 553SS is reported, showing the sunshine total for that hour in tenths of an hour.

[Colin Ure]

 Colin Ure, on 26 February 2012 - 11:53, said:

Excellent, thanks Dave - I've downloaded this - I just need to figure out how to run it ! Which in the five minutes I've looked at it is not completely obvious - missing something simple I bet.
The plan is ultimately to import the data into an Access database so if I can extract the information - decode it and then load it into Access that would work ok.

Did'nt manage to get this too work - think it may need Perl installed to run.
Not sure it would do exactly what I want anyway, rather than passing an entire file to process (say a months worth) it looks like it only processes a single record at a time.
Although can't be 100% sure on that without running it.

I'm actually thinking of writng code to decode the Synop Reports myself - manage to find some good reference manuals and I'm currently looking through them.
It confirms exactly what you found regarding the sunshine total - still not worked out how I can tell the max and min temperature for a given day 00Z to 23Z - might not even be possible.
The 333 group of code does contain a max and min set of values but they tend to come in the 06Z report and relate to the previous 12 hours - so if the max temp was in the early hours of the morning this would be the value reported not the max from the day before.

Some more investigation required me thinks.

This post has been edited by Colin Ure: 27 February 2012 - 19:57

[Colin Ure]

I've spent the last few days investigating the WMO FM-12 Synopic reporting format and I believe I should be able to write something to extract the data (at least initially the elements that are important to me) and load them into my planned MS Access database.

To assist with my understanding I've been using Ogimet's decoding of the reports as a guide - it's from here that I plan to extract the raw data from, although a question on that in a second.

I noticed that Ogimet as well as reporting the mean windspeed also reports on the max. wind gust - this does not appear to be contained within the FM-12 format and I cannot view a corresponding record in the data which matches what is being output.
My thought was that this could be extracted from the METAR reports but there is a problem with that! First of all I'm looking at Edinburgh Gogarbank for my Synopic reports (03166) - this station does not appear to create METAR reports (is this only for Airports?)
Edinburgh Airport (only a mile or so from Gogarbank) does produce METAR reports but the wind gusts do not match exactly the wind gusts reported on Ogimet (out by a knot or two) - also they appear to be reported on different days (as close as I can match) on Ogimet.

Does anyone know where the Max. Gusts on Ogimet are sourced from? Is there anywhere I can automatically extract this information from on the web?

The additional question I eluded to above is where else can you download historic WMO FM-12 Synopic reports? Had a hunt around the WMO and NOAA sites at lunchtime but could not find an obvious place to download this information from.
Ideally I would be looking for a text or csv file which would contain upto a months worth of data - with the current month being updated throughout the month.

Also, if anyone is interested I'll be happy to share the code (when I get it written that is) which should be easily convertable to Excel (VBA) and operate on a similar process. Let me know and I'll let you know when I have it up and working.

Cheers
Colin

[Dave K]

WRT wind gusts, see http://weatherfaqs.org.uk/node/108

Quote

... when we move away from the main section of the SYNOP, there is some variability allowed for in the 'additional' data section as under:

Code 3778 ('9-group' sections)
910ff Highest gust during the 10-minute period immediately preceding the observation.
911ff Highest gust .... (see note below*)
912ff Highest mean wind speed .... (see note below*)
913ff Mean wind speed .... (see note below*)
914ff Lowest mean wind speed .... (see note below*)

*Note: " ... during the period covered by W1W2 (i.e. past weather) in group 7wwW₁W₂, unless a different period of reference is indicated by group 907tt; or during the 10-minute period immediately preceding the time of observation indicated by group 904tt "
where tt =
00 at observation time
01 .. 09 units of 6 minutes before observation (i.e. 02=12 minutes before etc.)
10 1 hr before observation (the most often used in this context)
11 .. 60 continuing the scheme of units of 6 minutes before observation. (i.e. 39=3hr 54mins)
61 6 to 7 hrs
62 7 to 8 hrs etc. ... up to
66 11 to 12 hrs
67 12 to 18hr
68 more than 18hr
69 time unknown
70 began during observation
71 ended during observation

So for an example from Altnaharra

201202281950 AAXX 28204 03044 45584 /2119 10105 20067 30050 40148 57006 91950 333 553// 20000 84/24 90710 91132 91032=


That is a peak of 32 knots since the last hourly observation, I think!
I guess that there is some coding in the Ogimet software that takes the highest reading from all the synops in the period?

Have a play at http://www.mundomanz...&action=display

which also gives a detailed decoding

This post has been edited by Big Dave's Gusset: 01 March 2012 - 20:41

[Colin Ure]

 Big Dave, on 01 March 2012 - 20:28, said:

WRT wind gusts, see http://weatherfaqs.org.uk/node/108



So for an example from Altnaharra

201202281950 AAXX 28204 03044 45584 /2119 10105 20067 30050 40148 57006 91950 333 553// 20000 84/24 90710 91132 91032=


That is a peak of 32 knots since the last hourly observation, I think!
I guess that there is some coding in the Ogimet software that takes the highest reading from all the synops in the period?

Have a play at http://www.mundomanz...&action=display

which also gives a detailed decoding

Ah, that's where it is - I've been looking in the WMO No 306 Codes Manual and it did not state this in the 333 section - 9SpSpspsp detailed in that section had a note of Supplementary Information (Table 3778).
This I thought was another table like most of the code tables in the document - but it's actually a table at the end of the document listing a whole host of additional weather parameters including Wind Gusts.

A whole new area to analyse - Thanks


I'll need to start looking at both the link you provided above and the weatherfaq's site - thanks for all your help on this Dave, much appreciated.