Climate Prediction Center - wgrib2: converting MERRA HDF file to grib2 Search CPC Search About Us Contact Us >>>>wgrib2 - example converting MERRA HDF to grib2 wgrib2: Converting MERRA HDF file to GRIB-2 Introduction MERRA is a NASA reanalysis project that has created reanalyses in HDF format. For some people, grib is a more convenient format. Here is our first attempt to convert a MERRA HDF file into grib-2.

Converting GRIB (1 or 2). Support for GRIB2 files was added in version 4.3.0. A tool called ncl_convert2nc was created to convert all or part of a GRIB file to.

Surveying the situation we see that Lats or the simpler to use Lats4d can convert HDF into grib-1. Grib-1 can be converted into grib-2 by the cnvgrib utility except that lats creates a custom grib table. Therefore a program would have to be written that converts the grib-1 file with the custom grib table into a grib table using the standard NCEP table.

Doable but the program wouldn't be very generic. In addition the process would take 3 steps (convert to grib1, translate grib1, convert to grib2). Can we do better? Looking at the programs that can read HDF, we see a familiar face, GrADS. GrADS can read the HDF and write it out as a binary file.

Wgrib2 can then read the binary file, a grib-2 template and metadata and then create a grib-2 file. 1: The template Grib-2 files include much metadata that is constant such as grid, center that created the data, the process that created the data, etc. Making a template takes some knowledge and we'll assume it has already been done. For the MERRA template, I used copygb/cnvgrib to create a grib2 file with the proper grid definition and then used wgrib2 to fill in some of the metadata (center). Probably should have filled in more of the metadata but I didn't. 2: Writing the binary and meta data The script g2write takes three arguements, the GrADS variable/expression, the output file and some metadata (variable:level:forecast/analysis).

The script then creates the binary and metadata for one or more grids depending whether t is varying. These two files along with the template can be used to make a grib2 file. 3: All together • Start gradshdf. • open the HDF file, sdfopen hdf-file • run the script, merra2grb2.gs 4: What does merra2grb2 do?

• set the grid. Grads often uses an extra column.

Search by city. Press enter or select the go button to submit request About Installation Versions Documentation NDFD / NDGD General Info Degrib: Man Pages Before I start the 'tutorial', here are some links to 'man pages' that I have put together to describe the functionality of degrib and tkdegrib. • • Degrib: Tutorial Table of Contents: • • • What can I do with the data? • • • • • • • • • For more information on 'Degrib and GrADS': • For information on 'FORTRAN or C access to.flt files': • • • • • • • (brief description of directiory structure, and output file types) Now that you've installed the program, you need to download some data. You can do this by manually transfering the data from wherever it is to your machine. If you are working with NDFD or NDGD data,.

You could manually download the data from there, but the degrib program comes with two ways to automatically access those URLS. • On the 'Download' tab of the GUI (Graphical User Interface) program tkdegrib, one can highlight the sector (see for the definition of a sector), or the specific weather element in the sector that one is interested in, and then click either the 'Download by ftp' or the 'Download by http' buttons. At that point tkdegrib will contact the web and store the file in /degrib/data/'data set'/'sector', where 'data set' is either NDFD or NDGD, and 'sector' is the abreviated name of the sector. • From a command line or a script ( Note in MS-Windows replace /degrib/ with c: ndfd degrib20 ): • /degrib/bin/tcldegrib web.tcl ndfd midatlan,all which should get from the 'ndfd' data set all weather elements found in the 'midatlan' sector. • /degrib/bin/tcldegrib web.tcl ndfd conus,maxt,mint which should get from the 'ndfd' data set the maxt and mint weather elements found in the 'conus' sector.

• /degrib/bin/tcldegrib web.tcl ndgd conus,ozone01 which should get from the 'ndgd' data set the ozone01 forecast found in the 'conus' sector. If you can't find an appropriate NDFD sector, the NWS is providing an experimental web service whose intent is to allow the user to get a cutom defined sector in GRIB2. It is currently limited to a maximum of 15 degrees by 15 degrees.

Click for more information. • To use the service from the GUI tkdegrib, go to the 'File' menu, and select 'Configure'. Then click on the 'Custom' tab.

Enter in the coordinates that you are interested in and press 'OK'. Now when you highlight the 'custom' sector on the 'Download tab', it will talk to the web service using your coordinates. • To use the service from the command line, you need to first edit the file '/degrib/bin/ndfd.ini', to use your coordinates.

You are looking for the '[Custom_Sector]' section. After modifying '/degrib/bin/ndfd.ini', you can talk to the web service using: • /degrib/bin/tcldegrib web.tcl ndfd custom,all What can I do with the data?

To get an idea of what the data looks like, on 'MS-Windows', you can use 'tkdegrib' with 'superImageGen' and 'htmlmaker' to create images (.png) and a means of browsing those images with 'MS-Internet Explorer'. • In the GUI tkdegrib: Go to the 'Download' tab, highlight the sector that you want images of, and click 'Generate Images'. That should cause tkdegrib to convert the selected forecasts to 'mosaic (*.tlf)' files (see for a definition).

Then tkdegrib calls superImageGen to read the 'mosaic (*.tlf)' files and draw the images. Next htmlmaker is called to create html pages to help view the images, and finally IE is called to browse the html.

Note 1: Unfortunately, superImageGen is provided as an executable which only works on MS-Windows machines. Teamviewer 9 Serial Number Text. • From the command line: Currently there is no way to do this from the command line. To see what messages are inside a GRIB file, you can use degrib to inventory it. • In the GUI tkdegrib: Click on the 'GIS' tab, and browse for your file. Double click on it in the top half, and it should fill out the 'message #, a short version of the variable name, a long version of the variable name, the level or surface, the reference time, the valid time, and the difference between the valid time and the reference time' in the bottom. • From the command line: /degrib/bin/degrib 'GRIB file' -I where 'GRIB file' is replaced with the name of your GRIB file.

This should print out the message number, which byte it starts on, the GRIB version, the variable name (both short and long forms of it), the level or surface, the reference time, the valid time, and the forecast projection. To get a better feel for the data, you can use degrib to probe the file at a given lat/lon point, or a set of points in a 'point file'. • In the GUI tkdegrib: Currently there is no way to do this from the GUI.

• From the command line: /degrib/bin/degrib 'GRIB file' -P -pnt 38,-76 /degrib/bin/degrib 'GRIB file' -P -pntFile 'foo.txt' where 'GRIB file' is replaced with the name of your GRIB file, and 'foo.txt' is replaced with a file that contains station ID, lat, lon. This should read the file, and extract each message in the file. Then it will either bi-linearly interpolate to the given lat/lon point, or find the nearest neighbor, and output that data to stdout. See the (particularly the 'PROBE OPTIONS' section) for more details. To get 'meta data' or information describing a GRIB message, you can use degrib to create a.txt file.

Amongst the 'meta data' will be the maximum value, the minimum value, the number of missing values detected, etc. • In the GUI tkdegrib: Each time you 'convert' a message from a file, the GUI automatically creates a.txt file that contains the 'meta data'.

To convert a file, click on the 'GIS' tab, and browse for your file. Double click on it in the top half, and it should fill out the inventory part in the bottom half. Select the message that you want in the bottom half, and choose the output name (or press 'Recommend' to have tkdegrib 'recommend' one. Next choose an 'File Type', and press the 'Generate file' button at the bottom. • From the command line: /degrib/bin/degrib 'GRIB file' -C -msg 1 /degrib/bin/degrib 'GRIB file' -C -msg all /degrib/bin/degrib 'GRIB file' -C -Met -msg 2 -out message2.txt where 'GRIB file' is replaced with the name of your GRIB file. This should read the file, and extract the requested messages.

The -Met option (on by default) tells it to create a.txt file with the meta data. To turn off the 'meta' file option, use -nMet. See the (particularly the 'CONVERT OPTIONS' section) for more details. To get some diagnostics about the message before it has been parsed by the code, you can create a '.is0' file.

This can be useful for diagnosing packing and unpacking routines, but you will need the to understand the results. • In the GUI tkdegrib: Currently there is no way to do this from the GUI. • From the command line: /degrib/bin/degrib 'GRIB file' -C -nMet -IS0 -msg 1 /degrib/bin/degrib 'GRIB file' -C -IS0 -msg all where 'GRIB file' is replaced with the name of your GRIB file. You can use degrib to convert a given message to a.csv file, which is a comma delimited ASCII file that contains a x, y, lat, lon, value line for each cell in the grid.

This file can be imported into Excel (limited to the first 65,536 lines), or read by other files to populate a SQL database. • In the GUI tkdegrib: Click on the 'GIS' tab, and browse for your file. Double click on it in the top half, and it should fill out the inventory part in the bottom half. Select the message that you want in the bottom half, and choose the output name (or press 'Recommend' to have tkdegrib 'recommend' one. Next choose 'CSV' for 'File Type', and press the 'Generate.csv file' button at the bottom.

• From the command line: /degrib/bin/degrib 'GRIB file' -C -msg 1 -Csv /degrib/bin/degrib 'GRIB file' -C -msg all -nMet -Csv where 'GRIB file' is replaced with the name of your GRIB file. This should read the file, and extract the requested messages. The -Csv option tells it to create the.csv file. See the (particularly the 'CONVERT OPTIONS' section) for more details. You can use degrib to convert a given message to three different types of ESRI Shapefiles. The first type (Point), contains a lat/lon for the center of each cell, along with the value of that cell.

The second type (Small Polygon) contains a 'small' 5 vertex polygon around the edge, along with the value of each cell. The third type (Large Polygon) contains large polygons that were created by merging all the identical cells from the small polygons. • In the GUI tkdegrib: Click on the 'GIS' tab, and browse for your file. Double click on it in the top half, and it should fill out the inventory part in the bottom half. Select the message that you want in the bottom half, and choose the output name (or press 'Recommend' to have tkdegrib 'recommend' one. Next choose 'SHP' for 'File Type'.

Now choose the 'Type of.shp file' (either point, small polygon, or large polygon) and press the 'Generate.shp file' button at the bottom. • From the command line: /degrib/bin/degrib 'GRIB file' -C -msg 1 -Shp -poly 0 /degrib/bin/degrib 'GRIB file' -C -msg 1 -Shp -poly 1 /degrib/bin/degrib 'GRIB file' -C -msg 1 -Shp -poly small /degrib/bin/degrib 'GRIB file' -C -msg 1 -Shp -poly 2 /degrib/bin/degrib 'GRIB file' -C -msg 1 -Shp -poly big where 'GRIB file' is replaced with the name of your GRIB file.

221g Status Tracker more. This should read the file, and extract the requested messages. The -Shp option tells it to create the.shp file.

The '-poly 0' says to create a 'point' shapefile, the '-poly 1' or '-poly small' says to create the 'small polygon' shapefile, while the '-poly 2' or '-poly big' says to create the 'large polygon' shapefile. See the (particularly the 'CONVERT OPTIONS' section and the 'SHP SPECIFIC OPTIONS') for more details. You can use degrib to convert a given message to what I termed to be a '.flt' file. The.flt file consists of NxM floats where N, and M are the dimensions of the grid.

The reason for the.flt name was because that (along with a '.hdr' file) was what ESRI Spatial analyst appeared to need. It turned out that 'GrADS' used the same format except that in addition to the '.flt' file it required a control file (.ctl). • In the GUI tkdegrib: Click on the 'GIS' tab, and browse for your file. Double click on it in the top half, and it should fill out the inventory part in the bottom half.

Select the message that you want in the bottom half, and choose the output name (or press 'Recommend' to have tkdegrib 'recommend' one. Next choose 'FLT' for 'File Type'. Now if you want to create the GrADS '.ctl' file, click on 'Create GrADS.ctl file'. Then press the 'Generate.flt file' button at the bottom.

• From the command line: /degrib/bin/degrib 'GRIB file' -C -msg 1 -Flt /degrib/bin/degrib 'GRIB file' -C -msg 1 -Flt -Interp -GrADS /degrib/bin/degrib 'GRIB file' -C -msg 1 -Flt -GrADS where 'GRIB file' is replaced with the name of your GRIB file. This should read the file, and extract the requested messages. The -Flt option tells it to create the.flt file. The '-GrADS' option says to create the '.ctl' file.

The '-Interp' allows GrADS to work better since it tells degrib to interpolate the grid to a lat/lon coverage grid, which saves GrADS from having to do so. See the (particularly the 'CONVERT OPTIONS' section and the 'FLT SPECIFIC OPTIONS') for more details. • For more information on 'degrib and GrADS': • For information on 'FORTRAN or C access to.flt files': You can use degrib to convert a given message to a NetCDF file ('.nc'). For more information about the NetCDF format, see • In the GUI tkdegrib: Click on the 'GIS' tab, and browse for your file. Double click on it in the top half, and it should fill out the inventory part in the bottom half.

Select the message that you want in the bottom half, and choose the output name (or press 'Recommend' to have tkdegrib 'recommend' one. Next choose 'NetCDF' for 'File Type'.

Then press the 'Generate NetCDF.nc file' button at the bottom. • From the command line: /degrib/bin/degrib 'GRIB file' -C -msg 1 -NetCDF 1 where 'GRIB file' is replaced with the name of your GRIB file. This should read the file, and extract the requested messages. The '-NetCDF 1' option tells it to create a NetCDF file using degrib's conventions version 1. This attempts to use the. See the (particularly the 'CONVERT OPTIONS' section) for more details.

You can use degrib to get a sub grid of certain types of GRIB2 files. It only works for GRIB2 files that were packed using simple pack, complex pack, or complex pack with second order differencing (which NDFD uses, but NCEP does not), as those are the only forms that it knows how to pack. Note you can't use this to convert a message from GRIB1 to GRIB2, because degrib currently doesn't have a mapping of the GRIB1 meta data to GRIB2 meta data. • In the GUI tkdegrib: Currently there is no way to do this from the GUI. • From the command line: /degrib/bin/degrib 'GRIB file' -C -msg 1 -Grib2 -lwlf 30,-105 -uprt 40,-85 where 'GRIB file' is replaced with the name of your GRIB file. This should read the file, and extract the requested messages.

The '-Grib2' option tells it to create a GRIB2 message. The '-lwlf 30,-105' tells it to use a subgrid with the lower left corner at lon=-105E, lat=30N.

The '-uprt 40,-85' tells it to use a subgrid with the upper right corner at lon=-85E, lat=40N. See the (particularly the 'CONVERT OPTIONS' section) for more details.

You can use degrib to create a flat file database which is made up of either a bunch of.flt files and an index file (.ind), or a single cube file (.dat) and an index file (.ind). This is useful for CGI access to the NDFD, since probing the uncompressed form of the data is faster than probing the GRIB2 files. • In the GUI tkdegrib: Currently there is no way to do this from the GUI. • From the command line: /degrib/bin/degrib 'GRIB file' -Data -Index simple.ind /degrib/bin/degrib 'GRIB file' -Data -Index simple.ind -out simple.dat where 'GRIB file' is replaced with the name of your GRIB file. This should read the file, create a new file 'simple.dat', append each converted grid to simple.dat, and create an index file simple.ind.

The '-Data' option tells it to create the 'database', and the '-Index simple.ind' tells it what to name the index file. The '-out simple.dat' tells it what to name the '.dat' file. By default it uses the index filename replacing the extension with '.dat'. See the (particularly the 'DATABASE CREATE OPTIONS' section) for more details.

You can use degrib to probe a flat file database which is made up of either a bunch of.flt files and an index file (.ind), or a single cube file (.dat) and an index file (.ind). This is useful for CGI access to the NDFD, since probing the uncompressed form of the data is faster than probing the GRIB2 files. • In the GUI tkdegrib: Currently there is no way to do this from the GUI.

• From the command line: /degrib/bin/degrib 'index file' -DP -pnt 35,-76 where 'index file' is replaced with the name of the index file (.ind) that you created when using the '-Data' command. This should read the index file, figure out where 35,-76 is, open the (.dat) file or the set of (.flt) files, jump to and read that cell, and return the results on the command line. See the (particularly the 'DATABASE PROBE OPTIONS' section) for more details. Calculate the total amounts from a specified range of time.

This is intented for QPF, SnowAmount, or IceAccum data when you want the total precipitation for an event. • In the GUI tkedegrib: Currently there is no way to do this from the GUI. • /degrib/bin/degrib 'GRIB file' -StormTotal -startTime 'timestamp' -endTime 'timestamp' -Grib2 where 'GRIB file' is the name of your GRIB file, and the two timestamps are replaced with times such as 2012-05-31T12:00:00 and 2012-06-01T00:00:00. This should create a new GRIB2 file by combining any GRIB messages in the original file between the specified points in time.

Keep in mind that the inventory option shows the end of the time frame and these times are exclusive. For example if the data is for 6 hour chunks, then to include 2012-06-01T18:00:00 in the output, you'd need a start time of 2012-06-01T12:00:00. You can use degrib to convert National Digital Forecast Database (NDFD) GRIB files into Extensible Markup Language (XML). Currently, the only dialect of XML supported is Digital Weather Markup Language (DWML). You can find more information about DWML by reviewing its or its. You can tell degrib to format DWML encoded NDFD data and send it to standard output using the command line template below.

• In the GUI tkdegrib: Currently there is no way to do this from the GUI. • From the command line: Quick Start executable GRIB_files_dir probeType -pnt latidute,longitude -XML optNum -ndfdConven optNum -ndfdVars elemList -Icon optNum -startTime timeString -endTime timeString -geoData GeoDataDir Example: /degrib/bin/degrib. -P -pnt 38.98,-77.07 -XML 1 -ndfdConven 2 -Icon 1 /degrib/data/geodata For more information on each command line option refer to the Step-by-Step Instructions for Building the degrib Command to Generate DWML We are going to use the command template shown in the 'quick start' section to guide the building of a command to generate DWML encoded NDFD data. Follow each step in the table below to build the degrib command. Steps to Build the degrib Command Template Component What your input might look like Locate the degrib executable and note its path. The executable name and path begin the command line input.

Executable /degrib/bin/degrib /degrib/bin/degrib_DP Locate the NDFD data. The directory containing the data is GRIB_files_dir in the command template. GRIB_files_dir /degrib/data Determine how you must probe the data. If you have files containing GRIB messages, then you will use the -P option. If, on the other hand, you have files called 'data cubes' that contain float values that have been decoded from the original GRIB files, then you will use the -DP option.

ProbeType -P Determine the latitude and longitude for each point that you want data for. You can enter more than one '-pnt latitude,longitude' option on the same command line if you want to probe more than one point at a time. You can also put all your points in file and reference it using the -pntFile [File] option (See for more details). -pnt latidute,longitude -pnt 38.98,-77.07 -pnt 39.31,-77.83 Determine which DWML product/format is appropriate for your data needs. The following samples may help you make the selection: -XML 1 - -XML 1 - -XML 2 - -XML 3 - -XML 4 - In addition to the numbered -XML convention, the following can be used: -XML time-series -XML glance -XML '12 hourly' -XML '24 hourly' -XML optNum -XML 1 -XML '12 hourly' -XML glance Using the names of the data files, determine which ndfd naming convention you are following (See for more details). -ndfdConven optNum -ndfdConven 1 Using the names of the data files as the ndfd element name, determine which ndfd elements you want in the output DWML.

(See the section on -ndfdConven for a list of element names). If you want all available ndfd elements in the DWML, then you don't have to include this option. Note, the -ndfdVars option is only applicable for the time-series (-XML 1) option.

-ndfdVars elemList -ndfdVars wspd,temp,wx,sky,temp_r Determine if you want to see icon elements in the output DWML. If so set the -Icon option to 1. If you choose to format icon elements, be sure you have data files for probability of precipitation, sky cover, temperature, weather, and wind speed. These data files are needed by degrib to derive the icon elements. If you don't want icons in the DWML then you don't have to include this option.

Icons by default are formatted for the 'glance' product (-XML 2) and the 'time-series' product (-XML 1) when there is no -ndfdVars argument. -Icon optNum -Icon 1 Determine the time when degrib should start including NDFD in the output DWML. This is a UTC time. If you want the data for all available times, then you don't have to include this or the -endTime option. -startTime timeString -startTime 2006-12-25T23:00:00 Determine the time when degrib should stop including NDFD in the output DWML. This is a UTC time.

If you want the data for all available times, then you don't have to include this or the -startTime option.