[gdal-dev] Gdal doesn't see NaN values as NoData?

[M***@yukon.ca]

Hi Folks,

I have a 32bit NDVI image that I'm told was exported from Google Earth Engine, with possibly a trip through R along the way. In ArcGIS desktop and Qgis there are "nan" values reported for the image collar, however gdalinfo doesn't say anything about them. If I project the image in gdalwarp the output image contains both nan AND nodata. This is rather confusing.

a) Why does gdalinfo not report the presence of nan? Meaning: how do I know when I will need to invoke (b)?
b) What do I need to do so that applications will use nan as nodata?

Fig1: Projected output. Pink == nodata, red == 0, black == nan.


Here's the file  https://sft.gov.yk.ca/link/61czGFXOyyFsh387CxbTEF
(255 MB, link valid for 3 months. The image is free and libre.)

Gdalinfo report:

Size is 14742, 6761
Coordinate System is:
GEOGCRS["WGS 84",
    DATUM["World Geodetic System 1984",
        ELLIPSOID["WGS 84",6378137,298.257223563,
            LENGTHUNIT["metre",1]]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433]],
    CS[ellipsoidal,2],
        AXIS["geodetic latitude (Lat)",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433]],
        AXIS["geodetic longitude (Lon)",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433]],
    ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
Origin = (-137.935952550438884,61.086337635411589)
Pixel Size = (0.000269494585236,-0.000269494585236)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (-137.9359526,  61.0863376) (137d56' 9.43"W, 61d 5'10.82"N)
Lower Left  (-137.9359526,  59.2642847) (137d56' 9.43"W, 59d15'51.43"N)
Upper Right (-133.9630634,  61.0863376) (133d57'47.03"W, 61d 5'10.82"N)
Lower Right (-133.9630634,  59.2642847) (133d57'47.03"W, 59d15'51.43"N)
Center      (-135.9495080,  60.1753112) (135d56'58.23"W, 60d10'31.12"N)
Band 1 Block=256x256 Type=Float32, ColorInterp=Gray
  Description = NDVI
  Min=-0.888 Max=0.992
  Minimum=-0.888, Maximum=0.992, Mean=0.103, StdDev=0.141
  Overviews: 7371x3381, 3686x1691, 1843x846, 922x423, 461x212
  Metadata:
    STATISTICS_COVARIANCES=0.01992952634076865
    STATISTICS_MAXIMUM=0.9921875
    STATISTICS_MEAN=0.10253506184277
    STATISTICS_MINIMUM=-0.88755017518997
    STATISTICS_SKIPFACTORX=1
    STATISTICS_SKIPFACTORY=1
    STATISTICS_STDDEV=0.14117197434607


Gdalwarp command:

set _opt=-co compress=zstd -co level=17 -co num_threads=all_cpus -co predictor=yes -co bigtiff=yes

gdalwarp -wm 512 %_opt% -of cog -t_srs epsg:3579 -r med NDVI1984.tif t\NDVI1984.tif


GDAL 3.1.4, released 2020/10/20

Thanks!

Matt Wilkie
Geomatics Analyst
Environment | Technology, Innovation and Mapping
T 867-667-8133 | http://yukon.ca/
Hours: 08:30-16:30, Tue-Wed: Office, Thu-Fri: Remote.

[Even Rouault]

Matt,

Post by M***@yukon.ca
a) Why does gdalinfo not report the presence of nan? Meaning: how do I know when I will need to invoke (b)?

It doesn't report nan as the nodata value, because the file lacks the
GDAL_NODATA tiff tag set at "nan". So the issue is on the producer side.

Post by M***@yukon.ca
b) What do I need to do so that applications will use nan as nodata?

gdalwarp -dstnodata nan ...

or if you want to remap nan to some other value

gdalwarp -dstnodata -9999 ...

In most cases, you'd need to explicitly add the srcnodata value with
-srcnodata nan , but nan is a special case. When gdalwarp finds a pixel
at NaN value, be it the declared nodata value or not, it assumes this is
nodata, hence you can omit -srcnodata nan. Wouldn't be true if 0 or some
other value was the undeclared nodata  value.


I've some doubt that it is produced by GEE. Seems move to come from Esri
software seeing:

$ tiffinfo NDVI1984.tif
[...]

  Tag 42112: <GDALMetadata>
  <Item name="PyramidResamplingType" domain="Esri">NEAREST</Item>
  <Item name="DESCRIPTION" sample="0" role="description">NDVI</Item>
</GDALMetadata>

Even

[M***@yukon.ca]

Post by M***@yukon.ca
a) Why does gdalinfo not report the presence of nan? Meaning: how do I know when I will need to invoke (b)?

It doesn't report nan as the nodata value, because the file lacks the GDAL_NODATA tiff tag set at "nan". So the issue is on the producer side.

Ok, noted. I will try trace this backward and report there.

Post by M***@yukon.ca
b) What do I need to do so that applications will use nan as nodata?

gdalwarp -dstnodata nan ...
or if you want to remap nan to some other value
gdalwarp -dstnodata -9999 ...

Wonderful, thank you.

In most cases, you'd need to explicitly add the srcnodata value with
-srcnodata nan , but nan is a special case. When gdalwarp finds a pixel
at NaN value, be it the declared nodata value or not, it assumes this is
nodata, hence you can omit -srcnodata nan. Wouldn't be true if 0 or some
other value was the undeclared nodata  value.

Hmmm. Part of me expects gdalwarp to then record it found and used undeclared nodata by setting the "nodata=nan" in the metadata. Regardless thanks for this important detail.

-Matt