Discussion:
[gdal-dev] nodata value and alpha in Float32
Javier Jimenez Shaw
2021-05-31 14:55:27 UTC
Permalink
Hi

I have a GeoTIFF with this characteristics:
- 6 bands
- last band is "Alpha" (with values 0 or 255, nothing else)
- Float32
- NoData value = -10000

The 5 first bands may have "nodata" pixels, not necessarily on all bands
simultaneously. (nodata pixels are usually broken or saturated pixels from
the sensor, and each band comes from a different sensor). The valid pixels
have values between 0 and 1

When I try to generate embedded overviews (with average interpolation) with
C++ or CreateCopy as COG (with default values), looks like it is using the
-10000 values as valid values, and producing results in the overviews like
-9990.6... that obviously are not anymore considered "nodata".

The first noticeable effect opening the file in QGIS, is that the min and
max value for each band are not anymore 0 and 1, but nonsense negative
numbers.

Can both Alpha and NoData live together properly?

Thanks.

PS I have the impression that with external ovr file it does not happen. I
am testing more.
.___ ._ ..._ .. . ._. .___ .. __ . _. . __.. ... .... ._ .__
Entre dos pensamientos racionales
hay infinitos pensamientos irracionales.
Even Rouault
2021-05-31 15:28:46 UTC
Permalink
Javier,

mixing Alpha and NoData is generally not a good idea. Behavior will be
different according to which part of the code is triggered. As far as I
can see, the warping code should be able to handle both, but overview
building will use the return of GDALRasterBand::GetMaskBand(), and when
you have both nodata and alpha, this should be from the nodata value.

Confirmed by the following little experiment, so I'm not sure why you
get what you get:

$ cat test.asc
ncols        2
nrows        2
xllcorner    440720.000000000000
yllcorner    3750120.000000000000
cellsize     60.000000000000
NODATA_value  -10000
 0.5 1.0
 2.5 -10000

$ cat test_alpha.asc
ncols        2
nrows        2
xllcorner    440720.000000000000
yllcorner    3750120.000000000000
cellsize     60.000000000000
NODATA_value  -10000
 255 255
 255 255

$ cat tmp.vrt
<VRTDataset rasterXSize="2" rasterYSize="2">
  <GeoTransform>  4.4072000000000000e+05, 6.0000000000000000e+01, 
0.0000000000000000e+00, 3.7513200000000000e+06,  0.0000000000000000e+00,
-6.0000000000000000e+01</GeoTransform>
  <VRTRasterBand dataType="Float32" band="1">
    <NoDataValue>-10000</NoDataValue>
    <ComplexSource>
      <SourceFilename relativeToVRT="1">test.asc</SourceFilename>
      <SourceBand>1</SourceBand>
      <NODATA>-10000</NODATA>
    </ComplexSource>
  </VRTRasterBand>
  <VRTRasterBand dataType="Float32" band="2">
    <NoDataValue>-10000</NoDataValue>
    <ColorInterp>Alpha</ColorInterp>
    <ComplexSource>
      <SourceFilename relativeToVRT="1">test_alpha.asc</SourceFilename>
      <SourceBand>1</SourceBand>
      <NODATA>-10000</NODATA>
    </ComplexSource>
  </VRTRasterBand>
</VRTDataset>

$ gdal_translate  tmp.vrt test.tif

$ gdaladdo  -r average test.tif 2

$ gdal_translate test.tif /vsistdout/ -of aaigrid -b 1 -outsize 1 1
ncols        1
nrows        1
xllcorner    440720.000000000000
yllcorner    3751200.000000000000
cellsize     120.000000000000
NODATA_value  -10000
 1.3333333730697631836

Even
Post by Javier Jimenez Shaw
Hi
- 6 bands
- last band is "Alpha" (with values 0 or 255, nothing else)
- Float32
- NoData value = -10000
The 5 first bands may have "nodata" pixels, not necessarily on all
bands simultaneously. (nodata pixels are usually broken or saturated
pixels from the sensor, and each band comes from a different sensor).
The valid pixels have values between 0 and 1
When I try to generate embedded overviews (with average interpolation)
with C++ or CreateCopy as COG (with default values), looks like it is
using the -10000 values as valid values, and producing results in the
overviews like -9990.6... that obviously are not anymore considered
"nodata".
The first noticeable effect opening the file in QGIS, is that the min
and max value for each band are not anymore 0 and 1, but nonsense
negative numbers.
Can both Alpha and NoData live together properly?
Thanks.
PS I have the impression that with external ovr file it does not
happen. I am testing more.
.___ ._ ..._ .. . ._. .___ .. __ . _. . __..  ... .... ._ .__
Entre dos pensamientos racionales
hay infinitos pensamientos irracionales.
_______________________________________________
gdal-dev mailing list
https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
http://www.spatialys.com
My software is free, but my time generally not.
Javier Jimenez Shaw
2021-05-31 17:14:11 UTC
Permalink
Thank Even.

I think I found the reason of the problem. It is reproduced with more
bands. I created a second band just swapping the values in the diagonal.
Then, adding the second band to the tmp.vrt (2 normal + 1 alpha), and
following your steps, it is noticeable in the second band of the output:

$ cat test2.asc
ncols 2
nrows 2
xllcorner 440720.000000000000
yllcorner 3750120.000000000000
cellsize 60.000000000000
NODATA_value -10000
-10000 1.0
2.5 0.5

$ cat tmp.vrt
<VRTDataset rasterXSize="2" rasterYSize="2">
<GeoTransform> 4.4072000000000000e+05, 6.0000000000000000e+01,
0.0000000000000000e+00, 3.7513200000000000e+06, 0.0000000000000000e+00,
-6.0000000000000000e+01</GeoTransform>
<VRTRasterBand dataType="Float32" band="1">
<NoDataValue>-10000</NoDataValue>
<ComplexSource>
<SourceFilename relativeToVRT="1">test.asc</SourceFilename>
<SourceBand>1</SourceBand>
<NODATA>-10000</NODATA>
</ComplexSource>
</VRTRasterBand>
<VRTRasterBand dataType="Float32" band="2">
<NoDataValue>-10000</NoDataValue>
<ComplexSource>
<SourceFilename relativeToVRT="1">test.2.asc</SourceFilename>
<SourceBand>1</SourceBand>
<NODATA>-10000</NODATA>
</ComplexSource>
</VRTRasterBand>
<VRTRasterBand dataType="Float32" band="3">
<NoDataValue>-10000</NoDataValue>
<ColorInterp>Alpha</ColorInterp>
<ComplexSource>
<SourceFilename relativeToVRT="1">test_alpha.asc</SourceFilename>
<SourceBand>1</SourceBand>
<NODATA>-10000</NODATA>
</ComplexSource>
</VRTRasterBand>
</VRTDataset>

$ gdal_translate test.tif /vsistdout/ -of aaigrid -b 2 -outsize 1 1
ncols 1
nrows 1
xllcorner 440720.000000000000
yllcorner 3751200.000000000000
cellsize 120.000000000000
NODATA_value -10000
-3332.166748046875

I have not looked at the code, but seems that the mask is requested only
for the first band. Then the pixels in the other bands that are nodata, but
are valid in the first band are computed as valid... using -10000 in this
case. Looks like requesting the mask per band would fix it, right?

If this is the case, the alpha band is not the problem (confirmed removing
it from the tmp.vrt file).

Cheers
.___ ._ ..._ .. . ._. .___ .. __ . _. . __.. ... .... ._ .__
Entre dos pensamientos racionales
hay infinitos pensamientos irracionales.
Post by Even Rouault
Javier,
mixing Alpha and NoData is generally not a good idea. Behavior will be
different according to which part of the code is triggered. As far as I can
see, the warping code should be able to handle both, but overview building
will use the return of GDALRasterBand::GetMaskBand(), and when you have
both nodata and alpha, this should be from the nodata value.
Confirmed by the following little experiment, so I'm not sure why you get
$ cat test.asc
ncols 2
nrows 2
xllcorner 440720.000000000000
yllcorner 3750120.000000000000
cellsize 60.000000000000
NODATA_value -10000
0.5 1.0
2.5 -10000
$ cat test_alpha.asc
ncols 2
nrows 2
xllcorner 440720.000000000000
yllcorner 3750120.000000000000
cellsize 60.000000000000
NODATA_value -10000
255 255
255 255
$ cat tmp.vrt
<VRTDataset rasterXSize="2" rasterYSize="2">
<GeoTransform> 4.4072000000000000e+05, 6.0000000000000000e+01,
0.0000000000000000e+00, 3.7513200000000000e+06, 0.0000000000000000e+00,
-6.0000000000000000e+01</GeoTransform>
<VRTRasterBand dataType="Float32" band="1">
<NoDataValue>-10000</NoDataValue>
<ComplexSource>
<SourceFilename relativeToVRT="1">test.asc</SourceFilename>
<SourceBand>1</SourceBand>
<NODATA>-10000</NODATA>
</ComplexSource>
</VRTRasterBand>
<VRTRasterBand dataType="Float32" band="2">
<NoDataValue>-10000</NoDataValue>
<ColorInterp>Alpha</ColorInterp>
<ComplexSource>
<SourceFilename relativeToVRT="1">test_alpha.asc</SourceFilename>
<SourceBand>1</SourceBand>
<NODATA>-10000</NODATA>
</ComplexSource>
</VRTRasterBand>
</VRTDataset>
$ gdal_translate tmp.vrt test.tif
$ gdaladdo -r average test.tif 2
$ gdal_translate test.tif /vsistdout/ -of aaigrid -b 1 -outsize 1 1
ncols 1
nrows 1
xllcorner 440720.000000000000
yllcorner 3751200.000000000000
cellsize 120.000000000000
NODATA_value -10000
1.3333333730697631836
Even
Hi
- 6 bands
- last band is "Alpha" (with values 0 or 255, nothing else)
- Float32
- NoData value = -10000
The 5 first bands may have "nodata" pixels, not necessarily on all bands
simultaneously. (nodata pixels are usually broken or saturated pixels from
the sensor, and each band comes from a different sensor). The valid pixels
have values between 0 and 1
When I try to generate embedded overviews (with average interpolation)
with C++ or CreateCopy as COG (with default values), looks like it is using
the -10000 values as valid values, and producing results in the overviews
like -9990.6... that obviously are not anymore considered "nodata".
The first noticeable effect opening the file in QGIS, is that the min and
max value for each band are not anymore 0 and 1, but nonsense negative
numbers.
Can both Alpha and NoData live together properly?
Thanks.
PS I have the impression that with external ovr file it does not happen. I
am testing more.
.___ ._ ..._ .. . ._. .___ .. __ . _. . __.. ... .... ._ .__
Entre dos pensamientos racionales
hay infinitos pensamientos irracionales.
_______________________________________________
-- http://www.spatialys.com
My software is free, but my time generally not.
Even Rouault
2021-05-31 17:37:22 UTC
Permalink
Javier,
Post by Javier Jimenez Shaw
I have not looked at the code, but seems that the mask is requested
only for the first band.
You're right. GDALRegenerateOverviewsMultiBand() in
https://github.com/OSGeo/gdal/blob/master/gdal/gcore/overview.cpp#L5057
always takes the mask from the first band. Which is OK for an alpha band
or a per-dataset materialized mask, but not for nodata. Please file a
ticket about that.

Even
Post by Javier Jimenez Shaw
Then the pixels in the other bands that are nodata, but are valid in
the first band are computed as valid... using -10000 in this case.
Looks like requesting the mask per band would fix it, right?
If this is the case, the alpha band is not the problem (confirmed
removing it from the tmp.vrt file).
Cheers
.___ ._ ..._ .. . ._. .___ .. __ . _. . __..  ... .... ._ .__
Entre dos pensamientos racionales
hay infinitos pensamientos irracionales.
Javier,
mixing Alpha and NoData is generally not a good idea. Behavior
will be different according to which part of the code is
triggered. As far as I can see, the warping code should be able to
handle both, but overview building will use the return of
GDALRasterBand::GetMaskBand(), and when you have both nodata and
alpha, this should be from the nodata value.
Confirmed by the following little experiment, so I'm not sure why
$ cat test.asc
ncols        2
nrows        2
xllcorner    440720.000000000000
yllcorner    3750120.000000000000
cellsize     60.000000000000
NODATA_value  -10000
 0.5 1.0
 2.5 -10000
$ cat test_alpha.asc
ncols        2
nrows        2
xllcorner    440720.000000000000
yllcorner    3750120.000000000000
cellsize     60.000000000000
NODATA_value  -10000
 255 255
 255 255
$ cat tmp.vrt
<VRTDataset rasterXSize="2" rasterYSize="2">
  <GeoTransform>  4.4072000000000000e+05, 6.0000000000000000e+01, 
0.0000000000000000e+00, 3.7513200000000000e+06, 
0.0000000000000000e+00, -6.0000000000000000e+01</GeoTransform>
  <VRTRasterBand dataType="Float32" band="1">
    <NoDataValue>-10000</NoDataValue>
    <ComplexSource>
      <SourceFilename relativeToVRT="1">test.asc</SourceFilename>
      <SourceBand>1</SourceBand>
      <NODATA>-10000</NODATA>
    </ComplexSource>
  </VRTRasterBand>
  <VRTRasterBand dataType="Float32" band="2">
    <NoDataValue>-10000</NoDataValue>
    <ColorInterp>Alpha</ColorInterp>
    <ComplexSource>
      <SourceFilename
relativeToVRT="1">test_alpha.asc</SourceFilename>
      <SourceBand>1</SourceBand>
      <NODATA>-10000</NODATA>
    </ComplexSource>
  </VRTRasterBand>
</VRTDataset>
$ gdal_translate  tmp.vrt test.tif
$ gdaladdo  -r average test.tif 2
$ gdal_translate test.tif /vsistdout/ -of aaigrid -b 1 -outsize 1 1
ncols        1
nrows        1
xllcorner    440720.000000000000
yllcorner    3751200.000000000000
cellsize     120.000000000000
NODATA_value  -10000
 1.3333333730697631836
Even
Post by Javier Jimenez Shaw
Hi
- 6 bands
- last band is "Alpha" (with values 0 or 255, nothing else)
- Float32
- NoData value = -10000
The 5 first bands may have "nodata" pixels, not necessarily on
all bands simultaneously. (nodata pixels are usually broken or
saturated pixels from the sensor, and each band comes from a
different sensor). The valid pixels have values between 0 and 1
When I try to generate embedded overviews (with average
interpolation) with C++ or CreateCopy as COG (with default
values), looks like it is using the -10000 values as valid
values, and producing results in the overviews like -9990.6...
that obviously are not anymore considered "nodata".
The first noticeable effect opening the file in QGIS, is that the
min and max value for each band are not anymore 0 and 1, but
nonsense negative numbers.
Can both Alpha and NoData live together properly?
Thanks.
PS I have the impression that with external ovr file it does not
happen. I am testing more.
.___ ._ ..._ .. . ._.  .___ .. __ . _. . __..  ... .... ._ .__
Entre dos pensamientos racionales
hay infinitos pensamientos irracionales.
_______________________________________________
gdal-dev mailing list
https://lists.osgeo.org/mailman/listinfo/gdal-dev <https://lists.osgeo.org/mailman/listinfo/gdal-dev>
--
http://www.spatialys.com <http://www.spatialys.com>
My software is free, but my time generally not.
--
http://www.spatialys.com
My software is free, but my time generally not.
Loading...