[gdal-dev] Introducing the cogger and godal projects

[thomas bonfort]

Hello gdal,

We're releasing two projects on github under an Apache-2.0 licence
which may be of interest to the GDAL community.

The first one, https://github.com/airbusgeo/cogger is a lightweight
geotiff to COG converter that reshuffles the bytes of a tiled geotiff
to make it cloud compatible. Whereas it does not replace the standard
gdal tools for initially creating a geotiff, it is much faster than
the last "-co COPY_SRC_OVERVIEWS=YES" classical gdal_translate method
for outputting a cog, and also allows more flexibility when varying
compression methods or tile sizes need to applied on overviews. Cogger
can be used as a standalone command line tool, or as a go library
function for interacting with remote storage without the need for
local files.

The second one, https://github.com/airbusgeo/godal is a golang wrapper
for GDAL's raster api. While there is an already established golang
wrapper for gdal that exists (https://github.com/lukeroth/gdal), godal
differs by offering:
- robust error handling and reporting
- a stable compatible api promise (to the extent of what is possible
to achieve given gdal's own api)
- a vsi abstraction that hands off i/o to native golang code
- a more user-friendly API that is not an exact mirror of the gdal
api, making lesser used functionality optional. (For example, instead
of the GDALOpen, GDALOpenEx and GDALOpenShared methods, godal exposes
a single Open() method with optional arguments validated at compile
time to pass in open options, shared mode, sibling files, etc...)
Although the raster handling side of godal is mostly feature complete
with gdal, the vector and spatial-referencing part is not.

Outside contributions to both these projects are welcome through github.

Best regards,
Thomas

[Even Rouault]

Hi Thomas,

cogger is really cool, and certainly a useful complement to the GDAL COG
driver (we could probably do something similar in it directly, if the
input file is a TIFF, but that doesn't fit in a very natural way in the
GDAL machinery). Feel free to submit a PR to the COG driver doc pointing
to cogger.

Same against https://gdal.org/api/index.html#gdal-ogr-in-other-languages
regarding your golang wrapper.

Even

Post by thomas bonfort
Hello gdal,
We're releasing two projects on github under an Apache-2.0 licence
which may be of interest to the GDAL community.
The first one, https://github.com/airbusgeo/cogger is a lightweight
geotiff to COG converter that reshuffles the bytes of a tiled geotiff
to make it cloud compatible. Whereas it does not replace the standard
gdal tools for initially creating a geotiff, it is much faster than
the last "-co COPY_SRC_OVERVIEWS=YES" classical gdal_translate method
for outputting a cog, and also allows more flexibility when varying
compression methods or tile sizes need to applied on overviews. Cogger
can be used as a standalone command line tool, or as a go library
function for interacting with remote storage without the need for
local files.
The second one, https://github.com/airbusgeo/godal is a golang wrapper
for GDAL's raster api. While there is an already established golang
wrapper for gdal that exists (https://github.com/lukeroth/gdal), godal
- robust error handling and reporting
- a stable compatible api promise (to the extent of what is possible
to achieve given gdal's own api)
- a vsi abstraction that hands off i/o to native golang code
- a more user-friendly API that is not an exact mirror of the gdal
api, making lesser used functionality optional. (For example, instead
of the GDALOpen, GDALOpenEx and GDALOpenShared methods, godal exposes
a single Open() method with optional arguments validated at compile
time to pass in open options, shared mode, sibling files, etc...)
Although the raster handling side of godal is mostly feature complete
with gdal, the vector and spatial-referencing part is not.
Outside contributions to both these projects are welcome through github.
Best regards,
Thomas
_______________________________________________
gdal-dev mailing list
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[M***@yukon.ca]

Is cogger specifically for the scenario where your converting a large imagery library that already exists, but isn't cloud optimized? i.e. Does it offer any advantages over the one step `gdal_translate -of cog ...` when starting fresh?

Cheers,

Matt
Geomatics Analyst | Environment | T 867-667-8133 | Yukon.ca
Hours: 08:30-16:30, Tue-Wed: Office, Thu-Fri: Remote.

[thomas bonfort]

I haven't extensively used -of COG (the cogger code actually predates the
cog driver) but iirc there are at least some cases where it uses an
intermediate file, which would imply that cogger does offer some speedups.
I'll let Even confirm...
Regards,
Thomas

Post by M***@yukon.ca
Is cogger specifically for the scenario where your converting a large
imagery library that already exists, but isn't cloud optimized? i.e. Does
it offer any advantages over the one step `gdal_translate -of cog ...` when
starting fresh?
Cheers,
Matt
Geomatics Analyst | Environment | T 867-667-8133 | Yukon.ca
Hours: 08:30-16:30, Tue-Wed: Office, Thu-Fri: Remote.

[Even Rouault]

I haven't benchmarked cogger but I'd expect it to be much faster than
gdal_translate -of COG (if your input is GeoTIFF and always properly
tiled and compressed of course!). gdal_translate -of COG uses generic
GDAL API to acquire input data, which implies decompression /
recompression. And in particular, if using input GeoTIFF with lossy
compressed methods such as JPEG or WebP, cogger will avoid adding any
new loss.

Post by thomas bonfort
I haven't extensively used -of COG (the cogger code actually predates
the cog driver) but iirc there are at least some cases where it uses
an intermediate file, which would imply that cogger does offer some
speedups. I'll let Even confirm...
Regards,
Thomas
Is cogger specifically for the scenario where your converting a
large imagery library that already exists, but isn't cloud
optimized? i.e. Does it offer any advantages over the one step
`gdal_translate -of cog ...` when starting fresh?
Cheers,
Matt
Geomatics Analyst | Environment | T 867-667-8133 | Yukon.ca
Hours: 08:30-16:30, Tue-Wed: Office, Thu-Fri: Remote.
_______________________________________________
gdal-dev mailing list
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[M***@yukon.ca]


I'd expect it to be much faster than gdal_translate -of COG (if your input is GeoTIFF and always properly tiled and compressed of course!).

I’m confident I know what a properly compressed geotiff is, but what is a properly tiled one? (or if it’s easier to define: what are improperly tiled geotiff?)


-Matt

From: Even Rouault <***@spatialys.com>
Sent: June 4, 2021 9:33 AM
To: thomas bonfort <***@gmail.com>; Matt.Wilkie <***@yukon.ca>
Cc: Gdal Dev <gdal-***@lists.osgeo.org>
Subject: Re: [gdal-dev] Introducing the cogger and godal projects


*** External email: Do not click on links or attachments except from trusted senders. ***

******************************************************************************************




I haven't benchmarked cogger but I'd expect it to be much faster than gdal_translate -of COG (if your input is GeoTIFF and always properly tiled and compressed of course!). gdal_translate -of COG uses generic GDAL API to acquire input data, which implies decompression / recompression. And in particular, if using input GeoTIFF with lossy compressed methods such as JPEG or WebP, cogger will avoid adding any new loss.
Le 04/06/2021 à 18:28, thomas bonfort a écrit :
I haven't extensively used -of COG (the cogger code actually predates the cog driver) but iirc there are at least some cases where it uses an intermediate file, which would imply that cogger does offer some speedups. I'll let Even confirm...
Regards,
Thomas
Le ven. 4 juin 2021 à 18:14, <***@yukon.ca<mailto:***@yukon.ca>> a écrit :
Is cogger specifically for the scenario where your converting a large imagery library that already exists, but isn't cloud optimized? i.e. Does it offer any advantages over the one step `gdal_translate -of cog ...` when starting fresh?

Cheers,

Matt
Geomatics Analyst | Environment | T 867-667-8133 | Yukon.ca
Hours: 08:30-16:30, Tue-Wed: Office, Thu-Fri: Remote.



_______________________________________________

gdal-dev mailing list

gdal-***@lists.osgeo.org<mailto:gdal-***@lists.osgeo.org>

https://lists.osgeo.org/mailman/listinfo/gdal-dev

[Even Rouault]

I meant that the input GeoTIFF file should use the tile size and
compression method that you expect for the final COG, to be usable with
cogger. If you need to re-tile or change compression method, use the
GDAL COG driver

Post by M***@yukon.ca

I'd expect it to be much faster than gdal_translate -of COG (if your
input is GeoTIFF and always properly tiled and compressed of course!).
I’m confident I know what a properly compressed geotiff is, but what
is a properly tiled one? (or if it’s easier to define: what are
improperly tiled geotiff?)
-Matt//
*Sent:* June 4, 2021 9:33 AM
*Subject:* Re: [gdal-dev] Introducing the cogger and godal projects
**** External email: Do not click on links or attachments except from
trusted senders. ****
********************************************************************************************
I haven't benchmarked cogger but I'd expect it to be much faster than
gdal_translate -of COG (if your input is GeoTIFF and always properly
tiled and compressed of course!). gdal_translate -of COG uses generic
GDAL API to acquire input data, which implies decompression /
recompression. And in particular, if using input GeoTIFF with lossy
compressed methods such as JPEG or WebP, cogger will avoid adding any
new loss.
I haven't extensively used -of COG (the cogger code actually
predates the cog driver) but iirc there are at least some cases
where it uses an intermediate file, which would imply that cogger
does offer some speedups. I'll let Even confirm...
Regards,
Thomas
Is cogger specifically for the scenario where your converting
a large imagery library that already exists, but isn't cloud
optimized? i.e. Does it offer any advantages over the one step
`gdal_translate -of cog ...` when starting fresh?
Cheers,
Matt
Geomatics Analyst | Environment | T 867-667-8133 | Yukon.ca
Hours: 08:30-16:30, Tue-Wed: Office, Thu-Fri: Remote.
_______________________________________________
gdal-dev mailing list
https://lists.osgeo.org/mailman/listinfo/gdal-dev <https://lists.osgeo.org/mailman/listinfo/gdal-dev>
--
http://www.spatialys.com <https://imsva91-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=http%3a%2f%2fwww.spatialys.com&umid=BB1EB12E-C3F3-3E05-A4FC-DC578DED8704&auth=c132af8ee7c9d1278d61a701569070a095ce962e-b9e494bc988145611c35e65f34295c2b143834aa>
My software is free, but my time generally not.

[Sean Gillies via gdal-dev]

Hi Thomas,

Congratulations! These look like great projects.

Post by thomas bonfort
Hello gdal,
We're releasing two projects on github under an Apache-2.0 licence
which may be of interest to the GDAL community.
The first one, https://github.com/airbusgeo/cogger is a lightweight
geotiff to COG converter that reshuffles the bytes of a tiled geotiff
to make it cloud compatible. Whereas it does not replace the standard
gdal tools for initially creating a geotiff, it is much faster than
the last "-co COPY_SRC_OVERVIEWS=YES" classical gdal_translate method
for outputting a cog, and also allows more flexibility when varying
compression methods or tile sizes need to applied on overviews. Cogger
can be used as a standalone command line tool, or as a go library
function for interacting with remote storage without the need for
local files.
The second one, https://github.com/airbusgeo/godal is a golang wrapper
for GDAL's raster api. While there is an already established golang
wrapper for gdal that exists (https://github.com/lukeroth/gdal), godal
- robust error handling and reporting
- a stable compatible api promise (to the extent of what is possible
to achieve given gdal's own api)
- a vsi abstraction that hands off i/o to native golang code
- a more user-friendly API that is not an exact mirror of the gdal
api, making lesser used functionality optional. (For example, instead
of the GDALOpen, GDALOpenEx and GDALOpenShared methods, godal exposes
a single Open() method with optional arguments validated at compile
time to pass in open options, shared mode, sibling files, etc...)
Although the raster handling side of godal is mostly feature complete
with gdal, the vector and spatial-referencing part is not.
Outside contributions to both these projects are welcome through github.
Best regards,
Thomas

I'm very interested in "a vsi abstraction that hands off i/o to native
golang code".