[gdal-dev] New JP2Grok Driver for JPEG 2000

[Aaron Boxer]

Hi Folks,

I would like to draw people's attention once again to the pending JPEG 2000
driver PR
https://github.com/OSGeo/gdal/pull/3449
which was opened 3 months ago. Since I last posted, the driver now has an
autotest script courtesy of Frank Warmerdam, and all tests are passing.

In a nutshell, performance is 2-5 times faster than the only other viable
open source driver, from OpenJPEG, in several common geospatial work flows.
And performance rises by around 2X if the new HTJ2K entropy coder is used.
The AGPL license will not work for everyone, which is why the driver is
disabled by default.

We've already discussed this at length in the previous thread and in the
PR, but if you would like to see this new driver merged, please head over
to the PR and give it a thumbs up. Or, if you don't want this driver
merged, please share your feedback on the PR.

Kind Regards,
Aaron

[Jean-Roc Morreale (ml)]

Hi Aaron,

Is the HTJ2K's output still readable by a regular GDAL/OpenJPEG ?

Regards,
Jean-Roc Morreale

[Aaron Boxer]

Hi Jean-Roc,
Good question - unfortunately OpenJPEG doesn't currently support HTJ2K,
it only supports JPEG 2000 Part 1. I am sure that this will eventually be
integrated,
but it isn't there at the moment.
Cheers,
Aaron

On Mon, May 3, 2021 at 12:05 PM Jean-Roc Morreale (ml) <

Post by Jean-Roc Morreale (ml)
Hi Aaron,
Is the HTJ2K's output still readable by a regular GDAL/OpenJPEG ?
Regards,
Jean-Roc Morreale

[Sean Gillies via gdal-dev]

Hi Aaron, and everyone,

It seems like interoperability could be harmed if we release GDAL with a
JP2 driver that writes JPEG 2000 files that the main open source JP2 driver
can't read. Would it make sense to add compatibility to OpenJPEG before the
PR gets merged? Or are we already in a state of inoperability between
JP2ECW, JP2MRSID, and JP2OpenJPEG?

Post by Aaron Boxer
Hi Jean-Roc,
Good question - unfortunately OpenJPEG doesn't currently support HTJ2K,
it only supports JPEG 2000 Part 1. I am sure that this will eventually be
integrated,
but it isn't there at the moment.
Cheers,
Aaron
On Mon, May 3, 2021 at 12:05 PM Jean-Roc Morreale (ml) <

Post by Jean-Roc Morreale (ml)
Hi Aaron,
Is the HTJ2K's output still readable by a regular GDAL/OpenJPEG ?
Regards,
Jean-Roc Morreale

[Even Rouault]

Sean,

my quick look at the JP2Grok driver code is that it doesn't generate the
HTJ2K / Part 15 variant of JPEG2000, at least there's nothing suggesting
that (didn't try running it). I guess that there would be a need for a
dedicated creation option in the driver for that to happen, so there
shouldn't be interoperability issues of the files it generates with
other JPEG2000 capable drivers.

I'm not sure which support of HTJ2K the ECW and MrSID SDKs have. I
believe there's such support Kakadu in recent releases, but might be
subject to extra fees (or perhaps for  faster support)

There's a BSD 2 licensed implementation of HTJ2K in
https://github.com/aous72/OpenJPH . From what I see, Grok HTJ2K support
has incorporated some of that code.  So for a BSD 2 compatible use in
GDAL, either a GDAL driver should be built from OpenJPH as a standalone
library, or relevant parts of OpenJPH should be "ported" to OpenJPEG.

Even

Hi  Aaron, and everyone,
It seems like interoperability could be harmed if we release GDAL with
a JP2 driver that writes JPEG 2000 files that the main open source JP2
driver can't read. Would it make sense to add compatibility to
OpenJPEG before the PR gets merged? Or are we already in a state of
inoperability between JP2ECW, JP2MRSID, and JP2OpenJPEG?
Hi Jean-Roc,
Good question - unfortunately OpenJPEG doesn't currently support HTJ2K,
it only supports JPEG 2000 Part 1. I am sure that this will
eventually be integrated,
but it isn't there at the moment.
Cheers,
Aaron
On Mon, May 3, 2021 at 12:05 PM Jean-Roc Morreale (ml)
Hi Aaron,
Is the HTJ2K's output still readable by a regular GDAL/OpenJPEG ?
Regards,
Jean-Roc Morreale
--
Sean Gillies
_______________________________________________
gdal-dev mailing list
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[Aaron Boxer]

Hi Sean and Even,
Correct, the current JP2Grok does not support the generation of Part 15
code streams.
It's trivial to add, as it just involves adding a new encoding mode, but I
was waiting for a merge
before looking into new driver features.

Also, yes, OpenJPH is being used for the new entropy encoder. As OpenJPH
library is in its
infancy and is missing many features of a mature implementation, I would
recommend
the best approach to be integrating OpenJPH into OpenJPEG and upgrading the
gdal driver
accordingly.

Cheers,
Aaron

Post by Even Rouault
Sean,
my quick look at the JP2Grok driver code is that it doesn't generate the
HTJ2K / Part 15 variant of JPEG2000, at least there's nothing suggesting
that (didn't try running it). I guess that there would be a need for a
dedicated creation option in the driver for that to happen, so there
shouldn't be interoperability issues of the files it generates with other
JPEG2000 capable drivers.
I'm not sure which support of HTJ2K the ECW and MrSID SDKs have. I believe
there's such support Kakadu in recent releases, but might be subject to
extra fees (or perhaps for faster support)
There's a BSD 2 licensed implementation of HTJ2K in
https://github.com/aous72/OpenJPH . From what I see, Grok HTJ2K support
has incorporated some of that code. So for a BSD 2 compatible use in GDAL,
either a GDAL driver should be built from OpenJPH as a standalone library,
or relevant parts of OpenJPH should be "ported" to OpenJPEG.
Even
Hi Aaron, and everyone,
It seems like interoperability could be harmed if we release GDAL with a
JP2 driver that writes JPEG 2000 files that the main open source JP2 driver
can't read. Would it make sense to add compatibility to OpenJPEG before the
PR gets merged? Or are we already in a state of inoperability between
JP2ECW, JP2MRSID, and JP2OpenJPEG?

Post by Aaron Boxer
Hi Jean-Roc,
Good question - unfortunately OpenJPEG doesn't currently support HTJ2K,
it only supports JPEG 2000 Part 1. I am sure that this will eventually be
integrated,
but it isn't there at the moment.
Cheers,
Aaron
On Mon, May 3, 2021 at 12:05 PM Jean-Roc Morreale (ml) <

Post by Jean-Roc Morreale (ml)
Hi Aaron,
Is the HTJ2K's output still readable by a regular GDAL/OpenJPEG ?
Regards,
Jean-Roc Morreale

--
Sean Gillies
_______________________________________________
-- http://www.spatialys.com
My software is free, but my time generally not.
_______________________________________________
gdal-dev mailing list
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[jratike80]

Hi Even,

You are right, Kakadu 8.x supports HTJ2K. I just had a try with the
kdu_compress demo utility from KDU805 demo apps. The speed was good,
actually amazing. As a test image I used a 12000x12000 RGB image. The
kdu_compress command I took from this high qualite white-paper
https://htj2k.com/wp-content/uploads/white-paper.pdf.

kdu_compress -i p4433h.ppm -o tmp3.jph -rate 2 -tolerance 0 Qstep=0.001
-quiet -no_weights

As a reference I used

gdal_translate -of gtiff -co compress=jpeg p4433h.tif comp.tif

Results (total time disk-to-disk on Windows, SSD disk, laptop with Intel
core i7)

kdu_compress - JPH: 1.4 seconds (file size 36 MB, fixed size defined by
option -rate 2)
kdu_compress - JPH: 1.1 seconds (file size 90 MB, fixed size defined by
option -rate 5)
gdal_translate - TIFF: 13.1 seconds (file size 92 MB)

The Kakadu demo applications that can be acquired on request from the Kakadu
site are compiled without libtiff and therefore the input must be in ppm
format. Those who have access to Kakadu source code can compile the
applications with "KDU_INCLUDE_TIFF".

-Jukka Rahkonen-

Post by Even Rouault
I'm not sure which support of HTJ2K the ECW and MrSID SDKs have. I
believe there's such support Kakadu in recent releases, but might be
subject to extra fees (or perhaps for  faster support)

--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html

[Even Rouault]

Post by jratike80
gdal_translate -of gtiff -co compress=jpeg p4433h.tif comp.tif

for the gdal_translate to JPEG compressed TIFF, adding -co
NUM_THREADS=ALL_CPUS should help to keep the comparison fair, and if you
use gisinternal builds, I'm not completely sure they use libjpeg-turbo,
which would bring another performance boost.

[jratike80]

Hi,

gdal_translate -of gtiff -co compress=jpeg p4433h.tif -co
NUM_THREADS=ALL_CPUS comp.tif

Time: 4.7 seconds (with GDAL 3.3.0 gisinternals build)
Time: 11.5 seconds (with GDAL 3.1.4 OSGeo4W build)

-Jukka-



--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html

[Even Rouault]

Aaron,

did you bench timings using GDAL API & utilities (typically with
gdal_translate) ? The ones indicated in the PR use the utilities that
come which each library, but the integration within a GDAL driver might
have some influence on the timings.

What about single threaded usage ?

Even

Post by Aaron Boxer
Hi Folks,
I would like to draw people's attention once again to the pending JPEG
2000 driver PR
https://github.com/OSGeo/gdal/pull/3449
<https://github.com/OSGeo/gdal/pull/3449>
which was opened 3 months ago.  Since I last posted, the driver now
has an autotest script courtesy of Frank Warmerdam, and all tests are
passing.
In a nutshell, performance is 2-5 times faster than the only other
viable open source driver, from OpenJPEG, in several common geospatial
work flows. And performance rises by around 2X if the new HTJ2K
entropy coder is used. The AGPL license will not work for everyone,
which is why the driver is disabled by default.
We've already discussed this at length in the previous thread and in
the PR, but if you would like to see this new driver merged, please
head over to the PR and give it a thumbs up. Or, if you don't want
this driver merged, please share your feedback on the PR.
Kind Regards,
Aaron
_______________________________________________
gdal-dev mailing list
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[Aaron Boxer]

Post by Even Rouault
did you bench timings using GDAL API & utilities (typically with
gdal_translate) ? The ones indicated in the PR use the utilities that come
which each library, but the integration within a GDAL driver might have
some influence on the timings.

Yes, the timings will definitely be lower with the current driver, as it
doesn't take full advantage of some library features
such as

1. tile cache
2. multi-threading across tiles
3. combination of memory mapping and PLT markers, where parts of
the codestream that don't need to be decoded are not even read from disk at
all.

There's no reason why JP2Grok can't match the library utilities in
performance, once I support
the above features in the driver.

Post by Even Rouault
What about single threaded usage ?

I can certainly benchmark this.

Aaron

Post by Even Rouault
Hi Folks,
I would like to draw people's attention once again to the pending JPEG
2000 driver PR
https://github.com/OSGeo/gdal/pull/3449
which was opened 3 months ago. Since I last posted, the driver now has an
autotest script courtesy of Frank Warmerdam, and all tests are passing.
In a nutshell, performance is 2-5 times faster than the only other viable
open source driver, from OpenJPEG, in several common geospatial work flows.
And performance rises by around 2X if the new HTJ2K entropy coder is used.
The AGPL license will not work for everyone, which is why the driver is
disabled by default.
We've already discussed this at length in the previous thread and in the
PR, but if you would like to see this new driver merged, please head over
to the PR and give it a thumbs up. Or, if you don't want this driver
merged, please share your feedback on the PR.
Kind Regards,
Aaron
_______________________________________________
-- http://www.spatialys.com
My software is free, but my time generally not.