Log message:
hdf5: Update to 1.10.6
New Features
============
Configuration:
-------------
- Update CMake for VS2019 support
CMake added support for VS2019 in version 3.15. Changes to the CMake
generator setting required changes to scripts. Also updated version
references in CMake files as necessary.
(ADB - 2019/11/18, HDFFV-10962)
- Update CMake options to match new autotools options
Add configure options (autotools - CMake):
enable-asserts HDF5_ENABLE_ASSERTS
enable-symbols HDF5_ENABLE_SYMBOLS
enable-profiling HDF5_ENABLE_PROFILING
enable-optimization HDF5_ENABLE_OPTIMIZATION
In addition NDEBUG is no longer forced defined and relies on the CMake
process.
(ADB - 2019/10/07, HDFFV-100901, HDFFV-10637, TRILAB-97)
- Update CMake tests to use FIXTURES
CMake test fixtures allow setup/cleanup tests and other dependency
requirements as properties for tests. This is more flexible for
modern CMake code.
(ADB - 2019/07/23, HDFFV-10529)
- Windows PDB files are always installed
There are build configuration or flag settings for Windows that may not
generate PDB files. If those files are not generated then the install
utility will fail because those PDB files are not found. An optional
variable, DISABLE_PDB_FILES, was added to not install PDB files.
(ADB - 2019/07/17, HDFFV-10424)
- Add mingw CMake support with a toolchain file
There have been a number of mingw issues that have been linked under
HDFFV-10845. It has been decided to implement the CMake cross-compiling
technique of toolchain files. We will use a linux platform with the mingw
compiler stack for testing. Only the C language is fully supported, and
the error tests are skipped. The C++ language works for static but shared
builds have a shared library issue with the mingw Standard Exception Handling
library, which is not available on Windows. Fortran has a common cross-compile
problem with the fortran configure tests.
(ADB - 2019/07/12, HDFFV-10845, HDFFV-10595)
- Windows PDB files are installed incorrectly
For static builds, the PDB files for windows should be installed next
to the static libraries in the lib folder. Also the debug versions of
libraries and PDB files are now correctly built using the default
CMAKE_DEBUG_POSTFIX setting.
(ADB - 2019/07/09, HDFFV-10581)
- Add option to build only shared libs
A request was made to prevent building static libraries and only build
shared. A new option was added to CMake, ONLY_SHARED_LIBS, which will
skip building static libraries. Certain utility functions will build with
static libs but are not published. Tests are adjusted to use the correct
libraries depending on SHARED/STATIC settings.
(ADB - 2019/06/12, HDFFV-10805)
- Add options to enable or disable building tools and tests
Configure options --enable-tests and --enable-tools were added for
autotools configure. These options are enabled by default, and can be
disabled with either --disable-tests (or tools) or --enable-tests=no
(or --enable-tools=no). Build time is reduced ~20% when tools are
disabled, 35% when tests are disabled, 45% when both are disabled.
Re-enabling them after the initial build requires running configure
again with the option(s) enabled.
(LRK - 2019/06/12, HDFFV-9976)
- Change tools tests to search the error stack
There are some use cases which can cause the error stack of tools to be
different then the expected output. These tests now use grepTest.cmake;
this was changed to allow the error file to be searched for an expected string.
(ADB - 2019/04/15, HDFFV-10741)
Library:
--------
- Added S3 and HDFS Virtual File Drivers (VFDs) to HDF5
These new VFDs have been introduced in HDF5-1.10.6. Instructions to
enable them when configuring HDF5 on Linux and Mac may be found at
https://portal.hdfgroup.org/display/HDF5/Virtual+File+Drivers+-+S3+and+HDFS.
Installing on Windows requires CMake 3.13 and the following additional setup.
Install openssl library (with dev files);
from "Shining Light Productions". msi package preferred.
PATH should have been updated with the installation dir.
set ENV variable OPENSSL_ROOT_DIR to the installation dir.
set ENV variable OPENSSL_CONF to the cfg file, likely \
%OPENSSL_ROOT_DIR%\bin\openssl.cfg
Install libcurl library (with dev files);
download the latest released version using git: \
https://github.com/curl/curl.git
Open a Visual Studio Command prompt
change to the libcurl root folder
run the "buildconf.bat" batch file
change to the winbuild directory
nmake /f Makefile.vc mode=dll MACHINE=x64
copy libcurl-vc-x64-release-dll-ipv6-sspi-winssl dir to C:\curl \
(installation dir)
set ENV variable CURL_ROOT to C:\curl (installation dir)
update PATH ENV variable to %CURL_ROOT%\bin (installation bin dir).
the aws credentials file should be in %USERPROFILE%\.aws folder
set the ENV variable \
"HDF5_ROS3_TEST_BUCKET_URL=https://s3.us-east-2.amazonaws.com/hdf5ros3"
(ADB - 2019/09/12, HDFFV-10854)
C++ Library:
------------
- Added new wrappers for H5Pset/get_create_intermediate_group()
LinkCreatPropList::setCreateIntermediateGroup()
LinkCreatPropList::getCreateIntermediateGroup()
(BMR - 2019/04/22, HDFFV-10622)
Java Library:
----------------
- Fixed a failure in JUnit-TestH5P on 32-bit architectures
(JTH - 2019/04/30)
Support for new platforms, languages and compilers.
=======================================
- CMake added support for VS2019 in version 3.15. Updated scripts.
- macOS 10.13.6 Darwin 17.7.0 with Apple clang LLVM version 10.0.0
- macOS 10.14.6 Darwin 18.7.0 with Apple clang LLVM version 10.0.1
Bug Fixes since HDF5-1.10.5 release
==================================
Library
-------
- Improved performance when creating a large number of small datasets by
retrieving default property values from the API context instead of doing
skip list searches. More work is required to achieve parity with HDF5 1.8.
(CJH - 2019/12/10, HDFFV-10658)
- Fixed user-created data access properties not existing in the property list
returned by H5Dget_access_plist. Thanks to Steven Varga for submitting a
reproducer and a patch.
(CJH - 2019/12/9, HDFFV-10934)
- Inappropriate linking with deprecated MPI C++ libraries
HDF5 does not define *_SKIP_MPICXX in the public headers, so applications
can inadvertently wind up linking to the deprecated MPI C++ wrappers.
MPICH_SKIP_MPICXX and OMPI_SKIP_MPICXX have both been defined in H5public.h
so this should no longer be an issue. HDF5 makes no use of the deprecated
MPI C++ wrappers.
(DER - 2019/09/17, HDFFV-10893)
- fcntl(2)-based file locking incorrectly passed the lock argument struct
instead of a pointer to the struct, causing errors on systems where
flock(2) is not available.
File locking is used when files are opened to enforce SWMR semantics. A
lock operation takes place on all file opens unless the
HDF5_USE_FILE_LOCKING environment variable is set to the string \
"FALSE".
flock(2) is preferentially used, with fcntl(2) locks as a backup if
flock(2) is unavailable on a system (if neither is available, the lock
operation fails). On these systems, the file lock will often fail, which
causes HDF5 to not open the file and report an error.
This bug only affects POSIX systems. Win32 builds on Windows use a no-op
locking call which always succeeds. Systems which exhibit this bug will
have H5_HAVE_FCNTL defined but not H5_HAVE_FLOCK in the configure output.
This bug affects HDF5 1.10.0 through 1.10.5.
fcntl(2)-based file locking now correctly passes the struct pointer.
(DER - 2019/08/27, HDFFV-10892)
- Fixed a bug caused by a bad tag value when condensing object header
messages
There was an assertion failure when moving messages from running a
user test program with library release HDF5 1.10.4. It was because
the tag value (object header's address) was not set up when entering
the library routine H5O__chunk_update_idx(), which eventually
verifies the metadata tag value when protecting the object header.
The problem was fixed by replacing FUNC_ENTER_PACKAGE in \
H5O__chunk_update_idx()
with FUNC_ENTER_PACKAGE_TAG(oh->cache_info.addr) to set up the metadata tag.
(VC - 2019/08/23, HDFFV-10873)
- Fixed the test failure from test_metadata_read_retry_info() in
test/swmr.c
The test failure is due to an incorrect number of bins returned for
retry info (info.nbins). The # of bins expected for 101 read attempts
is 3 instead of 2. The routine H5F_set_retries() in src/H5Fint.c
calculates the # of bins by first obtaining the log10 value for
(read attempts - 1). For PGI/19, the log10 value for 100 read attempts
is 1.9999999999999998 instead of 2.00000. When casting the log10 value
to unsigned later on, the decimal part is chopped off causing the test
failure.
This was fixed by obtaining the rounded integer value (HDceil) for the
log10 value of read attempts first before casting the result to unsigned.
(VC - 2019/8/14, HDFFV-10813)
- Fixed an issue when creating a file with non-default file space info
together with library high bound setting to H5F_LIBVER_V18.
When setting non-default file space info in fcpl via
H5Pset_file_space_strategy() and then creating a file with both high and
low library bounds set to H5F_LIBVER_V18 in fapl, the library succeeds in
creating the file. File creation should fail because the feature of
setting non-default file space info does not exist in library release 1.8
or earlier.
This was fixed by setting and checking the proper version in the file
space info message based on the library low and high bounds when creating
and opening the HDF5 file.
(VC - 2019/6/25, HDFFV-10808)
- Fixed an issue where copying a version 1.8 dataset between files using
H5Ocopy fails due to an incompatible fill version
When using the HDF5 1.10.x H5Ocopy() API call to copy a version 1.8
dataset to a file created with both high and low library bounds set to
H5F_LIBVER_V18, the H5Ocopy() call will fail with the error stack indicating
that the fill value version is out of bounds.
This was fixed by changing the fill value message version to H5O_FILL_VERSION_3
(from H5O_FILL_VERSION_2) for H5F_LIBVER_V18.
(VC - 2019/6/14, HDFFV-10800)
- Fixed a bug that would cause an error or cause fill values to be
incorrectly read from a chunked dataset using the "single chunk" \
index if
the data was held in cache and there was no data on disk.
(NAF - 2019/03/06)
- Fixed a bug that could cause an error or cause fill values to be
incorrectly read from a dataset that was written to using H5Dwrite_chunk
if the dataset was not closed after writing.
(NAF - 2019/03/06, HDFFV-10716)
- Fixed memory leak in scale offset filter
In a special case where the MinBits is the same as the number of bits in
the datatype's precision, the filter's data buffer was not freed, causing
the memory usage to grow. In general the buffer was freed correctly. The
Minbits are the minimal number of bits to store the data values. Please
see the reference manual for H5Pset_scaleoffset for the details.
(RL - 2019/3/4, HDFFV-10705)
Configuration
-------------
- Correct option for default API version
CMake options for default API version are not mutually exclusive.
Change the multiple BOOL options to a single STRING option with the
strings; v16, v18, v110.
(ADB - 2019/08/12, HDFFV-10879)
Tools
-----
- h5repack was fixed to repack datasets with external storage
to other types of storage.
New test added to repack files and verify the correct data using h5diff.
(JS - 2019/09/25, HDFFV-10408)
(ADB - 2019/10/02, HDFFV-10918)
|
Log message:
Update to 1.10.2
Upstream changes:
HDF5 version 1.10.2 released on 2018-03-29
================================================================================
CONTENTS
- New Features
- Support for new platforms and languages
- Bug Fixes since HDF5-1.10.1
- Supported Platforms
- Tested Configuration Features Summary
- More Tested Platforms
- Known Problems
New Features
============
Configuration and Build Systems:
--------------------------------
- CMake builds
--------------
- Changed minimum CMake required version to 3.10.
This change removed the need to support a copy of the FindMPI.cmake module,
which has been removed, along with its subfolder in the config/cmake_ext_mod
location.
(ADB - 2018/03/09)
- Added pkg-config file generation
Added pkg-config file generation for the C, C++, HL, and HL C++ libraries.
In addition, builds on Linux will create h5cc, h5c++, h5hlcc, and h5hlc++ \
scripts in the bin
directory that use the pkg-config files. The scripts can be used to build \
HDF5 C and C++
applications (i.e, similar to the compiler scripts produced by the \
Autotools builds).
(ADB - 2018/03/08, HDFFV-4359)
- Refactored use of CMAKE_BUILD_TYPE for new variable, which understands
the type of generator in use.
Added new configuration macros to use new HDF_BUILD_TYPE variable. This
variable is set correctly for the type of generator being used for the build.
(ADB - 2018/01/08, HDFFV-10385, HDFFV-10296)
- Autotools builds
------------------
- Removed version-specific gcc/gfortran flags for version 4.0 (inclusive)
and earlier.
The config/gnu-flags file, which is sourced as a part of the configure
process, adds version-specific flags for use when building HDF5. Most of
these flags control warnings and do not affect the final product.
Flags for older versions of the compiler were consolidated into the
common flags section. Moving these flags simplifies maintenance of
the file.
The upshot of this is that building with ancient versions of gcc
(<= 4.0) will possibly no longer work without hand-hacking the file
to remove the flags not understood by that version of the compiler.
Nothing should change when building with gcc >= 4.1.
(DER - 2017/05/31, HDFFV-9937)
- -fno-omit-frame-pointer was added when building with debugging symbols
enabled.
Debugging symbols can be enabled independently of the overall build
mode in both the autotools and CMake. This allows (limited) debugging
of optimized code. Since many debuggers rely on the frame pointer,
we've disabled this optimization when debugging symbols are requested
(e.g.: via building with --enable-symbols).
(DER - 2017/05/31, HDFFV-10226)
Library:
--------
- Added an enumerated value to H5F_libver_t for H5Pset_libver_bounds().
Currently, the library defines two values for H5F_libver_t and supports
only two pairs of (low, high) combinations as derived from these values.
Thus the bounds setting via H5Pset_libver_bounds() is rather restricted.
Added an enumerated value (H5F_LIBVER_V18) to H5F_libver_t and
H5Pset_libver_bounds() now supports five pairs of (low, high) combinations
as derived from these values. This addition provides the user more
flexibility in setting bounds for object creation.
(VC - 2018/03/14)
- Added prefix option to VDS files.
Currently, VDS source files must be in the active directory to be
found by the virtual file. Adding the option of a prefix to be set
on the virtual file, using a data access property list (DAPL),
allows the source files to locate at an absolute or relative path
to the virtual file.
Private utility functions in H5D and H5L packages merged into single
function in H5F package.
New public APIs:
herr_t H5Pset_virtual_prefix(hid_t dapl_id, const char* prefix);
ssize_t H5Pget_virtual_prefix(hid_t dapl_id, char* prefix /*out*/, \
size_t size);
The prefix can also be set with an environment variable, HDF5_VDS_PREFIX.
(ADB - 2017/12/12, HDFFV-9724, HDFFV-10361)
- H5FDdriver_query() API call added to the C library.
This new library call allows the user to query a virtual file driver
(VFD) for the feature flags it supports (listed in H5FDpublic.h).
This can be useful to determine if a VFD supports SWMR, for example.
Note that some VFDs have feature flags that may only be present
after a file has been created or opened (e.g.: the core VFD will
have the H5FD_FEAT_POSIX_COMPAT_HANDLE flag set if the backing
store is switched on). Since the new API call queries a generic VFD
unassociated with a file, these flags will never be returned.
(DER - 2017/05/31, HDFFV-10215)
- H5FD_FEAT_DEFAULT_VFD_COMPATIBLE VFD feature flag added to the C library.
This new feature flag indicates that the VFD is compatible with the
default VFD. VFDs that set this flag create single files that follow
the canonical HDF5 file format.
(DER - 2017/05/31, HDFFV-10214)
- The H5I_REFERENCE value in the H5I_type_t enum (defined in H5Ipublic.h)
has been marked as deprectated.
This ID type value is not used in the C library. i.e.: There are no
hid_t values that are of ID type H5I_REFERENCE.
This enum value will be removed in a future major version of the library.
The code will remain unchanged in the HDF5 1.10.x releases and branches.
(DER - 2017/04/05, HDFFV-10252)
Parallel Library:
-----------------
- Enabled compression for parallel applications.
With this release parallel applications can create and write compressed
datasets (or the datasets with the filters such as Fletcher32 applied).
(EIP - 2018/03/29)
- Addressed slow file close on some Lustre file systems.
Slow file close has been reported on some Lustre file systems.
While the ultimate cause is not understood fully, the proximate
cause appears to be long delays in MPI_File_set_size() calls at
file close and flush.
To minimize this problem pending a definitive diagnosis and fix,
PHDF5 has been modified to avoid MPI_File_set_size() calls when
possible. This is done by comparing the library's EOA (End of
Allocation) with the file systems EOF, and skipping the
MPI_File_set_size() call if the two match.
(JRM - 2018/03/29)
- Optimized parallel open/location of the HDF5 super-block.
Previous releases of PHDF5 required all parallel ranks to
search for the HDF5 superblock signature when opening the
file. As this is accomplished more or less as a synchronous
operation, a large number of processes can experience a
slowdown in the file open due to filesystem contention.
As a first step in improving the startup/file-open performance,
we allow MPI rank 0 of the associated MPI communicator to locate
the base offset of the super-block and then broadcast that result
to the remaining ranks in the parallel group. Note that this
approach is utilized ONLY during file opens which employ the MPIO
file driver in HDF5 by previously having called H5Pset_fapl_mpio().
HDF5 parallel file operations which do not employ multiple ranks
e.g. specifiying MPI_COMM_SELF (whose MPI_Comm_size == 1)
as opposed to MPI_COMM_WORLD, will not be affected by this
optimization. Conversely, parallel file operations on subgroups
of MPI_COMM_WORLD are allowed to be run in parallel with each
subgroup operating as an independant collection of processes.
(RAW - 2017/10/10, HDFFV-10294)
- Added large (>2GB) MPI-IO transfers.
Previous releases of PHDF5 would fail when attempting to
read or write greater than 2GB of data in a single IO operation.
This issue stems principally from an MPI API whose definitions
utilize 32 bit integers to describe the number of data elements
and datatype that MPI should use to effect a data transfer.
Historically, HDF5 has invoked MPI-IO with the number of
elements in a contiguous buffer represented as the length
of that buffer in bytes.
Resolving the issue and thus enabling larger MPI-IO transfers
is accomplished first, by detecting when a user IO request would
exceed the 2GB limit as described above. Once a transfer request
is identified as requiring special handling, PHDF5 now creates a
derived datatype consisting of a vector of fixed sized blocks
which is in turn wrapped within a single MPI_Type_struct to
contain the vector and any remaining data. The newly created
datatype is then used in place of MPI_BYTE and can be used to
fulfill the original user request without encountering API
errors.
(RAW - 2017/09/10, HDFFV-8839)
C++ Library:
------------
- The following C++ API wrappers have been added to the C++ Library:
+ H5Lcreate_soft:
// Creates a soft link from link_name to target_name.
void link(const char *target_name, const char *link_name,...)
void link(const H5std_string& target_name,...)
+ H5Lcreate_hard:
// Creates a hard link from new_name to curr_name.
void link(const char *curr_name, const Group& new_loc,...)
void link(const H5std_string& curr_name, const Group& new_loc,...)
// Creates a hard link from new_name to curr_name in same location.
void link(const char *curr_name, const hid_t same_loc,...)
void link(const H5std_string& curr_name, const hid_t same_loc,...)
Note: previous version of H5Location::link will be deprecated.
+ H5Lcopy:
// Copy an object from a group of file to another.
void copyLink(const char *src_name, const Group& dst,...)
void copyLink(const H5std_string& src_name, const Group& dst,...)
// Copy an object from a group of file to the same location.
void copyLink(const char *src_name, const char *dst_name,...)
void copyLink(const H5std_string& src_name,...)
+ H5Lmove:
// Rename an object in a group or file to a new location.
void moveLink(const char* src_name, const Group& dst,...)
void moveLink(const H5std_string& src_name, const Group& dst,...)
// Rename an object in a group or file to the same location.
void moveLink(const char* src_name, const char* dst_name,...)
void moveLink(const H5std_string& src_name,...)
Note: previous version H5Location::move will be deprecated.
+ H5Ldelete:
// Removes the specified link from this location.
void unlink(const char *link_name,
const LinkAccPropList& lapl = LinkAccPropList::DEFAULT)
void unlink(const H5std_string& link_name,
const LinkAccPropList& lapl = LinkAccPropList::DEFAULT)
Note: additional parameter is added to previous H5Location::unlink.
+ H5Tencode and H5Tdecode:
// Creates a binary object description of this datatype.
void DataType::encode() - C API H5Tencode()
// Returns the decoded type from the binary object description.
DataType::decode() - C API H5Tdecode()
ArrayType::decode() - C API H5Tdecode()
CompType::decode() - C API H5Tdecode()
DataType::decode() - C API H5Tdecode()
EnumType::decode() - C API H5Tdecode()
FloatType::decode() - C API H5Tdecode()
IntType::decode() - C API H5Tdecode()
StrType::decode() - C API H5Tdecode()
VarLenType::decode() - C API H5Tdecode()
+ H5Lget_info:
// Returns the information of the named link.
H5L_info_t getLinkInfo(const H5std_string& link_name,...)
(BMR - 2018/03/11, HDFFV-10149)
- Added class LinkCreatPropList for link create property list.
(BMR - 2018/03/11, HDFFV-10149)
- Added overloaded functions H5Location::createGroup to take a link
creation property list.
Group createGroup(const char* name, const LinkCreatPropList& lcpl)
Group createGroup(const H5std_string& name, const \
LinkCreatPropList& lcpl)
(BMR - 2018/03/11, HDFFV-10149)
- A document is added to the HDF5 C++ API Reference Manual to show the
mapping from a C API to C++ wrappers. It can be found from the main
page of the C++ API Reference Manual.
(BMR - 2017/10/17, HDFFV-10151)
Java Library:
----------------
- Wrapper added for enabling the error stack.
H5error_off would disable the error stack reporting. In order
to re-enable the reporting, the error stack info needs to be
saved so that H5error_on can revert state.
(ADB - 2018/03/13, HDFFV-10412)
- Wrappers were added for the following C APIs:
H5Pset_evict_on_close
H5Pget_evict_on_close
H5Pset_chunk_opts
H5Pget_chunk_opts
H5Pset_efile_prefix
H5Pget_efile_prefix
H5Pset_virtual_prefix
H5Pget_virtual_prefix
(ADB - 2017/12/20)
- The H5I_REFERENCE value in the H5I_type_t enum (defined in H5Ipublic.h)
has been marked as deprectated.
JNI code which refers to this value will be removed in a future
major version of the library. The code will remain unchanged in the
1.10.x releases and branches.
See the C library section, above, for further information.
(HDFFV-10252, DER, 2017/04/05)
Tools:
------
- h5diff has a new option to display error stack.
Updated h5diff with the --enable-error-stack argument, which
enables the display of the hdf5 error stack. This completes the
improvement to the main tools: h5copy, h5diff, h5dump, h5ls and
h5repack.
(ADB - 2017/08/30, HDFFV-9774)
Support for new platforms, languages and compilers.
=======================================
- None
|