File libluminate/illumulti.c

RCS Header

This file contains the functions IlluMultiSave(), IlluMultiLoad() and IlluMultiRead() designed to handle distributed storage and retrieval of data on local drives of machines in a Beowulf cluster. This should allow rapid loading of timestep data for "playback" from what is essentially a giant RAID-0 array of distributed disks, at enormously higher speeds than via NFS from a hard drive or RAID array on the head node. The danger of course is that if one node's disk goes down, you don't have a valid data set any more, but that's the nature of RAID-0, right?

The filenames saved are:

If one were saving timesteps, one might include a timestep number in the basename, and also timestep and simulation time in the metadata. The metadata can also hold simulation parameters, etc.

This supports 1-D, 2-D and 3-D distributed arrays. As an extra feature, you can load a multi-CPU distributed array scattered over lots of files into a single CPU, to facilitate certain modes of data visualization.


Included Files


Preprocessor definitions

#define ORBIT2 1

#define _REENTRANT 1

#define IMLIB2_EXISTS 1

#define DPRINTF( fmt, args... )

#define __FUNCT__ "IlluMultiParseXML"

#define __FUNCT__ "IlluMultiParseData"

#define __FUNCT__ "IlluMultiStoreXML"

#define __FUNCT__ "IlluMultiStoreData"

#define __FUNCT__ "checkagree"

#define __FUNCT__ "IlluMultiRead"

#define __FUNCT__ "IlluMultiLoad"

#define __FUNCT__ "IlluMultiSave"


Global Function IlluMultiLoad()

This creates a new distributed array of the appropriate size and loads the data into the vector contained in it (as retrieved by DAGetVector()). It also reads the user metadata parameters into arrays stored at the supplied pointers.

int IlluMultiLoad ( MPI_Comm comm, char* basename, DA* theda, PetscScalar* wx, PetscScalar* wy, PetscScalar* wz, field_plot_type** fieldtypes, int* usermetacount, char*** usermetanames, char*** usermetadata )

int IlluMultiLoad
It returns zero or an error code.
MPI_Comm comm
MPI communicator, if NULL it uses PETSC_COMM_WORLD.
char* basename
Base file name.
DA* theda
Pointer to a DA object (to be created by this function).
PetscScalar* wx
Physical overall width in the x-direction.
PetscScalar* wy
Physical overall width in the y-direction.
PetscScalar* wz
Physical overall width in the z-direction.
field_plot_type** fieldtypes
Data (plot) types for field variables.
int* usermetacount
Pointer to an int where we put the number of user metadata parameters loaded.
char*** usermetanames
Pointer to a char ** where the loaded parameter names are stored. This is malloced by this function, so a call to free() is needed to free up its data.
char*** usermetadata
Pointer to a char ** where the loaded parameter strings are stored. This is malloced by this function, so a call to free() is needed to free up its data.
First it gets the parameters from the XML file.
Next it creates a distributed array based on those parameters, and sets the names of its fields.
Then it streams the data into the distributed array's vector in one big slurp.


Global Function IlluMultiRead()

This reads the data into an existing distributed array and vector, checking that the sizes are right etc.

int IlluMultiRead ( MPI_Comm comm, DA theda, Vec X, char* basename, int* usermetacount, char*** usermetanames, char*** usermetadata )

int IlluMultiRead
It returns zero or an error code.
MPI_Comm comm
MPI communicator, if NULL it uses PETSC_COMM_WORLD.
DA theda
Distributed array object controlling the data to read.
Vec X
Vector into which to read the data.
char* basename
Base file name.
int* usermetacount
Pointer to an int where we put the number of user metadata parameters loaded.
char*** usermetanames
Pointer to a char ** where the loaded parameter names are stored. This is malloced by this function, so a call to free() is needed to free up its data.
char*** usermetadata
Pointer to a char ** where the loaded parameter strings are stored. This is malloced by this function, so a call to free() is needed to free up its data.
First it gets the properties of the distributed array for comparison with the metadata.
Next it parses the XML metadata file into the document tree, and reads its content into the appropriate structures, comparing parameters with those of the existing distributed array structure.
Then it streams in the data in one big slurp.


Global Function IlluMultiSave()

This saves the vector X in multiple files, two per process.

int IlluMultiSave ( MPI_Comm comm, DA theda, Vec X, char* basename, PetscScalar wx, PetscScalar wy, PetscScalar wz, field_plot_type* fieldtypes, int usermetacount, char** usermetanames, char** usermetadata, int compressed )

int IlluMultiSave
it returns zero or an error code.
MPI_Comm comm
MPI communicator, if NULL it uses PETSC_COMM_WORLD.
DA theda
Distributed array object controlling data saved.
Vec X
Vector whose data are actually being saved.
char* basename
Base file name.
PetscScalar wx
 
PetscScalar wy
 
PetscScalar wz
 
field_plot_type* fieldtypes
 
int usermetacount
Number of user metadata parameters.
char** usermetanames
User metadata parameter names.
char** usermetadata
User metadata parameter strings.
int compressed
Data compression: if zero then no compression (fastest), 1-9 then gzip compression level, 10-15 then gzip --best. If 16-31 then save guint32s representing relative values between min and max for each field, compressed according to this value minus 16. Likewise for 32-47 and guint16s, and 48-63 for guint8s. Yes, these alternative formats lose information and can't be used for accurate checkpointing, but they should retain enough data for visualization (except perhaps for the guint8s, which are possibly acceptable for vectors but certainly not contours).
First a check to verify a supported value of compressed, but no fancy guint* compression for complex!
Then get the distributed array parameters and processor number, and store all this data in the XML .meta file.
Finally, the data just stream out to the data file or gzip pipe in one big lump.


Local Function IlluMultiParseData()

This function reads in the data stored by IlluMultiStoreData(), complete with int/gzip compression.

static int IlluMultiParseData ( MPI_Comm comm, PetscScalar* globalarray, char* basename, int rank, int compressed, int gridpoints, int dof, int wrongendian, PetscScalar* fieldmin, PetscScalar* fieldmax )

int IlluMultiParseData
It returns zero or an error code.
MPI_Comm comm
MPI communicator, if NULL it uses PETSC_COMM_WORLD.
PetscScalar* globalarray
Array into which to load the (local) data.
char* basename
Base file name.
int rank
CPU number to read data for.
int compressed
Data compression: if zero then no compression (fastest), 1-9 then gzip compression level, 10-15 then gzip --best. If 16-31 then save guint32s representing relative values between min and max for each field, compressed according to this value minus 16. Likewise for 32-47 and guint16s, and 48-63 for guint8s. Yes, these alternative formats lose information and can't be used for accurate checkpointing, but they should retain enough data for visualization (except perhaps for the guint8s, which are possibly acceptable for vectors but likely not contours).
int gridpoints
Number of gridpoints to read data for.
int dof
Degrees of freedom at each node, a.k.a. number of field variables.
int wrongendian
Tells whether the data are stored in the opposite endian format from this platform, and thus must be switched when the data are streamed in.
PetscScalar* fieldmin
Minimum value of each field variable.
PetscScalar* fieldmax
Maximum value of each field variable.

Local Function IlluMultiParseXML()

This function reads in the XML metadata document and returns the various parameter values in the addresses pointed to by the arguments. It is called by IlluMultiLoad() and IlluMultiRead().

static int IlluMultiParseXML ( MPI_Comm comm, char* basename, int rank, int* compressed, int* wrongendian, int* dim, int* px, int* py, int* pz, int* nx, int* ny, int* nz, PetscScalar* wx, PetscScalar* wy, PetscScalar* wz, int* xm, int* ym, int* zm, int* dof, int* sw, DAStencilType* st, DAPeriodicType* wrap, char*** fieldnames, field_plot_type** fieldtypes, PetscScalar** fieldmin, PetscScalar** fieldmax, int* usermetacount, char*** usermetanames, char*** usermetadata )

int IlluMultiParseXML
It returns zero or an error code.
MPI_Comm comm
MPI communicator, if NULL it uses PETSC_COMM_WORLD.
char* basename
Base file name.
int rank
CPU number to read data for.
int* compressed
Data compression: if zero then no compression (fastest), 1-9 then gzip compression level, 10-15 then gzip --best. If 16-31 then save guint32s representing relative values between min and max for each field, compressed according to this value minus 16. Likewise for 32-47 and guint16s, and 48-63 for guint8s. Yes, these alternative formats lose information and can't be used for accurate checkpointing, but they should retain enough data for visualization (except perhaps for the guint8s, which are possibly acceptable for vectors but likely not contours).
int* wrongendian
Tells whether the data are stored in the opposite endian format from this platform, and thus must be switched when the data are streamed in.
int* dim
Dimensionality of the space.
int* px
Number of processors in the x-direction.
int* py
Number of processors in the y-direction.
int* pz
Number of processors in the z-direction.
int* nx
Number of grid points over the entire array in the x-direction.
int* ny
Number of grid points over the entire array in the y-direction.
int* nz
Number of grid points over the entire array in the z-direction.
PetscScalar* wx
Physical overall width in the x-direction, PETSC_NULL if not needed.
PetscScalar* wy
Physical overall width in the y-direction, PETSC_NULL if not needed.
PetscScalar* wz
Physical overall width in the z-direction, PETSC_NULL if not needed.
int* xm
Number of grid points over the local part of the array in the x-direction.
int* ym
Number of grid points over the local part of the array in the y-direction.
int* zm
Number of grid points over the local part of the array in the z-direction.
int* dof
Degrees of freedom at each node, a.k.a. number of field variables.
int* sw
Stencil width.
DAStencilType* st
Stencil type, given by the PETSc enum values.
DAPeriodicType* wrap
Periodic type, given by the PETSc enum values.
char*** fieldnames
Names of the field variables.
field_plot_type** fieldtypes
Data (plot) types for field variables, PETSC_NULL if not needed.
PetscScalar** fieldmin
Minimum value of each field variable.
PetscScalar** fieldmax
Maximum value of each field variable.
int* usermetacount
Number of user metadata parameters.
char*** usermetanames
User metadata parameter names.
char*** usermetadata
User metadata parameter strings.
For GlobalSize, since there's no *size attribute (for an 0.1 version document), assume 1.
If the type attribute is missing from the Field node (as it is in version 0.1 documents), assume FIELD_SCALAR.


Local Function IlluMultiStoreData()

This function stores the data file.

static int IlluMultiStoreData ( MPI_Comm comm, PetscScalar* globalarray, char* basename, int rank, int compressed, int gridpoints, int dof, PetscScalar* fieldmin, PetscScalar* fieldmax )

int IlluMultiStoreData
It returns zero or an error code.
MPI_Comm comm
MPI communicator, if NULL it uses PETSC_COMM_WORLD.
PetscScalar* globalarray
Array from which to save the (local) data.
char* basename
Base file name.
int rank
CPU number to read data for.
int compressed
Data compression: if zero then no compression (fastest), 1-9 then gzip compression level, 10-15 then gzip --best. If 16-31 then save guint32s representing relative values between min and max for each field, compressed according to this value minus 16. Likewise for 32-47 and guint16s, and 48-63 for guint8s. Yes, these alternative formats lose information and can't be used for accurate checkpointing, but they should retain enough data for visualization (except perhaps for the guint8s, which are possibly acceptable for vectors but likely not contours).
int gridpoints
Number of gridpoints to store data for.
int dof
Degrees of freedom at each node, a.k.a. number of field variables.
PetscScalar* fieldmin
Minimum value of each field variable.
PetscScalar* fieldmax
Maximum value of each field variable.

Local Function IlluMultiStoreXML()

This function opens, stores and closes the XML metadata file for IlluMulti format storage. It is called by IlluMultiSave().

static int IlluMultiStoreXML ( MPI_Comm comm, char* basename, int rank, int compressed, int dim, int px, int py, int pz, int nx, int ny, int nz, PetscScalar wx, PetscScalar wy, PetscScalar wz, int xm, int ym, int zm, int dof, int sw, int st, int wrap, char** fieldnames, field_plot_type* fieldtypes, PetscReal* fieldmin, PetscReal* fieldmax, int usermetacount, char** usermetanames, char** usermetadata )

int IlluMultiStoreXML
It returns zero or an error code.
MPI_Comm comm
MPI communicator, if NULL it uses PETSC_COMM_WORLD.
char* basename
Base file name.
int rank
CPU number to store data for.
int compressed
Data compression: if zero then no compression (fastest), 1-9 then gzip compression level, 10-15 then gzip --best. If 16-31 then save guint32s representing relative values between min and max for each field, compressed according to this value minus 16. Likewise for 32-47 and guint16s, and 48-63 for guint8s. Yes, these alternative formats lose information and can't be used for accurate checkpointing, but they should retain enough data for visualization (except perhaps for the guint8s, which are possibly acceptable for vectors but certainly not contours).
int dim
Dimensionality of the space.
int px
Number of processors in the x-direction.
int py
Number of processors in the y-direction.
int pz
Number of processors in the z-direction.
int nx
Number of grid points over the entire array in the x-direction.
int ny
Number of grid points over the entire array in the y-direction.
int nz
Number of grid points over the entire array in the z-direction.
PetscScalar wx
Physical overall width in the x-direction.
PetscScalar wy
Physical overall width in the y-direction.
PetscScalar wz
Physical overall width in the z-direction.
int xm
Number of grid points over the local part of the array in the x-direction.
int ym
Number of grid points over the local part of the array in the y-direction.
int zm
Number of grid points over the local part of the array in the z-direction.
int dof
Degrees of freedom at each node, a.k.a. number of field variables.
int sw
Stencil width.
int st
Stencil type, given by the PETSc enum values.
int wrap
Periodic type, given by the PETSc enum values.
char** fieldnames
Names of the field variables.
field_plot_type* fieldtypes
Data (plot) types for field variables.
PetscReal* fieldmin
Minimum value of each field variable.
PetscReal* fieldmax
Maximum value of each field variable.
int usermetacount
Number of user metadata parameters.
char** usermetanames
User metadata parameter names.
char** usermetadata
User metadata parameter strings.
The XML tags in the .meta file consist of:






IlluMultiPrimary tag, with attributes version,
endian (big or little) and compression
(none, 1-9, best; long*, short* or char*)
GlobalCPUsNumber of CPUs in each direction, with
attributes dimensions, xwidth, ywidth and zwidth
GlobalSizeSize of the entire distributed array, with
attributes xwidth, ywidth, zwidth, xsize**, ysize**, zsize** and fields
LocalSizeSize of the local part of the array, with
attributes xwidth, ywidth and zwidth
StencilStencil and periodic data, with attributes
width, type and periodic (using PETSc enum values)
FieldData on each field, attributes name, type*, min and max
UserUser parameters, attributes name and value
*Lossy compression to smaller data types.
**Represents new attribute for IlluMulti 0.2 file format.


Local Function checkagree()

Ancillary routine for IlluMultiRead(): checks agreement of parameters and reports disagreement if necessary.

static inline int checkagree ( MPI_Comm comm, int da, int file, char* parameter )

int checkagree
Returns 0 if they agree, 1 otherwise.
MPI_Comm comm
MPI communicator, if NULL it uses PETSC_COMM_WORLD.
int da
Integer parameter from the existing DA.
int file
Integer parameter read from the file.
char* parameter
Parameter name for reporting.