ncpx : NetCDF Parallel Extractor

1. Overview
2. Download
3. Installation
4. Examples
5. Usage
6. Bugs
7. Wishlist
8. References
9. News

Overview

Extract and append variables in parallel from a NetCDF file via MPI. Uses combine-batched asynchronous I/O with Parallel-NetCDF 1.2. Arena memory is managed to prevent swapping and fragmentation. Allows header padding, variable alignment and 64bit format. Command-line syntax mimics NetCDF Operators (NCO), so "ncpx" can be swapped in to replace "ncks" (although "ncks" is still a powerful, general purpose tool that can't be entirely replaced!).

Download

Source Code: ncpx-0.1.1.tgz

Installation

1. Download ncpx.
2. Decompress with

   tar -zxvf ncpx-*gz; cd ncpx-*/src
   

3. make
4. make test TMPDIR=/tmp
5. make bench TMPDIR=/tmp MAKEDATA=1
6. make install PREFIX=/usr/bin
7. make clean

Examples

###############################################################################
# Extract two variables, without their coordinate variables, overwriting output, 
# using a header pad, run by 2 PEs.
mpirun -np 2 ncpx -O --nocoords --header_pad 127000 -v var1,var2 test1.nc tmp1.nc

###############################################################################
# Append two variables along with their coordinate variables.
mpirun -np 2 ncpx -A -v var3,var4 2test1.nc tmp1.nc

Usage

ncpx [-A|--apn|--append] [-C|--nocoords] [-D] [--hdr_pad|--header_pad N]
     [--help] [-O|--ovr|--overwrite] -v v1[,...] [-V|--version] in.nc out.nc

-A, --apn, --append      Append to existing output file, if any
-C, --nocoords           Associated coordinate variables should not be processed
-D                       Enable debug messages.
--hdr_pad, --header_pad N
                         Allows extra header space like NC_HEADER_ALIGN_SIZE.
--help                   Print this usage message.
-O, --ovr, --overwrite   Overwrite existing output file, if any
-T, --time               Print elapsed runtime statistics
-v, --variable v1[,...]  Variable(s) to process
-V, --version            Print version..
in.nc                    Input file name
out.nc                   Output file name

Environment Variables:
NC_HEADER_ALIGN_SIZE     Allows some extra space between the end of the header
                         describing the entire file and the first variable.
                         Takes precendence over "--hdr_pad".

NC_VAR_ALIGN_SIZE        Aligning the start of a variable to a block boundary
                         can often eliminate all unaligned file system accesses.

Bugs

Wishlist

References

1. NetCDF Data Format
2. Parallel NetCDF Wiki

News

Version 0.1.1 (10/10/2011)

* Bug fix; creates new file in append mode if no output file exists for appending.
  Thanks to Jeff Durachta for discovering.

Version 0.1.0 (1/2/2011)

* Internal enhancements: added striding, indexing, tuning, task management.

Version 0.0.0 (12/19/2010)

* Initial release.