SourceForge.net Logo

ncpx : NetCDF Parallel Extractor

  1. Overview
  2. Download
  3. Installation
  4. Examples
  5. Usage
  6. Bugs
  7. Wishlist
  8. References
  9. News


Overview

Extract and append variables in parallel from a NetCDF file via MPI. Uses combine-batched asynchronous I/O with Parallel-NetCDF 1.2. Arena memory is managed to prevent swapping and fragmentation. Allows header padding, variable alignment and 64bit format. Command-line syntax mimics NetCDF Operators (NCO), so "ncpx" can be swapped in to replace "ncks" (although "ncks" is still a powerful, general purpose tool that can't be entirely replaced!).


Download

Source Code: ncpx-0.1.1.tgz


Installation

  1. Download ncpx.
  2. Decompress with
    tar -zxvf ncpx-*gz; cd ncpx-*/src
    
  3. make
  4. make test TMPDIR=/tmp
  5. make bench TMPDIR=/tmp MAKEDATA=1
  6. make install PREFIX=/usr/bin
  7. make clean


Examples

###############################################################################
# Extract two variables, without their coordinate variables, overwriting output, 
# using a header pad, run by 2 PEs.
mpirun -np 2 ncpx -O --nocoords --header_pad 127000 -v var1,var2 test1.nc tmp1.nc

###############################################################################
# Append two variables along with their coordinate variables.
mpirun -np 2 ncpx -A -v var3,var4 2test1.nc tmp1.nc


Usage

ncpx [-A|--apn|--append] [-C|--nocoords] [-D] [--hdr_pad|--header_pad N]
     [--help] [-O|--ovr|--overwrite] -v v1[,...] [-V|--version] in.nc out.nc

-A, --apn, --append      Append to existing output file, if any
-C, --nocoords           Associated coordinate variables should not be processed
-D                       Enable debug messages.
--hdr_pad, --header_pad N
                         Allows extra header space like NC_HEADER_ALIGN_SIZE.
--help                   Print this usage message.
-O, --ovr, --overwrite   Overwrite existing output file, if any
-T, --time               Print elapsed runtime statistics
-v, --variable v1[,...]  Variable(s) to process
-V, --version            Print version..
in.nc                    Input file name
out.nc                   Output file name

Environment Variables:
NC_HEADER_ALIGN_SIZE     Allows some extra space between the end of the header
                         describing the entire file and the first variable.
                         Takes precendence over "--hdr_pad".

NC_VAR_ALIGN_SIZE        Aligning the start of a variable to a block boundary
                         can often eliminate all unaligned file system accesses.


Bugs


Wishlist


References

  1. NetCDF Data Format
  2. Parallel NetCDF Wiki


News

Version 0.1.1 (10/10/2011)

* Bug fix; creates new file in append mode if no output file exists for appending.
  Thanks to Jeff Durachta for discovering.

Version 0.1.0 (1/2/2011)

* Internal enhancements: added striding, indexing, tuning, task management.

Version 0.0.0 (12/19/2010)

* Initial release.


Copyright (C) 2010,2011 Remik Ziemlinski