News
Download
Documentation
Instruments
About
Author: Attila Kovacs <attila[AT]sigmyne.com>
Last updated: 23 Sep 2018
This document contains information specific to using CRUSH-2 with a SOFIA instrument in general, without discussing the specific use for the particular instrument per se. Separate READMEs cover the particulars of each instrument.
You may use this as a standalone guide, with instructions for installation and basic use, or in conjucntion with the main CRUSH-2 README. Either way, it is recommended that you also familiarize yourself with the contents of the main CRUSH-2 README and its Section 1 (Getting Started) especially.
CRUSH is a reduction pipeline, designed mainly to remove correlated signals (correlated noise) in the detector time-streams to arrive at clean & independent bolometer signals, which are then used to derive a model of the source (usually an image).
As such it is not an interactive reduction software (e.g. as opposed to e.g. BoA). The term scripting in CRUSH mainly means defining configuration options (in the command line or through configuration files) which are parsed in the order they are read.
During the reduction CRUSH aims to arrive at a consistent set of solutions for various correlated signals, the corresponding gains and the pixel weights, as well as tries to identify and flag problematic data, and applies appropriate spectral filtering.
This means a series of reduction steps, which are iterated a few times until the required self-consistent solutions are arrived at.
To learn more about the details please refer to Kovacs, A., “CRUSH: fast and scalable data reduction for imaging arrays,” Proc. SPIE 7020, 45, (2008). If that does not satisfy your curiousity, then you can find yet more explanation in Kovacs, A., PhD thesis, Caltech (2006).
Install Java (if necessary), e.g. from www.java.com. If you already have Java, check that it is version 1.8.0 (a.k.a. Java 8) or later, by typing:
> java -versionNote, that The GNU java a.k.a. gij (default on some older RedHat and Fedora systems) is painfully slow and unreliable, and will not always run CRUSH correctly. If you need Java, you can download the latest JRE from
Unpack the tarball in the desired location (e.g. under ~/astrotools/):
> cd ~/astrotools
> tar xzf crush-2.xx-x.tar.gzVerify that CRUSH works:
> cd crush
> ./crushYou should see a brief information screen on how to use CRUSH.
To create system-wide access to the crush executables, you may optionally wish to run install.sh (as root or with sudo). It will link the executables to /usr/bin, and install man pages.
> cd crush
> sudo bash install.shYou can check the success of the above optional step by typing:
> man crushIf all is in order, you should see the UNIX manual page on the crush.
CRUSH ships with a default Java configuration. On Windows and the most common UNIX platforms (Linux, Mac OS X, BSD, and Solaris), it will automatically attempt to set an optimal configuration. On other platforms, it comes with fail-safe default values (default java, 32-bit mode and 1GB of RAM use).
You can override the defaults by placing your settings in arbitrary files under /etc/crush2/startup or ~/.crush2/startup (for the equivalent configuration under Windows, please refer to README.windows)
(Any settings in the user’s home under ~/.crush2/startup will override the system-wide values in /etc/crush2/startup or C:\Program Data\startup. If multiple config files exist in the same location, these will be parsed by the shell in non-specific order).
E.g., placing the following lines in ~/.crush2/startup/java.conf overrides all available settings:
JAVA="/usr/java/latest/bin/java"
USEMB="4000"
JVM="server"
EXTRAOPTS="-Djava.awt.headless=true"Upon startup CRUSH will find and apply these settings, so it will use /usr/java/latest/bin/java to run CRUSH with 4GB of RAM, using the HotSpot server VM, and in headless mode (without display, mouse or keyboard).
Below is a guide to the variables that you can override to set your own Java runtime configuration:
JAVA Set to the location of the Java executable you want to use.
E.g. `java` to use the default Java, or
`/usr/java/latest/bin/java` to use the latest from Oracle or
OpenJDK.
USEMB Set to the maximum amount of RAM (in MB) available to
CRUSH. E.g. "4000" for 4GB. On a 32-bit OS, or with a 32-bit
Java installation no more than 2GB of RAM may be accesssed.
In practice, the maximum 32-bit OS/Java values range from
around "1000" (32-bit Windows / Java) to "1900" (32-bit
Linux / Java).
JVM Usually set to `server` for Oracle or OpenJDK. If using
IBM's Java, set it to "" (empty string). On ARM platforms,
you probably get better performance using `jamvm` or
`avian`. o see what VM options are available, run
`java -help`. The VM options are listed near the top of the
resulting help screen.
EXTRAOPTS Any other non-standard options you may want to pass to the
Java VM should go here. Typically set to "".You can also specify environment variables, and add shell commands (bash), since these configuration files are in fact sourced as bash scripts before launching Java / CRUSH. For example you can add:
CRUSH_NO_UPDATE_CHECK="1"
CRUSH_NO_VM_CHECK="1"
echo "Will try to parse my own configuration now... "
if [ -f ~/mycrushconfig.sh ] ; then
echo -n "OK"
source ~/mycrushconfig.sh
else
echo -n "Not found"
fiThe above will disable update checking (not recommended!) and VM checking (also not recommended!) and will source the contents of ~/mycrushconfig.sh if and when such a file exists.
The default method of locating files is by file name, which may specify either an absolute path, e.g.:
> crush hawc+ /data/2016-12-03_HA_F354_009_CAL_unk_HAWD_HWPOpen_RAW.fitsor it can be filename/path relative to datapath
> crush hawc+ 2016-12-03_HA_F354_009_CAL_unk_HAWD_HWPOpen_RAW.fitsThe two are equilalent assuming that datapath is set to /data/hirmes in the second case, e.g. in the user configuration file ~/.crush2/hirmes/default.cfg, or on the command-line.
Often the simpler way of locating input files is by a combination of flight and scan numbers. This is often shorter, and allows to specify multiple scans and ranges with more ease.
Scan lookup by flight and scan number requires you to set datapath to point to the data directory. E.g., by placing the line in the user configuration for HAWC+ (E.g. ~/.crush2/hawc+/default.cfg):
datapath /data/hawc+Now, you may simply reduce scan 9 from flight 354 as:
> crush hawc+ -flight=354 9You can also reduce multiple scans, from multiple flight together. E.g.:
> crush hawc+ -flight=354 104-105 129 -flight=362 13 16 33-35The above will co-reduce 3 scans (104, 105, 129) from flight 354 with 5 scans (13, 16, 33, 34, 35) from flight 362.
Note. Locating scans by flight and scan number may not be possible for simulated data, or data that has been renamed from it’s original file name.
You may specify reduction options as command-line arguments to CRUSH. Each option in the line begins with a dash -, and should not contain white spaces, unless these are enclosed in quotes. E.g. below is a valid option line, with three options, one of which contains a white-space characters:
crush hawc+ -faint -name=“My HAWC+ image.fits” -projection=TAN …
Below is a brief guide to a few useful options, grouped by functionality.
The default reduction is generally OK for sources smaller than the field of view (<FoV/2) with S/N in the 10–1000 range. Outside this range, the following options can be used to obtain better results:
-bright Reduce very bright sources (S/N > 1000).
-faint Reduce faint sources (S/N < 30).
-deep Optimized for the extraction of deep-field point
sources. It is similar to `-faint` (above) but it also
spatially filters the map, removing all signals above a
few beam scales. Use only for very faint point sources.
-extended Reduce extended sources (>FoV/2). The retention of
scales larger than ~FoV/2 come at a price of increased
map noise (on those scales). See Section 2.4 of the
main README on the 'Recovery of Extended Emission'. Can
be used on its own (assuming the default brightness
setting) or together with `-faint` or `-bright`.
-sourcesize=X Can be used together with `-extended` (above), to
tweak just how extended the source is expected to be.
The argument is an approximate source diameter in
arcseconds. E.g. `-sourcesize=300.0`.
-source.sign=Z By default, CRUSH assumes that sources are seen in
emission (+), and biases the reduction as such to
produce images without filter bowls or other negative
reduction artifatcs. However, your object may contain
absorption features, for which you may want ot bias in
the opposite direction (-), or not bias at all (0).
Set the sign accordingly to +, -, or 0. E.g.
`-source.sign=+`These options change how the output FITS image will be constructed. All these options have reasonable default values, and so you should only use them to override those if needed.
-grid=X Set the pixelization of the map to X arcseconds. (The
default pixelization is chosen to be be around 1/5th of
a beam for each HAWC+ band).
-projection=XXX Change the WCS spherical projection. The following
projections are supported:
SFL -- Sanson-Flamsteed
SIN -- Slant Orthographic
TAN -- Gnomonic
ZEA -- Zenithal Equal Area
MER -- Mercator
CAR -- Plate-Carree
AIT -- Hammer-Aitoff
GLS -- Global Sinusoidal
STG -- Stereographic
ARC -- Zenithal Equidistant
The default is SFL (Sanson-Flamsteed). E.g.
`-projection=TAN`.
-ecliptic Produce maps in ecliptic coordinates, instead of the
default equatorial (same as `-system=ecliptic`).
-galactic Produce maps in galactic coordinates, instead of the
default equatorial (same as `-system=galactic`).
-gzip Compress the output (e.g. FITS) with GZIP, if possible.
(The output file name will have .gz extension added as
needed.)
-supergalactic Produce maps in super-galactic coordinates, instead of
the default equatorial (same as
`-system=supergalactic`).
-horizontal Produce maps in horizontal coordinates, instead of
the default equatorial (same as `-system=horizontal`).
-focalplane Produce maps in focal-plane coordinates, instead of the
default equatorial (same as `-system=focalplane`).
-final:smooth=X Smooth the final map by a Gaussian with X arcsec FWHM.
Alternatively, X can be one of `minimal`, `halfbeam`
`2/3beam` or `beam`. E.g. `-smooth=8.0` or
`-smooth=2/3beam`. To turn smoothing off completely
use `-final:forget=smooth`.From this point on, the documentation is of more technical nature, intended for expert users only.
atran.amcoeffs=<list> A list of comma-separated polynomial
coefficients for the airmass correction term in the
Vacca et al. ATRAN-based static atmospheric model, for
the polynomial expansion around 41 kft altitude and 45
deg elevation.
@See: 'atran.altcoeffs', 'atran.reference'
atran.altcoeffs=<list> A lust of comma-separated polynomial
coefficients for the altutide correction term in the
Vacca et al. ATRAN-based static atmospheric model, for
the polynomial expansion around 41 kft altitude and 45
deg elevation.
@See 'atran.amcoeffs', 'atran.reference'
atran.reference=X The typical transmission at the reference point of 41
kft altitude and 45 deg elevation for the Vacca et al.
ATRAN-based static atmospheric model. Together with
'atran.amcoeffs', 'atran.altcoeffs', this defines
a standard atmospheric correction model for SOFIA.
The option 'tau=atran' activates this model as the
one used for calculating atmospheric corrections in
CRUSH.
@See: 'tau', 'atran.amcoeffs', 'atran.altcoeffs'
calibrated Set this option when the reduction includes the final
calibration (opacities and other calibration
corrections) to produce Level 3 data. Otherwise, CRUSH
will produce Level 2 output.
flight=N Set the flight number for locating data files using
scan numbers only. Remember to set 'datapath' also to
specify the folder where scan files are stored.
@See: 'datapath'
organization Specify the organization at which CRUSH is being used
for reducing data. The value of this option is stored
directly in the FITS ORIGIN header key as required by
the DCS. If you want the ORIGIN key to be set properly
you might consider adding the organization option to
`~/.crush2/sofia/default.cfg`, e.g. as:
organization SOFIA Science and Mission Ops
pwv41k=X For a simple static atmoshpere model, set the typical
precipitable water vapor (PWV) level to X microns at
41 kft altitude.
Based on it, and on 'pwvscale', CRUSH will calculate
and appropriate PWV value for each scan. The PWV
value can then be converted into a typical zenith tau
value, provided the 'tau.pwv.a' and 'tau.pwv.b' scaling
parameters are defined. One may then select this tau
value to use for opacity corrections via the 'tau=pwv'
option.
@See: 'pwvscale', 'tau.pwv.a', 'tau.pwv.b', 'tau'
pwvscale=X Set the scale height (kft) for the PWV, locally around
the reference value at 41 kft (see 'pwv41k').
@See: 'pwv41k'
tau=atran Selects the static ATRAN-based atmospheric model by
Bill Vacca for the opacity corrections.
@See: 'atran.amcoeffs', 'atran.altcoeffs'
'atran.reference'
tau=pwv Selects the built-in static atmospheric model of CRUSH
for the opacity corrections.
@See: 'pwv41k', 'pwvscale', 'tau.pwv.a', 'tau.pwv.b'ac.airspeed (km/h) Airspeed.
ac.alt (m) Altitude.
ac.altkft (kft) Altitude.
ac.dir (deg) Direction of heading.
ac.gndspeed (km/h) Groundspeed
ac.lat (rad) Geodetic latitude
ac.latd (deg) Geodetic latitude
ac.lon (ra) Geodetic longitude
ac.lond (deg) Geodetic longitude
ac.trkangle (rad) Tracking angle
array.sibsx (pix) Instrument boresight pixel in x.
array.sibsy (pix) Instrument boresight pixel in y.
chop.amp (arcsec) Chop amplitude
chop.angle (deg) Chop angle
chop.flag 'C' if chopping or '-' if not
chop.frequency (Hz) Chop frequency
chop.profile Chop profile
chop.sys Chop coordinate system
chop.tilt (deg) Chop tilt angle
chop.tip (deg) Chop tip angle
dfoc (um) Focus offset.
env.tamb (C) Ambient temperature
env.pwv (um) Precipitable Water Vapor level
env.sect (C) Secondary mirror temperature.
env.t1 (C) T1 temperature of the primary.
env.t2 (C) T2 temperature of the primary.
env.t2 (c) T3 temperature of the primary.
inst.cfg Instrument configuration
inst.datatype Type of data produced by instrument.
inst.exp (s) Total exposure time
inst.inttime (s) Ttoal on-source integration time
inst.mode Operating mode
inst.slit Slit element name
inst.spec1 First spectral element (from SPECTEL1)
inst.spec2 Second spectral element (from SPECTEL2)
inst.wave (um) Central wavelength
mssn.id SOFIA Mission ID.
mssn.leg Flight leg number
mssn.plan Flight plan ID.
nod.amp (arcsec) Nod amplitude
nod.angle (deg) Nodding angle
nod.dwell (s) Nod dwelling time
nod.n Nod cycles
nod.pattern Nod pattern style
nod.pos Nod beam position
nod.settle (s) Nod settling time
nod.style Nod style
nod.sys Nod coordinate system name
obs.aor SOFIA AOR ID
obs.aot SOFIA AOT ID
obs.dict FITS keyword dictionary version ID.
obs.fgid File group ID.
obs.imgid Image ID
obs.obj Object name
obs.objtype Object type
obs.obsid Observation ID.
obs.src Data source
orig.creator Creator software & version. E.g. 'crush v2.42-1'
orig.file Original file name at creation
orig.observer Name(s) of observer(s) or PI(s).
orig.operator Name(s) of the operator(s).
orig.org Organization where file was created / reduced.
pnt.dSIBSX (pixels) SIBS pointing increment in X
pnt.dSIBSY (pixels) SIBS pointinh increment in Y
proc.stat Header key status. E.g. 'ORIGINAL'
proc.level Product level
proc.nspec Number of spectra
proc.product Product type
proc.q Product integer quality
proc.quality Product quality string.
scan.Ax (arcsec) X amplitude
scan.Ay (arcsec) Y amplitude
scan.angle (deg) Scan position angle
scan.cross Is cross scanning?
scan.dec (deg) Declination
scan.dY (arcsec) Raster scan step size
scan.frel Lissajous Y/X frequency ratio
scan.iters Number of iterations
scan.nsubs Number of subscans
scan.PA (deg) Current scan position angle
scan.pattern Scanning pattern
scan.phi0 (deg) Lissajous Y/X relative phase.
scan.ra (hour) R.A.
scan.speed (arcsec/s) Scanning speed.
scan.strips Number of raster strips
scan.sys Scan coordinate system name.
scan.T (s) Scan duration
scan.t0 (s) Lissajous time shift.
scan.trk Is tracking enabled?
scan.type Type of scan, e.g. 'LISSAJOUS'.
scan.X (arcsec) Raster scan length
spec.bw (MHz) Total spectral bandwidth.
spec.df (MHz) Frequency spacing of channels.
spec.obsfreq (GHz) Observing frequency at the reference channel.
spec.restfreq (GHz) Rest frequency at the reference channel.
spec.tsys (K) System temperature (if applicable).
spec.vsys Radial velocity reference system.
spec.vframe (km/s) Radial velocity of refence frame wrt observer.
spec.vrad (km/s) Source radial velocity wrt observer.
spec.z redshift of source.
tel.focus (um) Telescope z-focus value
tel.bdec Boresight declination
tel.bra Boresight RA
tel.cfg Telescope configuration
tel.el Telescope elevation
tel.epoch Epoch for equatorial coordinates (e.g. 2000.0)
tel.fbc FBC status
tel.los Telescope line-of-sight angle
tel.rdec Requested declination
tel.rew Timestamp of last rewind.
tel.rra Requested RA
tel.trkerr Had tracking errors?
tel.trkmode Tracking mode
tel.vpa Vertical position angle (i.e. parallactic angle)
tel.xel Telescope cross-elevation angle
tel.za Telescope zenith angleCopyright (C)2018 – Attila Kovacs <attila[AT]sigmyne.com>