*** CRUSH User's Manual Extension for the SCUBA-2 *** Attila Kovacs Last updated: 8 Sep 2015 Table of Contents ================= 1. Introduction 1.1 Computing requirements 1.1 Performance 2. Setup and configuration 2.1 Starlink 2.2 Your personal settings 3. Quickstart guide 3.1 Pointing and calibration 3.2 Source brightness 3.3 Extended sources 3.3 Compact sources 3.5 Smoothing 4. Advanced Topic: Reading Scans 4.1 FITS vs. SDF 4.2 '450um' vs '850um' reduction mode 4.3 Locating scans by date and scan number 4.4 Specifying scans via file names 5. Advanced Topic: Calibration and Pointing 5.1 Extinction Correction 5.2 Calibration Correction 5.3 Pointing Corrections 6. SCUBA-2 specific pixel divisions 7. Glossary of SCUBA-2 specific options 8. SCUBA-2 specific log quantities ##################################################################### 1. Introduction ##################################################################### This document contains information specific to using CRUSH to reduce SCUBA-2 data. It is assumed that you are already familiar with the contents of the main CRUSH README (inside the distribution directory), especially its Section 1 (Getting Started). If you run into difficulty following this document, you will probably find it useful to study the main README a little longer. 1.1. Computing requirements =========================== CRUSH does not require any special computing hardware to reduce SCUBA-2 data. You just need a run-of-the-mill PC or Mac (laptop, desktop or server) with a few GB or RAM (to fit the SCUBA-2 data), running any flavor of UNIX (such as Linux, BSD, or MacOS) or Windows. (CRUSH will also run on ARM-based platforms, but don't expect blazing speeds on these mobile-class devices...) You will also need Java 8 (a.k.a 1.8.0) or later on the machine, but you probably have that already... Your main limitation will likely be the RAM available, which will limit how many scans you can reduce at once. But, you might be surprised just how far a few GB will get you... 1.2. Performance ================ CRUSH is lean and mean when it comes to memory use and reduction speed. It is also thoroughly parallelized, and will use all processing cores in your machine effectively. If your machine has N processing cores, you can expect to reduce a minute of full 850um SCUBA-2 data (4 subarrays) in about an Nth of a minute. Thus, with a dual-core i5 mobile CPU, my Lenovo T430s laptop churns through a minute of 850um SCUBA-2 data in around 30 seconds. An i7 desktop with 4 cores should get there in 15 seconds, while a 12-core Xeon should clock in at around 5 seconds reduction time per minute of observation (provided you are not slowed down by disk access when reading the data). ##################################################################### 2. Setup and configuration ##################################################################### Start by installing CRUSH, as instructed by the main README. 2.1 Starlink ============ CRUSH will readily read SCUBA-2 data in FITS format, when these are available. More typically, though, your SCUBA-2 data comes in NDF (.sdf) format of the Starlink suite, and so you will need to convert your files to FITS for using with CRUSH. Fortunately, CRUSH can make the conversion seamless, as long as the necessary Starlink tools are accessible. (Once you have FITS files, you no longer need Starlink, or the original .sdf files.) To set up the automatic NDF to FITS conversion, first make sure the Starlink suite is installled on your machine. (You only really need the 'ndf2fits' tool included in Starlink...). You can now continue personalizing CRUSH. 2.2 Your personal settings ========================== Start by creating (or editing) '.crush2/scuba2/default.cfg' in your home directory (this file will contain your personalized configuration entries). E.g. on UNIX and Mac OS: > mkdir -p ~/.crush2/scuba2 Then create/edit 'default.cfg' therein, e.g. with 'nano': > nano ~/.crush2/scuba2/default.cfg (Replace 'nano' with your preferred editor, such as 'vi' or 'emacs' or 'gedit'.) In this file you can specify your default values for the SCUBA-2 scan data directory ('datapath'), and where CRUSH should write its output files, such as the reduced images ('outpath'). You might also want to specify where the Starlink 'ndf2fits' conversion tool is. E.g.: datapath /home/data/scuba2 outpath ~/images ndf2fits /usr/local/Starlink/bin/convert/ndf2fits (The 'ndf2fits' setting may not be needed if you already have FITS files, or if the STARLINK_DIR environment variable points to your Starlink distribution folder...). Also, make sure the output directory actually exists. :-) You can also specify and override any of these settings ad-hoc on the command line later. E.g.: > crush scuba2 -datapath=/home/myData -outpath=../myImages [...] ##################################################################### 3. Quickstart guide ##################################################################### Now, let's reduce some data. Suppose you want to reduce pointing scan #62 from 2015-06-30, at 850um: > crush scuba2 -850um -date=2015-06-30 62 (The '-850um' or '-450um' options select the imaging band for the reduction -- CRUSH will default to 850um imaging if the imaging band is not explicitly specified.) You may have to prepend the to 'crush' when invoking. E.g. use './crush [...]' (if running 'crush' from within its own directory), or '~johndoe/crush/crush [...]'. After the reduction completes, you look at your images with CRUSH's own 'show' tool. E.g.: > show ~/images/URANUS.20150630.62.850um.fits And, you can post-process output images using 'imagetool' (and also 'show', which takes all of imagetool's options). E.g.: > imagetool -smooth=7.5 ~/images/URANUS.20150630.62.850um.fits (To see a help screen with the available image processing options, run 'imagetool -help' or check out the online manuals on the CRUSH site). Of course, you may also use your own favorite FITS viewer/tool, if you prefer. 3.1. Pointing and Calibration ============================= For pointing scans, CRUSH will automatically perform a pointing fit at the end of the reduction, complete with peak and integrated (aperture) flux measures. Suppose, you find that the pointing was off by -1.2,4.6 arcsec (in Az/El). And, suppose the flux (peak or integrated) you found was too low by a factor of 1.14. You can the apply these corrections (to this scan, or others) as: > crush scuba2 [...] -pointing=-1.2,4.6 -scale=1.14 [...] Now, you are ready to reduce some science scans from the same day (say scan numbers 63-66, 70 with the above pointing and 81-83 with a different pointing): > crush scuba2 [...] -pointing=-1.2,4.6 63-66 70 -pointing=2.3,1.8 81-83 Similarly, you can apply different calibration scalings too for the different scans. In general, any options will be applied to all scans after the option is set, redefined again, cleared (using -forget), or blacklisted (via -blacklist) on the command line. Thus, global options should be up front before any of the scans are listed... CRUSH comes with a rough built-in calibration for SCUBA-2, with around 10% rms blind calibration error. It is based on a handful of primary calibrator scans on Uranus and Neptune, and the temperature model in GILDAS/ASTRO. It is possible that with a more thorough analysis and a larger representative sample of calibrator scans, the blind calibration may be improved. If you would like to help improve the blind calibration, please contact Attila Kovacs . 3.2. Source brightness ====================== The examples above produce a default reduction, which is meant to work with a wide variety of sources (brightness and extent), but you may do better than that for your particular target. For example, if your source is faint you may use the '-faint' option. For deep fields (very faint point sources) you can use '-deep' which will aggressively filter the large scales... E.g.: > crush scuba2 -faint [...] or > crush scuba2 -deep [...] For very bright sources (which might be flagged excessively by the default settings) you may try '-bright'. You can find more details on pointing and calibration in the sections below. A more detailed general overview is in the main README (e.g. the Quick Start guide there, and the various advanced topics). 3.3 Extended sources ==================== By default, CRUSH is optimized for reducing compact sources, and to provide the cleanest possible images. This works well up to scales of 2 to 4 arcmins across (depending on how bright the source is). If you have a more extended source, you may try the 'extended' option. E.g.: > crush scuba2 -extended [...] or > crush scuba2 -faint -extended [...] (I.e., 'extended' may be combined with other options like 'faint' to tune the reduction for your particular source). The 'extended' option will recover more of the large scales (up to FoV or, for bright sources, beyond the FoV -- see the corresponding section in the main README), but you will pay the price of reduced sensitivity (increased noise, especially on the large scales). You can further fine-tune 'extended' mode reductions by specifying an approximate source extent (FWHM) via the 'sourcesize' option (in arcsecs), and by changing the number of iterations via 'rounds'. E.g. > crush scuba2 -extended -sourcesize=300 -rounds=20 [...] (The above will set the expected source size to 300" fwhm and will iterate 20 times.) The larger the 'sourcesize', and the more you iterate, the more extended emission you will recover, but you will also see more and more noise, especially on the larger scales (such as a wavy background). You need to make you own call as to what is best. You can find more information on the recovery of extended emission in the main README, under the 'Advanced Topics' section. 3.4 Compact Sources =================== For compact sources (<~ 2') you may get a cleaner image if you remove gradients from the exposures. The rationale is that the gradient removal will reject sky-noise to first order. However, it will also toss out any structure that is comparable to, or larger than, FoV/2. However, structures much smaller than that will remain largely unaffected. Gradient removal is default for 'faint' and 'deep' mode reductions. If you want to turn on gradient removal otherwise, you can do it via the 'gradients' flag. E.g.: > crush scuba2 -gradients [...] 3.5 Smoothing ============= You have the option to smooth your images via the 'smooth' option. Beware, however, that CRUSH typically applies one smoothing for intermediate maps (to increase redundancy between iterations), and another for the final output map. Therefore, to smooth the output map by a 6" FWHM Gaussian, you will want to use: > crush scuba2 -final:smooth=6.0 [...] (The prepended "final:" directive instructs crush to apply this setting in the final iteration, overriding whatever setting was specified for it before.) Beyond specifying the size (FWHM) of the smoothing kernel, you can also set one of: 'minimal', 'halfbeam', '2/3beam', or 'beam' to specify the smoothing kernel relative to the beam size ('minimal' is 1/3 of a beam). E.g.: > crush scuba2 -final:smooth=halfbeam [...] Default reductions are unsmoothed, whereas 'faint' images will be smoothed with '2/3beam', and 'deep' images will be 'beam' smoothed, by default. To disable smoothing your output image, you can use: > crush scuba2 -final:forget=smooth [...] To learn more about the effects of smoothing, please refer to the section 'Pixellization and Smoothing' under 'Advanced Topics' in the main README. ##################################################################### 4. Advanced Topic: Reading Scans ##################################################################### 4.1 FITS vs. SDF ================ The SCUBA-2 data are recorded in the proprietary SDF format of Starlink. While the SDF files can be accessed with the Starlink software suite, the format itself is not easily readable independently. Therefore, CRUSH accesses these data via the Starlink suite. The Starlink suite provides a utility for converting SDF files to FITS, which is the format expected by CRUSH. The 'ndf2fits' tool usually resides in the 'bin/convert' subdirectory of the Starlink installation directory. You have two choices for handling the conversion. You can either do this manually, or let CRUSH convert the files on-the-fly, as needed. In either case, you will need access to the Starlink software suite. Manual Conversion ----------------- If you opt for manual conversion, make sure to use the 'proexts' option of 'ndf2fits'. E.g.: > ndf2fits .sdf .fits proexts You should name the FITS files exactly like the originals SDF ones, changing only the extension from '.sdf' to '.fits'. On-the-Fly Conversion --------------------- The alternative is to let CRUSH handle the conversion automatically, by calling on 'ndf2fits' internally. If 'ndf2fits' is in your path, you should be set to go. Otherwise, you will have to instruct crush where to find the 'ndf2fits' executable. You can do this by editing 'default.cfg' in the 'scuba2' subdirectory of crush. Change the value of the 'ndf2fits' keyword to indicate the location of the executable. E.g.: ndf2fits /usr/local/starlink/bin/convert/ndf2fits Alternatively, if the STARLINK_DIR environment variable is set, and points to the Starlink directory, as recommended by the Starlink installation, then the default setting (in config/scuba2/default.cfg): ndf2fits {@STARLINK_DIR}/bin/convert/ndf2fits will also work. If you set this up correctly, CRUSH will manage the conversion of SDF files automatically, writing the FITS files in the same directory as the SDF. 4.2 '450um' vs '850um' reduction mode ===================================== When reducing SCUBA-2 data with CRUSH, you should specify the wavelength, in which you are interested. This way CRUSH will load settings that are appropriate for mapping in that band. You can choose between '450um' or '850um' reduction modes. E.g., on the command-line: > crush scuba2 -450um [...] If you do not define the wavelength, then CRUSH defaults to '850um' reduction. (The wavelength does not have to be the first argument, but should precede the first scan number or name in the argument list). 4.3 Locating scans by date and scan number ========================================== The preferred way of reading SCUBA-2 data is by 'date' and scan number. The convention of SCUBA-2 is to spread the scan data over several files. Data from each subarray is dumped at some intervals (typically 30s), which are referred to as subscans. Thus, with two subarrays, the data for a 10 minute scan are typically spread over 40 files. (The file names begin with the subarray id, 's4a' for the current single 450um subarray, and 's8d' for the single 850um subarray, then continue with the date ('YYYY-MM-DD' or 'YYYYMMDD' format), scan number, and finally the subscan number.) Thus, given the location of the data ('datapath' option), the reduction wavelength ('450um' or '850um'), the date, and the scan number, CRUSH can easily find all files necessary. E.g.: > crush scuba2 -450um -datapath=/home/data/scuba2 -date=2010-03-01 36 The above line will load all usable subscans of scan #36. Some subscans may be designated unusable by CRUSH, either because the subscan is a dark measurement, or because it contains no coordinate data from the telescope, or because the subscan is shorter than a preset minimum length (3s). These will be dropped from the reduction queue. You can override the minimum length requirement by changing or commenting the 'subscans.minlength' option in 'scuba2/default.cfg' Scan ranges can be specified as usual, e.g.: > crush scuba2 -date=2010-03-01 39-42 45-48 -date=2010-03-02 12-15 17 4.4 Specifying scans via file names =================================== The alternative is to specify file names directly. You can use both full pathnames (which can contain wildcards and other shell-expansions) or give exact file names (without wildcards, etc.) relative to 'datapath'. Locating scans by file names is not recommended in general, since it allows for processing 450um scans in '850um' mode (and vice versa) or even mixing the data from the different wavelengths. On the other hand, you may want to use file-names for locating scans, if: * your scans are named differently from the standard convention, or * you want to control which subscans are processed. For example: > crush scuba2 -450um /data/scuba2/s4a20100301_00036_0002.sdf or, equivalently > crush scuba2 -450um -datapath=/data/scuba2 s4a20100301_00036_0002.sdf will both process only the 2nd subscan of scan 36 (taken on 2010-03-01) in '450um' mode. ##################################################################### 5. Advanced Topic: Calibration and Pointing ##################################################################### The default configuration of CRUSH provides approximate blind calibration, expected to be good to ~10% rms, when using aperture fluxes in apertures large enough to contain the SCUBA-2 error beams of the emission. When using peak fluxes, the blind calibration accuracy is expected to be worse due to the variations in main-beam efficiency with focus, elevation and the surface quality of the JCMT. To improve on the calibration accuracy, you can switch to a smoothed 183 GHz tau value based on the JCMT 183 GHz archive ('tau=jctables'), and/or provide order-of-unity corrections based on observations of nearby calibrator sources (via 'scale'). E.g. 5.1 Extinction Correction ========================= The JCMT records both 225GHz and 183GHz opacities from radiometer measurements. These are stored in the data files. By default, CRUSH uses the 183GHz value with an appropriate scaling relation to the SCUBA-2 bands, to obtain a guesstimate of the in-band opacities for the atmospheric extinction correction. You can get a more accurate local average value of the 183 GHz tau using the the 'tau=jctables' option (requires an online connection!). > crush scuba2 -tau=jctables [...] The above will download the table of measurements from the online JCMT 183 GHz tau archive for the date(s) of each scan, and calculate a local average around the time of observation within a Gaussian window (see 'tau.window' option in the GLOSSARY and in 'scuba2/default.cfg'). The resulting tau should be more precise than the default, but is noticeably slower to obtain (due to the size of the tables that need to be downloaded on demand) and requires a online connection. 5.2 Calibration Correction ========================== Even if the opacity is perfectly determined, there are remaining uncertainties in the calibration. These are mainly due to the beam-quality and variations thereof, due to imperfect focusing, gravitational flexing of the JCMT primary with elevation, or changes in surface quality (e.g. before and after a holography run). Because the JCMT beams are not perfectly Gaussian (esp. at 450um), the calibration scaling is also dependent on the type of flux measure used. You will need a different flux scaling when using apertures (of different sizes) or peak fluxes (for point sources). To calibrate your science target properly you should first determine an order-of-unity calibration correction using the same flux measure (peak flux or integrated flux in the *same* aperture) on a known calibrator source. You can use > crush scuba2 -point [...] to measure both peak fluxes and fluxes in 98% inclusive apertures on compact calibrator sources. You can then determine a appropriate calibration scaling as the ratio of expected flux to observed flux. To multiply your science data with this scaling, use the 'scale' option of crush. E.g.: > crush scuba2 [...] -scale=1.23 [...] The blind calibration of CRUSH is based on apertures large enough to capture 98% of the integrated source flux of point sources (including the first error beam), because this measure is much less dependent on beam quality. Therefore, if you are to use peak fluxes (e.g. for deep field source extraction) you should pay special attention to derive appropriate calibration scalings using the same method. 5.3 Pointing Corrections ======================== You can specify incremental pointing corrections as AZ,EL offsets for every group of scans, the usual way using the 'pointing' option. E.g.: > crush scuba2 [...] -pointing=-2.3,1.6 [...] ##################################################################### 6. SCUBA-2 specific pixel divisions ##################################################################### For a general overview of channel divisions, please consult Section 3. (Correlated Signals) in the main README document. subarrays A grouping of pixels by subarrays. rows Pixel grouping by detector columns. There are 40 columns on each subarray. Incidentally, the channels in a row are all read out through the same SQUID mux. Thus, the group 'cols' is also the grouping by SQUID multiplexer. cols Pixel grouping by detector rows. There are 32 rows on each subarray. blocks Pixel grouping by user-specified geometric blocks of n by m pixels. The size of the blocks is specified via the 'block' option. ##################################################################### 7. Glossary of SCUBA-2 specific options ##################################################################### 450um Select 450um imaging mode. 850um Select 850um imaging mode. block= block=x @Advanced Set the block size (pixels) for 'correlated.blocks'. The decorrelation block size will be by , or by pixels, depending whether there are 1 or 2 arguments (separated by 'x'). @See: 'correlated.blocks' blocks @Alias: 'correlated.blocks' cols @Alias: 'correlated.cols' @Expert Specifies the grouping of channels by geometric columns on the array, following the storage and array geometry convention of 32 rows by 40 columns. Since the 32 pixels of a 'column' are read through the same SQUID MUX, decorrelating on 'cols' is effectively decorrelates channels that share a MUX. @See: 'correlated.' darkcorrect @Expert @Since: 2.31 Attempt to use dark SQUID data to remove common mode noise on SQUID MUX. In practice, the use of this option may not improve reduction quality, and may in fact degrade it. Therefore, the option is disabled by default. date=YYYY-MM-DD date=YYYYMMDD Specify the the date of observation for locating scan data. When used together with 'datapath' and scan numbers, CRUSH can locate all the necessary input files easily. E.g.: > crush scuba2 -date=2014-05-30 75 @See: 'datapath' he3 @Alias: 'correlated.he3' Remove correlations with the detector temperature (at the He3 stage -- hence the name). This may, or may not improve the sensitivity of the data. mirror @Expert Indicate a mirrored focal plane layout. (Since one subarray is seen in transmission, the other in reflection, through the beam-splitter, one of the subarray layouts is a mirror image of the other... ndf2fits= Specify the path to the Starlink 'ndf2fits' executable (usually in bin/convert subdirectory of the Starlink distribution. You may not need to change this option if the STARLINK_DIR environment variable is set to point to the Starlink installation folder on your system, as recommended by the Starlink installation instructions. Otherwise, you can set this e.g.: ndf2fits /usr/local/starlink/bin/convert/ndf2fits pixelmm=X @Expert Specify the physical pixel size (separation) in mm, on the detector arrays. @See: 'platescale' platescale=X @Expert Specify the scaling from mm on the focal plane to arcsec on the sky. @See: 'pixelmm' rows @Alias: 'correlated.rows' @Expert Specifies the grouping of channels by geometric rows on the array, following the storage and array geometry convention of 32 rows by 40 columns. @See: 'correlated.' .position=dX,dY @Expert Set the position of pixel 0 (first row, first column) of subarray to dX, dY pixels from the optical center. E.g. s8a.position -37.3, -35.3 The offset will be applied *after* any rotation. @See: '.rotation' .rotation=X @Expert Set the orientation angle (degrees) of subarray on the focal plane. E.g. s8d.rotation -90 @See: .position' .scale=X @Expert Set the subarray scalings. Scaling the data from the different subarrays such that the pixels on a subarray show the same response, on average, from one subarray to the next, is useful for maintaining calibration when not all subarray data is available, or when reducing just a select set of subarrays. E.g.: s8a.scale 0.63 @See: 'subarray' subarray= @Expert Restrict the reduction to just the specified subarrays at the current wavelength. The argument is a list of subarray letter codes (a thru d) that specify which subarray data to use. E.g.: >crush scuba2 -850um -subarrays=a,c [...] Will reduce only the 850um subarrays a and c. @See: '.scale' ##################################################################### 8. SCUBA-2 specific log quantities ##################################################################### Coming soon... dir filter foc.X foc.Y foc.Z obsmode obspattern obstype ##################################################################### 9. Credits ##################################################################### This re-incarnation of the CRUSH support for SCUBA-2 would not have happened without the essential help from others. The thanks go to Tony Mroczkowski for the persistent prodding to create this module, and for seeking out the necessary training data for configuring it; and Ed Chapin, without whom I would not have made enough sense of the data themself. If you like the outcome, I suggest you invite them (and me) for a beer or two, if you happen to bump into them (or me). :-) ----------------------------------------------------------------------------- Copyright (C)2015 Attila Kovacs