API Documentation

Usage

This is the Python Fast Holographic Deconvolution package, only the observation ID (obs_id) and configuration (-c, –config) is required to start your run, but you should need to modify these arguments below to get something useful. If you don’t supply a configuration file, pyfhd will use the default configuration file in the resources/config directory from the pyfhd install.

usage: pyfhd [-h] [-c CONFIG] [-v] [-i INPUT_PATH] [--get-sample-data]
             [--recalculate-all | --no-recalculate-all | -r]
             [--silent | --no-silent | -s] [--log-file | --no-log-file | -l]
             [--instrument {mwa,ovro-lwa,hera,other}] [--dimension DIMENSION]
             [--elements ELEMENTS] [--kbinsize KBINSIZE] [--FoV FOV]
             [--deproject_w_term DEPROJECT_W_TERM]
             [--conserve-memory | --no-conserve-memory]
             [--memory-threshold MEMORY_THRESHOLD]
             [--min-baseline MIN_BASELINE] [--n-pol {0,2,4}]
             [--save-checkpoints | --no-save-checkpoints]
             [--obs-checkpoint | --no-obs-checkpoint]
             [--calibrate-checkpoint | --no-calibrate-checkpoint]
             [--gridding-checkpoint | --no-gridding-checkpoint]
             [--override-target-phasera OVERRIDE_TARGET_PHASERA]
             [--override-target-phasedec OVERRIDE_TARGET_PHASEDEC]
             [--calibrate-visibilities | --no-calibrate-visibilities | -cv]
             [--transfer-calibration TRANSFER_CALIBRATION]
             [--cal-stop | --no-cal-stop]
             [--cal-convergence-threshold CAL_CONVERGENCE_THRESHOLD]
             [--cal-adaptive-calibration-gain | --no-cal-adaptive-calibration-gain]
             [--cal-base-gain CAL_BASE_GAIN]
             [--min-cal-baseline MIN_CAL_BASELINE]
             [--max-cal-baseline MAX_CAL_BASELINE]
             [--cable-bandpass-fit | --no-cable-bandpass-fit]
             [--cal-bp-transfer CAL_BP_TRANSFER]
             [--calibration-polyfit | --no-calibration-polyfit]
             [--cal-amp-degree-fit CAL_AMP_DEGREE_FIT]
             [--cal-phase-degree-fit CAL_PHASE_DEGREE_FIT]
             [--cal-reflection-hyperresolve | --no-cal-reflection-hyperresolve]
             [--cal-reflection-mode-theory CAL_REFLECTION_MODE_THEORY]
             [--cal-reflection-mode-delay | --no-cal-reflection-mode-delay]
             [--cal-reflection-mode-file | --no-cal-reflection-mode-file]
             [--calibration-auto-fit | --no-calibration-auto-fit]
             [--calibration-auto-initialize | --no-calibration-auto-initialize]
             [--cal-gain-init CAL_GAIN_INIT]
             [--vis-baseline-hist | --no-vis-baseline-hist]
             [--bandpass-calibrate | --no-bandpass-calibrate]
             [--cal-time-average | --no-cal-time-average]
             [--auto-ratio-calibration | --no-auto-ratio-calibration]
             [--digital-gain-jump-polyfit | --no-digital-gain-jump-polyfit]
             [--cal-phase-fit-iter CAL_PHASE_FIT_ITER]
             [--max-cal-iter MAX_CAL_ITER]
             [--flag-model | --no-flag-model | -fm]
             [--flag-visibilities | --no-flag-visibilities | -fv]
             [--flag-calibration | --no-flag-calibration | -fc]
             [--flag-calibration-frequencies | --no-flag-calibration-frequencies | -fcf]
             [--flag-basic | --no-flag-basic | -fb] [-ft FLAG_TILES]
             [--flag-frequencies | --no-flag-frequencies | -ff]
             [--flag-freq-start FLAG_FREQ_START]
             [--flag-freq-end FLAG_FREQ_END] [--time-cut TIME_CUT]
             [-b SAVED_BEAM_FILE_PATH] [--uvbeam-file-path UVBEAM_FILE_PATH]
             [--uvbeam-freq-buffer UVBEAM_FREQ_BUFFER]
             [--analytic-beam-yaml ANALYTIC_BEAM_YAML]
             [--lazy-load-beam | --no-lazy-load-beam | -ll]
             [--recalculate-beam | --no-recalculate-beam]
             [--beam-nfreq-avg BEAM_NFREQ_AVG] [--psf-dim PSF_DIM]
             [--psf-resolution PSF_RESOLUTION]
             [--beam-mask-threshold BEAM_MASK_THRESHOLD]
             [--beam-model-version BEAM_MODEL_VERSION]
             [--beam-clip-floor | --no-beam-clip-floor]
             [--interpolate-kernel | --no-interpolate-kernel]
             [--beam-per-baseline | --no-beam-per-baseline]
             [--beam-offset-time BEAM_OFFSET_TIME]
             [--recalculate-grid | --no-recalculate-grid | -g]
             [--image-filter {filter_uv_uniform}]
             [--mask-mirror-indices | --no-mask-mirror-indices]
             [--grid-weights | --no-grid-weights]
             [--grid-variance | --no-grid-variance]
             [--grid-uniform | --no-grid-uniform]
             [--grid-spectral | --no-grid-spectral]
             [--dft-threshold | --no-dft-threshold] [-o OUTPUT_PATH]
             [--description DESCRIPTION]
             [--export-images | --no-export-images]
             [--snapshot-healpix-export | --no-snapshot-healpix-export]
             [--pad-uv-image PAD_UV_IMAGE]
             [--ring-radius-multi RING_RADIUS_MULTI]
             [--save-obs | --no-save-obs] [--save-params | --no-save-params]
             [--save-cal | --no-save-cal]
             [--save-visibilities | --no-save-visibilities]
             [--save-weights | --no-save-weights]
             [--save-healpix-fits | --no-save-healpix-fits]
             [--save-model | --no-save-model]
             [--calibration-plots | --no-calibration-plots]
             [--gridding-plots | --no-gridding-plots]
             [--image-plots | --no-image-plots] [-m {sav,uvfits}]
             [--model-file-path MODEL_FILE_PATH]
             [--allow-sidelobe-model-sources | --no-allow-sidelobe-model-sources]
             [--ps-kbinsize PS_KBINSIZE] [--ps-kspan PS_KSPAN]
             [--ps-beam-threshold PS_BEAM_THRESHOLD] [--ps-fov PS_FOV]
             [--ps-dimension PS_DIMENSION] [--ps-degpix PS_DEGPIX]
             [--ps-nfreq-avg PS_NFREQ_AVG]
             [--ps-tile-flag-list PS_TILE_FLAG_LIST] [--n-avg N_AVG]
             [--rephase-weights | --no-rephase-weights]
             [--restrict-healpix-inds | --no-restrict-healpix-inds]
             [--healpix-inds HEALPIX_INDS]
             [--split-ps-export | --no-split-ps-export]
             obs_id

Positional Arguments

obs_id: The Observation ID as per the MWA file naming standards. Assumes the fits files for this observation is in the uvfits-path. obs_id and uvfits replace file_path_vis from FHD

Named Arguments

-c, --config

Configuration File Path for pyfhd (By default will find pyfhd in the Python Path and use the default config file pyfhd/resources/config/pyfhd.yaml).

Default: /home/docs/checkouts/readthedocs.org/user_builds/pyfhd/envs/latest/lib/python3.11/site-packages/pyfhd/resources/config/pyfhd.yaml

-v, --version

show program’s version number and exit

-i, --input-path

Directory for the uvfits files and other inputs, by default it looks for a directory called input in the working directory

Default: ./input/

--get-sample-data

Copy sample data from pyfhd package directory to the current working directory. Will copy to an ‘input’ directory.

Default: False

--recalculate-all, --no-recalculate-all, -r

Forces pyfhd to recalculate all values. This will ignore values set for recalculate-grid, recalculate-beam, as it will set all of them to True

--silent, --no-silent, -s

This pyfhd stops all output to the terminal except in the case of an error and/or exception

--log-file, --no-log-file, -l

Logging in a log file is enabled by default, set to False in the config to disable logging to a file.

--instrument

Possible choices: mwa, ovro-lwa, hera, other

Set the instrument used for the FHD run.

Default: 'mwa'

--dimension

The number of pixels in the UV plane along one axis.

Default: 2048

--elements

The number of pixels in the UV plane along the other axis.

Default: 2048

--kbinsize

Size of UV pixels in wavelengths. Given a defined number of pixels in dimension, this sets the UV space extent. This will supersede degpix if dimension is also set.

Default: 0.5

--FoV, --fov

A proxy for the field of view in degrees. FoV is actually used to determine kbinsize, which will be set to !RaDeg/FoV. This means that the pixel size at phase center times dimension is approximately equal to FoV, which is not equal to the actual field of view owing to larger pixel sizes away from phase center. If set to 0, then kbinsize determines the UV resolution.

--deproject_w_term

Enables the function for simple_deproject_w_term and uses the parameter value for the direction value in the function

--conserve-memory, --no-conserve-memory

Optionally split many loops into chunks in the case of high memory usage.

Default: False

--memory-threshold

Set a memory threshold for each chunk in set in bytes. By default it is set at ~100MB

Default: 1000000000.0

--min-baseline

The minimum baseline length in wavelengths to include in the analysis

Default: 1.0

--n-pol

Possible choices: 0, 2, 4

Set number of polarizations to use (XX, YY versus XX, YY, XY, YX).

Default: 2

--dft-threshold, --no-dft-threshold

Set to True to use the DFT approximation. When set equal to 0 the true DFT is calculated for each source. It can also be explicitly set to a value that determines the accuracy of the approximation.

Default: False

Checkpoints

Activate checkpoints and Load up checkpoints

--save-checkpoints, --no-save-checkpoints

Activates pyfhd’s checkpointing system and saves them into the output directory

Default: False

--obs-checkpoint, --no-obs-checkpoint

Load the checkpoint just after creating the observation metadata dictionary, should contain the observation metadata dictionary, uncalibrated visibility parameters, array and weights. If calibrate-checkpoint has been set, then obs-checkpoint will be ignored

Default: False

--calibrate-checkpoint, --no-calibrate-checkpoint

Load the checkpoint after calibration containing the observation metadata dictionary with flagged tiles and frequencies, the calibration dictionary containing the gains and the calibrated visibility parameters, model, array and weights.

Default: False

--gridding-checkpoint, --no-gridding-checkpoint

Load the checkpoint after gridding containing the gridded uv planes for the image, weights, variance and filter, with an updated observation metadata dictionary. Should be used in conjunction with the calibrate-checkpoint option

Default: False

Instrument

Adjust parameters specific to your instrument

--override-target-phasera: RA of the target phase center, which overrides the value supplied in the metafits under the header keyword RAPHASE. If the metafits doesn’t exist, it ovverides the value supplied in the uvfits under the header keyword RA
--override-target-phasedec: dec of the target phase center, which overrides the value supplied in the metafits under the header keyword DECPHASE. If the metafits doesn’t exist, it overrides the value supplied in the uvfits under the header keyword Dec.

Calibration

Adjust Parameters for Calibration

--calibrate-visibilities, --no-calibrate-visibilities, -cv

Turn on the calibration of the visibilities. If turned on, calibration of the dirty, modelling, and subtraction to make a residual occurs. Otherwise, none of these occur and an uncalibrated dirty cube is output.

Default: False

--transfer-calibration

The file path of a calibration to be read-in, if you give a directory pyfhd expects there to be a file called <obs_id>_cal.hdf5 using the same observation as you plan to process.

--cal-stop, --no-cal-stop

Stops the code right after calibration, and saves unflagged model visibilities along with the obs structure in a folder called cal_prerun in the pyfhd file structure. This allows for post-processing calibration steps like multi-day averaging, but still has all of the needed information for minimal reprocessing to get to the calibration step. To run a post-processing run, see keywords model_transfer and transfer_psf

Default: False

--cal-convergence-threshold

Threshold at which calibration ends. Calibration convergence is quantified by the absolute value of the fractional change in the gains over the last calibration iteration. If this quantity is less than cal_convergence_threshold then calibration terminates.

Default: 1e-07

--cal-adaptive-calibration-gain, --no-cal-adaptive-calibration-gain

Controls whether to use a Kalman Filter to adjust the gain to use for each iteration of calculating calibration.

Default: False

--cal-base-gain

The relative weight to give the old calibration solution when averaging with the new. Set to 1. to give equal weight, to 2. to give more weight to the old solution and slow down convergence, or to 0.5 to give greater weight to the new solution and attempt to speed up convergence. If use_adaptive_calibration_gain is set, the weight of the new calibration solutions will be calculated in the range cal_base_gain/2. to 1.0

--min-cal-baseline

The minimum baseline length in wavelengths to be used in calibration.

Default: 50.0

--max-cal-baseline

The maximum baseline length in wavelengths to be used in calibration. If max_baseline is smaller, it will be used instead.

--cable-bandpass-fit, --no-cable-bandpass-fit

Average the calibration solutions across tiles within a cable grouping for the particular instrument. Dependency: instrument_config/<instrument>_cable_length.txt

Default: False

--cal-bp-transfer

Use a saved bandpass for bandpass calibration. Read in the specified file with calfits format greatly preferred.

--calibration-polyfit, --no-calibration-polyfit

Calculates a polynomial fit across the frequency band for the gain, and allows a cable reflection to be fit. The orders of the polynomial fit are determined by cal_phase_degree_fit and cal_amp_degree_fit. If unset, no polynomial fit or cable reflection fit are used.

Default: False

--cal-amp-degree-fit

The nth order of the polynomial fit over the whole band to create calibration solutions for the amplitude of the gain. Setting it to 0 gives a 0th order polynomial fit (one number for the whole band), 1 gives a 1st order polynomial fit (linear fit), 2 gives a 2nd order polynomial fit (quadratic), n gives nth order polynomial fit. Requires calibration_polyfit to be enabled.

Default: 2

--cal-phase-degree-fit

The nth order of the polynomial fit over the whole band to create calibration solutions for the phase of the gain. Setting it to 0 gives a 0th order polynomial fit (one number for the whole band), 1 gives a 1st order polynomial fit (linear fit), 2 gives a 2nd order polynomial fit (quadratic), n gives nth order polynomial fit. Requires calibration_polyfit to be enabled.

Default: 1

--cal-reflection-hyperresolve, --no-cal-reflection-hyperresolve

Hyperresolve and fit residual gains using nominal reflection modes (calculated from cal_reflection_mode_delay or cal_reflection_mode_theory), producing a finetuned mode fit, amplitude, and phase. Will be ignored if cal_reflection_mode_file is set because it is assumed that a file read-in contains mode/amp/phase to use.

Default: False

--cal-reflection-mode-theory

Calculate theoretical cable reflection modes given the velocity and length data stored in a config file named <instrument>_cable_length.txt. File must have a header line and at least five columns (tile index, tile name, cable length, cable velocity factor, logic on whether to fit (1) or not (0)). Can set it to positive/negative cable lengths (see cal_mode_fit) to include/exclude certain cable types.

Default: 150

--cal-reflection-mode-delay, --no-cal-reflection-mode-delay

Calculate cable reflection modes by Fourier transforming the residual gains, removing modes contaminated by frequency flagging, and choosing the maximum mode.

Default: False

--cal-reflection-mode-file, --no-cal-reflection-mode-file

Use predetermined cable reflection parameters (mode, amplitude, and phase) in the calibration solutions from a file. The specified format of the text file must have one header line and eleven columns: tile index tile name cable length cable velocity factor logic on whether to fit (1) or not (0) mode for X amplitude for X phase for X mode for Y amplitude for Y phase for Y. The file will be instrument_config of the input directory

Default: False

--calibration-auto-fit, --no-calibration-auto-fit

Use the autocorrelations to calibrate. This will suffer from increased, correlated noise and bit statistic errors. However, this will save the autos as the gain in the cal structure, which can be a useful diagnostic.

Default: False

--calibration-auto-initialize, --no-calibration-auto-initialize

initialize gain values for calibration with the autocorrelations. If unset, gains will initialize to 1 or the value supplied by cal_gain_init

Default: False

--cal-gain-init

Initial gain values for calibration. Selecting accurate inital calibration values speeds up calibration and can improve convergence. This keyword will not be used if calibration_auto_initialize is set.

Default: 1

--vis-baseline-hist, --no-vis-baseline-hist

Calculates the vis_baseline_hist dictionary containing the visibility resolution ratio average and standard deviation

Default: False

--bandpass-calibrate, --no-bandpass-calibrate

Calculates a bandpass. This is an average of tiles by frequency by polarization (default), beamformer-to-LNA cable types by frequency by polarization (see cable_bandpass_fit), or over the whole season by pointing by by cable type by frequency by polarization via a read-in file (see saved_run_bp). If unset, no by-frequency bandpass is used

Default: False

--cal-time-average, --no-cal-time-average

Performs a time average of the model/data visibilities over the time steps in the observation to reduce the number of equations that are used in the linear-least squares solver. This improves computation time, but will downweight longer baseline visibilities due to their faster phase variation.

Default: False

--auto-ratio-calibration, --no-auto-ratio-calibration

Calculates the auto ratios for cable reflections and enables global bandpass

Default: False

--digital-gain-jump-polyfit, --no-digital-gain-jump-polyfit

Perform polynomial fitting for the amplitude separately before and after the highband digital gain jump at 187.515E6.

Default: False

--cal-phase-fit-iter

Set the iteration number to begin phase calibration. Before this, phase is held fixed and only amplitude is being calibrated.

Default: 4

--max-cal-iter

Sets the maximum number of iterations allowed for the linear least-squares solver to converge during vis_calibrate_subroutine. Ideally do not set this number unless you notice some of the frequencies not reaching convergence within 100 iterations and do not set this number to 5 or less.

Default: 100

Flagging

Adjust Parameters for Flagging

--flag-model, --no-flag-model, -fm

Flag the imported model based on time offsets and the tiles. Turn off if you’re dealing with an already flagged model or simulation.

Default: False

--flag-visibilities, --no-flag-visibilities, -fv

Flag visibilities based on calculations in vis_flag

Default: False

--flag-calibration, --no-flag-calibration, -fc

Flags antennas based on calculations in vis_calibration_flag

Default: False

--flag-calibration-frequencies, --no-flag-calibration-frequencies, -fcf

If True, flags frequencies based off 0 calibration gain, if False, ignores the calibration gain for frequencies

Default: False

--flag-basic, --no-flag-basic, -fb

Flags Frequencies and Tiles based on your configuration, params and the visibility weights. The freq_use, tile_use arrays of obs will be adjusted and the vis_weights_arr adjusted to be in line with the freq_use and tile_use arrays. This should be True always, the only time you should consider turning off basic flagging is when you’re dealing with a simulated visibilities and weights in pyfhd

Default: False

-ft, --flag-tiles

A list of tile names to manually flag. I repeat, a list of tile names, NOT tile indices

Default: []

--flag-frequencies, --no-flag-frequencies, -ff

When set to False, pyfhd will not flag any frequencies inside of vis_flag_basic, vis_weights_update or vis_calibration_flag.

Default: False

--flag-freq-start

Frequency in MHz to begin the observation. Flags frequencies less than it. Replaces freq_start from FHD

--flag-freq-end

Frequency in MHz to end the observation. Flags frequencies greater than it. Replaces freq_end from FHD

--time-cut

Seconds to cut (rounded up to next time integration step) from the beginning of the observation. Can also specify a negative time to cut off the end of the observation. Specify a vector to cut at both the start and end.

Beam Setup

Adjust Parameters for the Beam Setup

-b, --saved-beam-file-path

The path to an FHD written beam file (h5 or sav). Cannot be used with uvbeam-file-path or analytic-beam-yaml, one of these three should be set to specify the beam.

--uvbeam-file-path

The path to a beam file readable by pyuvdata.UVBeam. Consider also setting uvbeam-freq-buffer. Cannot be used with saved-beam-file-path or analytic-beam-yaml, one of these three should be set to specify the beam.

--uvbeam-freq-buffer

Buffer around data frequency range to use when reading in the beam in Hz. Set to allow partial beam loading (when possible) to save memory. If not set (the default) the whole beam file will be read in. If set, should be at least 2 * beam frequency resolution (if set too low, beam interpolation errors can occur). We suggest setting it to 2e6 (meaning 2 MHz) for the standard MWA beam file, which has 1 MHz resolution.

--analytic-beam-yaml

The yaml specifier for a pyuvdata AnalyticBeam. See the pyuvdata docs for details, must be enclosed in quotes around the entire (multiline)specification. Cannot be used with saved-beam-file-path or uvbeam-file-path, one of these three should be set to specify the beam.

--lazy-load-beam, --no-lazy-load-beam, -ll

pyfhd will lazy load the beam HDF5 file, allowing pyfhd to be run on much smaller systems with much less memory than FHD

Default: False

--recalculate-beam, --no-recalculate-beam

Forces pyfhd to redo the beam setup using pyfhd’s beam setup.

Default: False

--beam-nfreq-avg

The number of fine frequency channels to calculate a beam for, using the average of the frequencies. The beam is a function of frequency, and a calculation on the finest level is most correct (beam_nfreq_avg=1). However, this is computationally difficult for most machines.

Default: 16

--psf-dim

Controls the span of the beam in u-v space. Some defaults are 30, 54 (1e6 mask with -2) or 62 (1e7 with -2).

Default: 54

--psf-resolution

Super-resolution factor of the psf in UV space. Values greater than 1 increase the resolution of the gridding kernel.

Default: 100

--beam-mask-threshold

The factor at which to clip the beam model. For example, a factor of 100 would clip the beam model at 100x down from the maximum value. This removes extraneous and uncertain modelling at low levels.

Default: 100

--beam-model-version

A number that indicates the tile beam model calculation. This is dependent on the instrument, and specific calculations are carried out in <instrument>_beam_setup_gain. MWA range: 0, 1 (or anything else captured in the else statement), 2 PAPER range: 1 (or anything else captured in the else statement), 2 HERA range: 2 (or anything else captured in the else statement)

Default: 2

--beam-clip-floor, --no-beam-clip-floor

Set to subtract the minimum non-zero value of the beam model from all pixels.

Default: False

--interpolate-kernel, --no-interpolate-kernel

Use interpolation of the gridding kernel while gridding and degridding, rather than selecting the closest super-resolution kernel.

Default: False

--beam-per-baseline, --no-beam-per-baseline

Set to true if the beams were made with corrective phases given the baseline location, which then enables the gridding to be done per baseline

Default: False

--beam-offset-time

Calculate the beam at a specific time within the observation. 0 seconds indicates the start of the observation, and the # of seconds in an observation indicates the end of the observation.

Default: 56

Gridding

Tune the Gridding in pyfhd

--recalculate-grid, --no-recalculate-grid, -g

Forces pyfhd to recalculate the gridding function. Replaces grid_recalculate from FHD

Default: False

--image-filter

Possible choices: filter_uv_uniform

Weighting filter to be applied to resulting snapshot images and fits files. Replaces image_filter_fn from FHD

Default: 'filter_uv_uniform'

--mask-mirror-indices, --no-mask-mirror-indices

Inside baseline_grid_location optionally exclude v-axis mirrored baselines

Default: False

--grid-weights, --no-grid-weights

Grid the weights for the uv plane

Default: False

--grid-variance, --no-grid-variance

Grid the variance for the uv plane

Default: False

--grid-uniform, --no-grid-uniform

Grid uniformally by applying a uniform weighted filter to all uv-planes

Default: False

--grid-spectral, --no-grid-spectral

Optionally use the spectral index information to scale the uv-plane in gridding

Default: False

Export

Adjust the outputs of the pyfhd pipeline

-o, --output-path

Set the output path for the current run, note a directory will still be created inside the given path

Default: ./output/

--description

A more detailed description of the current task, will get applied to the output directory and logging where all output will be stored. By default the date and time is used

--export-images, --no-export-images

Export fits files and images of the sky.

Default: True

--snapshot-healpix-export, --no-snapshot-healpix-export

Save model/dirty/residual/weights/variance cubes as healpix arrays, split into even and odd time samples, in preparation for epsilon.

Default: False

--pad-uv-image

Pad the UV image by this factor with 0’s along the outside so that output images are at a higher resolution.

Default: 1.0

--ring-radius-multi

Sets the multiplier for the size of the rings around sources in the restored images. Ring Radius will equal pad-uv-image * ring-radius-multi. To generate restored images without rings, set ring_radius = 0.

Default: 10

--save-obs, --no-save-obs

Save the obs dictionary created during pyfhd’s run

Default: False

--save-params, --no-save-params

Save the params dictionary created during pyfhd’s run

Default: False

--save-cal, --no-save-cal

Save the calibration dictionary created during pyfhd’s run

Default: False

--save-visibilities, --no-save-visibilities

Save the raw visibilities, calibrated data visibilities, the model visibilities, and the gridded uv planes

Default: False

--save-weights, --no-save-weights

Save the raw and calibrated weights from pyfhd’s run

Default: False

--save-healpix-fits, --no-save-healpix-fits

Create Healpix fits files. Healpix fits maps are in units Jy/sr. Replaces write_healpix_fits

Default: False

--save-model, --no-save-model

Save the model visibilities created transferred in during pyfhd’s run as HDF5.

Default: False

Plotting

Adjust the plotting of the pyfhd pipeline

--calibration-plots, --no-calibration-plots

Turns on the plotting of calibration solutions

Default: False

--gridding-plots, --no-gridding-plots

Turns on the plotting of the continuum gridding outputs

Default: False

--image-plots, --no-image-plots

Turns on the plotting of the continuum fits images

Default: False

Model

Tune the modelling in pyfhd

-m, --model-file-type

Possible choices: sav, uvfits

Set the file type of the model, by default it looks for sav files of format <obs_id>_params.sav and <obs_id>_vis_model_<pol_name>.sav. If you set uvfits you must put set path using –import-model-uvfits. This argument is required as pyfhd currently cannot produce a model.

Default: 'sav'

--model-file-path

In the case you chose sav for model-file-type then this will be a directory containing all the <obs_id>_params and <obs_id>_vis_model_<pol_name> sav files. In the case you chose uvfits, then the path is to a uvfits file, in which case make sure the phase centre of model data must match the ‘RA’ and ‘DEC’ values in the metafits file (NOT the ‘RAPHASE’ and ‘DECPHASE’).

Default: ./input

--allow-sidelobe-model-sources, --no-allow-sidelobe-model-sources

Allows pyfhd to model sources in the sidelobes for subtraction. Forces the beam_threshold to 0.01 in order to go down to 1% of the beam to capture sidelobe sources during the generation of a model calibration source catalog for the particular observation.

Default: False

HEALPIX

Adjust the HEALPIX output

--ps-kbinsize

UV pixel size in wavelengths to grid for Healpix cube generation. Overrides ps_fov and the kpix in the obs structure if set.

Default: 0.5

--ps-kspan

UV plane dimension in wavelengths for Healpix cube generation. Overrides ps_dimension and ps_degpix if set. If ps_kspan, ps_dimension, or ps_degpix are not set, the UV plane dimension is calculated from the FoV and the degpix from the obs structure.

Default: 0

--ps-beam-threshold

Minimum value to which to calculate the beam out to in image space. The beam in UV space is pre-calculated and may have its own beam_threshold (see that keyword for more information), and this is only an additional cut in image space.

Default: 0

--ps-fov

Field of view in degrees for Healpix cube generation. Overrides kpix in the obs dictionary if set.

--ps-dimension

UV plane dimension in pixel number for Healpix cube generation. Overrides ps_degpix if set. If ps_kspan, ps_dimension, or ps_degpix are not set, the UV plane dimension is calculated from the FoV and the degpix from the obs dictionary.

--ps-degpix

Degrees per pixel for Healpix cube generation. If ps_kspan, ps_dimension, or ps_degpix are not set, the UV plane dimension is calculated from the FoV and the degpix from the obs dictionary.

--ps-nfreq-avg

A factor to average up the frequency resolution of the HEALPix cubes from the analysis frequency resolution. By default averages by a factor of 2 when this is set to None.

--ps-tile-flag-list

A list of tile names to manually flag in the healpix export. I repeat, a list of tile names, NOT tile indices.

Default: []

--n-avg

Number of frequencies to average over to smooth the frequency band.

Default: 2

--rephase-weights, --no-rephase-weights

If turned off, target phase center is the pointing center (as defined by Cotter). Setting to False overrides override_target_phasera and override_target_phasedec

Default: False

--restrict-healpix-inds, --no-restrict-healpix-inds

Only allow gridding of the output healpix cubes to include the healpix pixels specified in a file. This is useful for restricting many observations to have consistent healpix pixels during integration, and saves on memory and walltime.

Default: False

--healpix-inds

In the event you want to restrict the healpix indices to a specified file, use a combination of restrict-healpix-inds and this argument to restrict the healpix indexes to your given file rather than a predetermined one from the obs dictionary.

--split-ps-export, --no-split-ps-export

Split up the Healpix outputs into even and odd time samples. This is essential to propogating errors in εppsilon. Requires more than one time sample.

Default: False

pyfhd Setup

class pyfhd.pyfhd_tools.pyfhd_setup.OrderedBooleanOptionalAction(*args, **kwargs)[source]

OrderedBooleanOptionalAction is a custom action based on BooleanOptionalAction that ensures that the long options are always first in the list of options. More specifically, it ensures the the positive long form is first (i.e. –foo), and the negative (i.e. –no-foo) is second. This was needed as configargparse when using checking the config file and the action set in the argparse, set the arg passed into pyfhd by either option[0] or option[1] if the value in the config file was set to True or False respectively. This also allows easy negation of values from the configuration file with the command line if we need to. For example we can now set silent: true in the configuration file, but then set –no-silent in the command line to override the config file.

This is also still ensures that short switches are available in pyfhd with the BooleanOptionalAction.

pyfhd.pyfhd_tools.pyfhd_setup.pyfhd_logger(pyfhd_config: dict) → Tuple[Logger, Path][source]

Creates the the logger for pyfhd. If silent is True in the pyfhd_config then the StreamHandler won’t be added to logger meaning there will be no terminal output even if logger is called later.

Parameters:

pyfhd_config (dict) – The options from the argparse in a dictionary

Returns:

logger (logging.Logger) – The logger with the appropriate handlers added.
output_dir (str) – Where the log and FHD outputs are being written to

pyfhd.pyfhd_tools.pyfhd_setup.pyfhd_parser()[source]

The pyfhd_parser configures the argparse for pyfhd

Returns:: The parser for pyfhd which contains the help strings for the terminal and Usage section of the docs.
Return type:: configargparse.ArgumentParser

pyfhd.pyfhd_tools.pyfhd_setup.pyfhd_setup(options: Namespace) → Tuple[dict, Logger][source]

Check for any incompatibilities among the options given for starting the pyfhd pipeline as some options do conflict with each other or have dependencies on other options. This function should catch all of those potential errors and exit the program with errors once these have been found before any output. This function should also replace fhd_setup

Parameters:

options (argparse.Namespace) – The parsed argparse object.

Returns:

pyfhd_config (dict) – The configuration dictionary for pyfhd containing all the options.
logger (logging.Logger) – The logger with the appropriate handlers added.

pyfhd.pyfhd_tools.pyfhd_setup.write_collated_yaml_config(pyfhd_config: dict, output_dir: Path, description: str = '')[source]

After all inputs have been validated using pyfhd.pyfhd_tools.pyfhd_setup, write out all the arguments gather in pyfhd_config and write out to a yaml configuration file. This yaml file can then be fed back into pyfhd to exactly duplicate the current run.

Parameters:

pyfhd_config (dict) – The options from the argparse in a dictionary
output_dir (Path) – Path to save the file to

Data Setup

pyfhd.data_setup.uvfits.create_layout(antenna_header: Header, antenna_data: FITS_rec, pyfhd_config: dict, logger: Logger) → dict[source]

Create a very explicit antenna and telescope position dictionary, incorperating timing (e.g. time system, reference, leap seconds), location (e.g. array center, coordinate frame, Earth’s rotation), and antenna information (e.g. names, numbers, coordinates, mount type, feed polarization). This is used to create the layout dictionary which is compatible with pyuvdata.

Parameters:

antenna_header (Header) – The header from the second table of the observation
antenna_data (FITS_rec) – The data from the second table of the observation
pyfhd_config (dict) – pyfhd’s configuration dictionary containing all the options for a run
logger (logging.Logger) – pyfhd’s logger

Returns:

layout – The antenna layout dictionary compatible with pyuvdata

Return type:

dict

See also

extract_header: Opens the UVFITS file and extracts the header and data, including the antenna_header and antenna_data.

pyfhd.data_setup.uvfits.create_params(pyfhd_header: dict, params_data: recarray, logger: Logger) → dict[source]

Given the extracted header, params data from the uvfits file, create the params dictionary to store the relevant visibility metadata

Parameters:

pyfhd_header (dict) – The resulting header fom the fits file stored in a dictonary
params_data (np.recarray) – The data from the fits file as taken from astropy.io.fits.getdata
logger (logging.Logger) – The pyfhd logger

Returns:

params – The visibility metadata stored as a dictionary (instead of recarray as a dict is faster)

Return type:

dict

Raises:

KeyError – If the UVFITS data returned doesn’t contain the variables then a KeyError will get thrown.

See also

astropy.io.fits.getdata: https://docs.astropy.org/en/stable/io/fits/api/files.html#getdata
extract_header: Extracts the header from the UVFITS file and returns the header and data

pyfhd.data_setup.uvfits.extract_header(pyfhd_config: dict, logger: Logger, model_uvfits=False) → tuple[dict, recarray, FITS_rec, Header][source]

Extract data from the uvfits header, the data extracted will contain metadata about the observation, antennas, visibilities

Parameters:

uvfits_path (str) – Path to the uvfits to open (either the data or the model)
pyfhd_config (dict) – This is the config created from the argparse
logger (logging.Logger) – The pyfhd logger
model_uvfits (bool) – If True, load in the model uvfits. If False, load in a data uvfits file, by default False

Returns:

pyfhd_header (dict) – The result from the extraction of the header of the UVFITS file, containing observation metadata mostly
params_data (np.recarray) – The data from the UVFITS file, containing the visibility metadata
antenna_data (FITS_rec) – The layout data which will be used in the create_layout function, mostly antenna metadata
antenna_header (Header) – The layout header which will be used in the create_layout function, mostly antenna metadatas

Raises:

KeyError – If the UVFITS file doesn’t contain all the data then a KeyError will be raised

pyfhd.data_setup.uvfits.extract_visibilities(pyfhd_header: dict, params_data: recarray, pyfhd_config: dict, logger: Logger) → tuple[ndarray[tuple[Any, ...], dtype[complex128]], ndarray[tuple[Any, ...], dtype[float64]]][source]

Extract the visibilities and their weights from the UVFITS data.

Parameters:

pyfhd_header (dict) – The resulting header fom the fits file stored in a dictonary
params_data (np.recarray) – The data from the fits file as taken from astropy.io.fits.getdata
pyfhd_config (dict) – This is the config created from the argprase
logger (logging.Logger) – The pyfhd Logger

Returns:

vis_arr (NDArray[np.complex128]) – The visibility array
vis_weights (NDArray[np.float64]) – The visibility weights array

See also

astropy.io.fits.getdata: https://docs.astropy.org/en/stable/io/fits/api/files.html#getdata
extract_header: Extracts the header from the UVFITS file and returns the header and data

pyfhd.data_setup.obs.create_obs(pyfhd_header: dict, params: dict, layout: dict, pyfhd_config: dict, logger: Logger) → dict[source]

create_obs takes all the data that has been read in and creates the obs data structure which holds data and metadata of the observation we’re doing a pyfhd run on. Inside this function the metafits file will be read as well.

Parameters:

pyfhd_header (dict) – The data from the UVFITS header
params (dict) – The data from the UVFITS file
layout (dict) – The data dictionary containing data and metadata about the antennas
pyfhd_config (dict) – pyfhd’s configuration dictionary
logger (logging.Logger) – The pyfhd logger

Returns:

obs – The observatiopn data structure for pyfhd containing data from the config and metadata from the observation UVFITS and METAFITS files.

Return type:

dict

pyfhd.data_setup.obs.project_slant_orthographic(meta: dict, obs: dict, epoch=2000) → dict[source]

Create an astrometry data structure holding key astrometry information. It’s essentially a WCS data structure, done as a Python dictionary allowing greater compatibility with other packages.

Parameters:

meta (dict) – The current metadata dictionary
obs (dict) – The current obs dictionary
epoch (float) – The equinox used for the dictionary structure

Returns:

astr – An astrometry structure built from meta and obs

Return type:

dict

pyfhd.data_setup.obs.read_metafits(obs: dict, pyfhd_header: dict, params: dict, pyfhd_config: dict, layout: dict, logger: Logger) → dict[source]

Reads the metafits file provided inside the same input directory as the UVFITS file. It will process the data found in the METAFITS file and then returns a meta dictionary. Which will eventually end up inside the obs dictionary.

Parameters:

obs (dict) – The current obs structure without the metadata
pyfhd_header (dict) – The data from the UVFITS header
params (dict) – The data from the UVFITS file
pyfhd_config (dict) – pyfhd’s configuration dictionary
layout (dict) – Info from the uvfits antenna table
logger (logging.Logger) – pyfhd’s logger

Returns:

meta – The dictionary holding the metadata from the UVFITS and METAFITS files

Return type:

dict

pyfhd.data_setup.obs.update_obs(obs: dict, dimension: int, kbinsize: float | int, beam_nfreq_avg: float | None = None, fov: float | None = None) → dict[source]

Inside the quickview function for exporting files we need to update the obs dictionary based on the new dimension and kbinsize given. This differs slightly from FHD as we only adjust the exact things required for this as we only use this function once in quickview.

Parameters:

obs (dict) – The original observation dictionary
dimension (int) – The new dimension for the size of each axes
kbinsize (float | int) – The new kbin
beam_nfreq_avg (float | None) – Set the new factor to average up the frequency resolution,by default None
fov (float | None) – Set a new field of view, by default None

Returns:

obs – The new updated obs dictionary

Return type:

dict

Beam Setup

pyfhd.beam_setup.beam.create_psf(obs: dict, pyfhd_config: dict, logger: Logger) → dict | File[source]

Creates the psf dictionary, which includes the hyperresolved uv-kernel used for gridding, by building the image-space response of a set of antennas and transforming it to uv-space, or by loading in a sav or HDF5 file from a FHD run. The base response of an antenna is retreived from a pyuvdata class/object. In the future it would be nice to have the ability to read in a beam fits file. Do note, that pyfhd was made with the assumption that the beam is the MWA beam, and assumes the beam does not differ on a per baseline basis. If you wish to use separate baselines you’ll need to add that functionality yourself.

Parameters:

obs (dict) – The observation metadata dictionary
pyfhd_config (dict) – The pyfhd configuration dictionary
logger (Logger) – pyfhd’s logger

Raises:

ValueError – If the beam file type is not recognized or if the beam file path is not set correctly.

Returns:

Return the psf dictionary containing the hyperresolved uv-beam and other metadata, or a h5py File object if lazy loading is enabled.

Return type:

dict | h5py.File

pyfhd.beam_setup.beam_utils.beam_image(psf: dict | File, obs: dict, pol_i: int, freq_i: int | None = None, abs=False, square=False) → ndarray[source]

Generates the average beam in image space for one polarization over all frequencies, or optionally for one frequency. The UV->sky transformation uses the inverse FFT for the beam, but the forward FFT for the image. This convention ensures the correct orientation of the UV-space beam for gridding visibilities. If the psf dictionary has Gaussian parameters, then the Gaussian decomposition is used to generate the analytic beam image.

Parameters:

psf (dict) – Beam dictionary
obs (dict) – Observation metadata dictionary
pol_i (int) – Index of the polarization to use
freq_i (int) – Index of the frequency to use, by default None
abs (bool, optional) – Return the absolute value of the beam image, by default False
square (bool, optional) – Return the square of the beam image, by default False

Returns:

beam_base – The average beam in image space for the specified polarization and all frequencies, or for a specific frequency if freq_i is set.

Return type:

np.ndarray

pyfhd.beam_setup.beam_utils.beam_image_hyperresolved(antenna: dict, beam: UVBeam | AnalyticBeam, ant_pol_1: int, ant_pol_2: int, freq_i: int, zen_int_x: float, zen_int_y: float, psf: dict) → ndarray[tuple[Any, ...], dtype[complexfloating]][source]

Build the hyperresolved image-space beam power for a station/tile. This is computed as the product of the F-matrices for the two antennas. The Jones matrices are decomposed into F and K, where F gives the complex sensitivity of each instrumental polarization to unpolarized emission (so it only has an instrumental polarization index, the phase is related to time delays which can vary spatially) and K is the projection from celestial polarization vector components (orthogonal on the sky) to instrumental polarization vector components (often non-orthogonal on the sky)

Parameters:

antenna (dict) – Antenna metadata dictionary
beam (UVBeam | AnalyticBeam) – UVBeam or AnalyticBeam object containing the beam model and metadata.
ant_pol_1 (int) – Polarization index for the first antenna
ant_pol_2 (int) – Polarization index for the second antenna
freq_i (int) – Frequency index for current iteration
zen_int_x (float) – x pixel index of the zenith for the beam power image
zen_int_y (float) – y pixel index of the zenith for the beam power image
psf (dict) – Beam metadata dictionary

Returns:

An estimation of the image-space beam power, normalized to the zenith power.

Return type:

NDArray[np.complexfloating]

pyfhd.beam_setup.beam_utils.beam_power(antenna: dict, beam: UVBeam | AnalyticBeam, ant_pol_1: int, ant_pol_2: int, freq_i: int, psf: dict, zen_int_x: float, zen_int_y: float, xvals_uv_superres: ndarray, yvals_uv_superres: ndarray, pyfhd_config: dict) → ndarray[tuple[Any, ...], dtype[complexfloating]][source]

Generate the hyperresolved image-space beam power to reduce aliasing artifacts, and fourier transform it to a specific grid in complex uv-space. Reduce artifacts further by applying an extremely low-level contiguous mask to the uv-space beam power and renomalizing the beam power to a volume of 1.

Parameters:

antenna (dict) – Antenna metadata dictionary
beam (UVBeam | AnalyticBeam) – UVBeam or AnalyticBeam object containing the beam model and metadata.
ant_pol_1 (int) – Polarization index for the first antenna
ant_pol_2 (int) – Polarization index for the second antenna
freq_i (int) – Frequency index for current iteration
psf (dict) – Beam metadata dictionary
zen_int_x (np.ndarray) – x pixel index of the zenith for the beam power image
zen_int_y (np.ndarray) – y pixel index of the zenith for the beam power image
xvals_uv_superres (np.ndarray) – A grid of the hyperresolved indices in the u direction for the uv-space beam power image
yvals_uv_superres (np.ndarray) – A grid of the hyperresolved indices in the v direction for the uv-space beam power image
pyfhd_config (dict) – The pyfhd configuration dictionary

Returns:

Hyperresolved uv-space of the beam power image, normalized to a volume of 1

Return type:

NDArray[np.complexfloating]

pyfhd.beam_setup.beam_utils.gaussian_decomp(x: ndarray, y: ndarray, p: ndarray, ftransform: bool = False, model_npix: float | None = None, model_res: float | None = None) → tuple[ndarray, float, float][source]

Create an analytically built Gaussian decomposition of the beam on the supplied x-y grid given the input Gaussian parameters. If ftransform is True, then the analytic Fourier Transform of the input gaussians on the supplied x-y grid is returned. Any number of Gaussians can be supplied in the p vector. To transfer Gaussian parameters from a different grid to the current x-y grid, the model_npix and model_res parameters can be supplied.

Parameters:

x (np.ndarray) – X-axis vector of pixel numbers
y (np.ndarray) – Y-axis vector of pixel numbers
p (np.ndarray) – Gaussian parameter vector, ordered as amp, offset x, sigma x, offset y, sigma y per lobe
ftransform (bool, optional) – Return the analytic Fourier Transform of the input gaussians on the supplied x-y grid, by default False
model_npix (float | None, optional) – The number of pixels on an axis used to derive the input parameters to convert to the current x-y grid, by default None
model_res (float | None, optional) – The grid resolution used to derive the input parameters to convert to the current grid resolution, by default None

Returns:

The Gaussian decomposition of the beam on the supplied x-y grid given the input Gaussian parameters, along with the analytic volume and squared volume of the Gaussian decomposition.

Return type:

tuple[np.ndarray, float, float]

Source Modeling

pyfhd.source_modeling.vis_model_transfer.convert_vis_model_arr_to_sav(vis_model_arr: ndarray[tuple[Any, ...], dtype[complex128]], pyfhd_config: dict, logger: Logger, model_vis_dir: str, n_pol: int)[source]

Converts the contents of vis_model_arr into an FHD .sav file format so we can import into existing IDL code with ease. First writes data to hdf5 format, then uses IDL code template to convert to IDL .sav format compatible with FHD. Sticks the outputs into model_vis_dir.

Parameters:

vis_model_arr (NDArray[np.complex128]) – Complex array hold the model visibilities
pyfhd_config (dict) – The options from argparse in a dictionary, that have been verified using pyfhd.pyfhd_tools.pyfhd_setup.pyfhd_setup.
logger (logging.Logger) – pyfhd logger to feed information to
model_vis_dir (str) – Directory location to write the output files to
n_pol (int) – Number of polarisations to write out (each is written to an individual) .sav file

pyfhd.source_modeling.vis_model_transfer.flag_model_visibilities(vis_model_arr: ndarray[tuple[Any, ...], dtype[complex128]], params_data: dict, params_model: dict, obs: dict, pyfhd_config: dict, logger: Logger) → ndarray[tuple[Any, ...], dtype[complex128]][source]

Account for time offset and tile flags, and check that the uvfits and compatible. Needs to check if auto-correlations are present

Parameters:

vis_model_arr (NDArray[np.complex128]) – The model visibility array from the model uvfits file
params_data (dict) – The params data from the observation uvfits
params_model (dict) – The params data from the model uvfits
obs (dict) – The observaton dictionary containing all the data from the observation uvfits file
pyfhd_config (dict) – The pyfhd configuration dictionary
logger (logging.Logger) – The pyfhd logger

Returns:

vis_model_arr_flagged – The flagged model visibility array

Return type:

NDArray[np.complex128]

pyfhd.source_modeling.vis_model_transfer.import_vis_model_from_sav(pyfhd_config: dict, obs: dict, logger: Logger) → tuple[ndarray[tuple[Any, ...], dtype[complex128]], dict][source]

Read a model visibility array in from multiple IDL sav files which are in a directory given by pyfhd_config[‘model-file-path’]. The data is assumed to be in the format of <obs_id>_params.sav, <obs_id>_vis_model_<pol_name>.sav. The pol_name follows the pol_names in the obs dictionary which are [‘XX’,’YY’,’XY’,’YX’,’I’,’Q’,’U’,’V’].

Parameters:

pyfhd_config (dict) – The pyfhd config dictionary
obs (dict) – The dictionary containing observation data and metadata
logger (logging.Logger) – pyfhd’s logger

Returns:

vis_model_arr (NDArray[np.complex128]) – Simulated model for the visibilities
params_model (dict) – The parameters for said model used for flagging

pyfhd.source_modeling.vis_model_transfer.import_vis_model_from_uvfits(pyfhd_config: dict, obs: dict, logger: Logger) → tuple[ndarray[tuple[Any, ...], dtype[complex128]], dict][source]

Read a model visibility array in from a uvfits with filepath given by pyfhd_config[‘model_file_path’]. Reads data in via pyfhd.data_setup.uvfits import extract_visibilities.

Parameters:

pyfhd_config (dict) – The options from argparse in a dictionary, that have been verified using pyfhd.pyfhd_tools.pyfhd_setup.pyfhd_setup.
obs (dict) – The observation dictionary as populated by pyfhd.data_setup.obs.create_obs

Returns:

vis_model_arr (NDArray[np.complex128]) – Simulated model for the visibilities
params_model (dict) – The parameters for said model used for flagging

pyfhd.source_modeling.vis_model_transfer.vis_model_transfer(pyfhd_config: dict, obs: dict, params: dict, logger: Logger) → tuple[ndarray[tuple[Any, ...], dtype[complex128]], dict][source]

Transfer in a simulated model of the visibilities from either a sav file or uvfits file.

Parameters:

pyfhd_config (dict) – pyfhd’s configuration dictionary containing all the options for a pyfhd run
obs (dict) – The Observation Metadata dictionary
params (dict) – Visibility metadata dictionary
logger (logging.Logger) – pyfhd’s logger

Returns:

vis_model_arr – Simulated model for the visibilities

Return type:

NDArray[np.complex128]

See also

pyfhd.source_modeling.vis_model_transfer.import_vis_model_from_sav: Import model from a sav file
pyfhd.source_modeling.vis_model_transfer.import_vis_model_from_uvfits: Import model from a uvfits file

Calibration

calibrate

pyfhd.calibration.calibrate.calibrate(obs: dict, params: dict, vis_arr: ndarray[tuple[Any, ...], dtype[complex128]], vis_weights: ndarray[tuple[Any, ...], dtype[float64]], vis_model_arr: ndarray[tuple[Any, ...], dtype[complex128]], pyfhd_config: dict, logger: Logger) → tuple[ndarray[tuple[Any, ...], dtype[complex128]], dict, dict][source]

Solve for the amplitude and phase of the electronic response of each tile or station, and apply these calibration solutions to the raw, data visiblities. Various options for initial estimates, time/tile averaging, and polynomial/cable reflections fitting are available.

Parameters:

obs (dict) – Observation metadata dictionary
params (dict) – Visibility metadata dictionary
vis_arr (NDArray[np.complex128]) – Uncalibrated data visiblities
vis_weights (NDArray[np.float64]) – Weights (flags) of the visibilities
vis_model_arr (NDArray[np.complex128]) – Simulated model visibilites
pyfhd_config (dict) – pyfhd’s configuration dictionary containing all the options set for a pyfhd run
logger (Logger) – pyfhd’s logger for displaying errors and info to the log files

Returns:

vis_cal (tuple[NDArray[np.complex128]) – The calibrated data visibilities
cal (dict) – The updated calibration dictionary
obs (dict) – Updated observation metadata dictionary
pyfhd_config (dict) – Updated pyfhd configuration dictionary (possibly - check for any warnings in the log)

pyfhd.calibration.calibrate.calibrate_qu_mixing(vis_arr: ndarray[tuple[Any, ...], dtype[complex128]], vis_model_arr: ndarray[tuple[Any, ...], dtype[complex128]], vis_weights: ndarray[tuple[Any, ...], dtype[float64]], obs: dict) → float[source]

Solve for the degenerate phase between pseudo Q (YY - XX) and pseudo U (YX + XY) for the calibrated data and the simulated model separately, and return their difference. This difference represents the excess mixing angle between Q and U due to the instrumental beam not captured in a typical polarization-independent calibration.

Parameters:

vis_arr (np.ndarray) – Uncalibrated data visiblities
vis_model_arr (np.ndarray) – Simulated model visibilites
vis_weights (np.ndarray) – Weights (flags) of the visibilities
obs (dict) – Observation metadata dictionary

Returns:

stokes_mix_phase – The excess mixing angle between Q and U from instrumental effects

Return type:

float

calibration_utils

pyfhd.calibration.calibration_utils.cal_auto_ratio_divide(obs: dict, cal: dict, vis_auto: ndarray[tuple[Any, ...], dtype[float64]], auto_tile_i: ndarray[tuple[Any, ...], dtype[integer]]) → tuple[dict, ndarray[tuple[Any, ...], dtype[float64]]][source]

Remove antenna-dependent parameters (i.e. cable reflections) from the calculated gains to reduce the bias on individual tile variation before the creation of averaged quantities like the global bandpass. Antenna-dependent parameters are estimated from the square root of the autocorrelation visibilities normalized via a reference tile.

Parameters:

obs (dict) – Observation metadata dictionary
cal (dict) – Calibration dictionary
vis_auto (NDArray[np.float64]) – Data autocorrelations
auto_tile_i (NDArray[np.integer]) – Index array of the tile array that have defined autocorrelations

Returns:

cal (dict) – Calibration dictionary with gains that have reduced antenna-dependent bias
auto_ratio (NDArray[np.float64]) – The estimation of the antenna-dependent bias through the square root of the autocorrelation visibilities normalized via a reference tile.

pyfhd.calibration.calibration_utils.cal_auto_ratio_remultiply(cal: dict, auto_ratio: ndarray[tuple[Any, ...], dtype[float64]], auto_tile_i: ndarray[tuple[Any, ...], dtype[integer]]) → dict[source]

Return antenna-dependent parameters to the calculated gains after averaged quantities have been calculated. The antenna-dependent parameters are captured via the square root of the referenced autocorrelation visiblities calculated in cal_auto_ratio_divide.

Parameters:

obs (dict) – Observation metadata dictionary
cal (dict) – Calibration dictionary
auto_ratio (NDArray[np.float64]) – Square root of the autocorrelation visibilities normalized via a reference tile
auto_tile_i (NDArray[np.integer]) – Index array of the tile array that have defined autocorrelations

Returns:

cal – Calibration dictonary containing the reformed gain

Return type:

dict

pyfhd.calibration.calibration_utils.calculate_adaptive_gain(gain_list: ndarray[tuple[Any, ...], dtype[float64]], convergence_list: ndarray[tuple[Any, ...], dtype[float64]], iter: int, base_gain: int | float, final_convergence_estimate: float | None = None)[source]

Perform a Kalman filter to calculate the best weighting to use in the next iteration of the linear least squares fitting between the data and simulated model, which reduces the number of total iterations till convergence.

The Kalman filter takes previous convergence values, or the percent change in calibration solution from the previous iteration, to calculate the relative weight to give to the next calculated iteration versus the previous iteration. For example, a gain (relative weight in this context) of 1 would give the old solution and the new solution the same weighting, and thus the next guess in the linear least square solver is the average of the new and old solution. A gain (relative weight) of 2 would give the new solution twice as much weight at the old solution, and thus the next guess in the linear least square solver is the sum of the old solution and twice the new solution, all divided by three.

Parameters:

gain_list (NDArray[np.float64]) – Relative weighting between the previous iteration and new iteration in the linear least squares solver for calibration solutions
convergence_list (NDArray[np.float64]) – An array of the percent change in the calibration solutions between one iteration and the next
iter (int) – The current iteration in the linear least squares solver
base_gain (int|float) – The initial relative weighting between the previous iteration and new iteration in the linear least squares solver for calibration solutions
final_convergence_estimate (float, optional) – The prediction of the percent change of the previous iteration and the forward model estimate from the Kalman filter, by default None

Returns:

gain – Relative weighting between the previous iteration and the new iteration in the calibration linear least squares solver

Return type:

float

pyfhd.calibration.calibration_utils.transfer_bandpass(obs: dict, cal: dict, pyfhd_config: dict, logger: Logger) → tuple[dict, dict][source]

Apply a previously saved bandpass via a calfits file (github:pyuvdata). Check adherance to standards, and match the polarizations, frequencies, timing, pointings, and tiles.

Parameters:

obs (dict) – Observation metadata dictionary
cal (dict) – Calibration dictionary
pyfhd_config (dict) – pyfhd’s configuration dictionary containing all the options set for a pyfhd run
logger (Logger) – pyfhd’s logger for displaying errors and info to the log files

Returns:

cal_bandpass (dict) – Calibration dictionary with bandpass gains
cal_remainder (dict) – calibration dictionary with residuals after removing the bandpass gains

pyfhd.calibration.calibration_utils.vis_baseline_hist(obs: dict, params: dict, vis_cal: ndarray[tuple[Any, ...], dtype[complex128]], vis_model_arr: ndarray[tuple[Any, ...], dtype[complex128]]) → dict[source]

Create diagnostic histograms of both the mean and sigma for the percent difference between calibrated data and simulated model visibilities as a function of baseline length.

Parameters:

obs (dict) – Observation metadata dictionary
params (dict) – Visibility metadata dictionary
vis_cal (NDArray[np.complex128]) – Calibrated data visibilities
vis_model_arr (NDArray[np.complex128]) – Simulated model visibilites

Returns:

vis_baseline_hist – Dictionary of the mean and sigma histograms for the percent difference between calibrated data and simulated model visibilities as a function of baseline length.

Return type:

dict

pyfhd.calibration.calibration_utils.vis_cal_auto_fit(obs: dict, cal: dict, vis_auto: ndarray[tuple[Any, ...], dtype[float64]], vis_auto_model: ndarray[tuple[Any, ...], dtype[float64]], auto_tile_i: ndarray[tuple[Any, ...], dtype[integer]]) → dict[source]

Solve for each tile’s calibration amplitude via the square root of the ratio of the data autocorrelation to the model autocorrelation using the definition of a gain. Then, remove dependence on the correlated noise term in the autocorrelations by scaling this amplitude down to the crosscorrelations gain via a simple, linear fit. Build a full, scaled autocorrelation gain solution by adding in the phase term via the crosscorrelation gains.

Parameters:

obs (dict) – Observation metadata dictionary
cal (dict) – Calibration dictionary
vis_auto (NDArray[np.float64]) – Data autocorrelations
vis_auto_model (NDArray[np.float64]) – Simulated model autocorrelations
auto_tile_i (NDArray[np.integer]) – Index array of the tile array that have defined autocorrelations

Returns:

cal – Calibration dictionary with scaled autocorrelation gains

Return type:

dict

pyfhd.calibration.calibration_utils.vis_cal_auto_init(obs: dict, cal: dict, vis_arr: ndarray[tuple[Any, ...], dtype[complex128]], vis_model_arr: ndarray[tuple[Any, ...], dtype[complex128]], vis_auto: ndarray[tuple[Any, ...], dtype[float64]], vis_auto_model: ndarray[tuple[Any, ...], dtype[float64]], auto_tile_i: ndarray[tuple[Any, ...], dtype[integer]]) → ndarray[tuple[Any, ...], dtype[complex128]][source]

Initialize the calibration solutions using the autocorrelations prior to the linear least squares fit for faster convergence. The auto-correlations and cross-correlations have separate formulisms for the initialization.

Autos – Square root of “Ratio of data autos to model autos” x “overall mean for both crosses and autos of ratio of data to model” / “overall mean of ratio of data autos to model autos, with 1’s for crosses”, results in a per-frequency, per-baseline gain scaling. Crosses – Square root of “overall mean for both crosses and autos of ratio of data to model” / “overall mean of ratio of data autos to model autos, with 1’s for crosses”, results in a single number gain scaling.

Parameters:

obs (dict) – Observation metadata dictionary
cal (dict) – Calibration dictionary
vis_arr (NDArray[np.complex128]) – Uncalibrated data visiblities
vis_model_arr (NDArray[np.complex128]) – Simulated model visibilites
vis_auto (NDArray[np.float64]) – Data auto-correlations
vis_auto_model (NDArray[np.float64]) – Simulated model auto-correlations
auto_tile_i (NDArray[np.integer]) – Index array for auto-correlation visibilities

Returns:

auto_gain – Initialization gain array

Return type:

NDArray[np.complex128]

pyfhd.calibration.calibration_utils.vis_cal_bandpass(obs: dict, cal: dict, pyfhd_config: dict, logger: Logger) → tuple[dict, dict][source]

Reduce the degrees of freedom on the per-frequency calibration amplitudes by averaging solutions together. Options include averaging over tiles which use a particular beamformer-to-receiver cable lengths/types, or averaging over all tiles for a global bandpass.

Parameters:

obs (dict) – Observation metadata dictionary
cal (dict) – Calibration dictionary
pyfhd_config (dict) – pyfhd’s configuration dictionary containing all the options set for a pyfhd run
logger (Logger) – pyfhd’s logger for displaying errors and info to the log files

Returns:

cal_bandpass (dict) – Calibration dictionary with bandpass gains
cal_remainder (dict) – calibration dictionary with residuals after removing the bandpass gains

pyfhd.calibration.calibration_utils.vis_cal_polyfit(obs: dict, cal: dict, auto_ratio: ndarray[tuple[Any, ...], dtype[float64]] | None, pyfhd_config: dict, logger: Logger) → dict[source]

Reduce the degrees of freedom on the per-frequency calibration amplitudes and phases by fitting the full frequency band with polynomials of a specified degree, with options for split polynomials over certain regions.

In addition, fit cable reflections with a text file of fits, theoretical fits given cable length/type, or the finding the maximum in delay space. Any of these can then be used as an initial estimate in a hyperresolved FFT of the residual calibration solutions to fit the final mode, phase, and amplitude. Residual calibration solutions can optionally be made by using an incoherent mean, or a mean over all tiles which do not have the same cable length/type, to reduce bias in the residual, and the cable reflections can also be fit using just the phases to reduce dependency on the polyphase filter bank shape.

Parameters:

obs (dict) – Observation metadata dictionary
cal (dict) – Calibration dictionary
auto_ratio (NDArray[np.float64] | None) – The estimation of the antenna-dependent bias through the square root of the autocorrelation visibilities normalized via a reference tile
pyfhd_config (dict) – pyfhd’s configuration dictionary containing all the options set for a pyfhd run
logger (Logger) – pyfhd’s logger for displaying errors and info to the log files

Returns:

cal (dict) – Calibration dictionary with polynomial gain fits
pyfhd_config (dict) – pyfhd’s configuration dictionary containing all the options set for a pyfhd run Is returned in case the digital_gain_jump_polyfit was set here.

pyfhd.calibration.calibration_utils.vis_calibration_apply(vis_arr: ndarray[tuple[Any, ...], dtype[complex128]], obs: dict, cal: dict, vis_model_arr: ndarray[tuple[Any, ...], dtype[complex128]], vis_weights: ndarray[tuple[Any, ...], dtype[float64]], logger: Logger) → tuple[ndarray[tuple[Any, ...], dtype[complex128]], dict][source]

Apply the calibration solutions to the input, data visibilities to create calibrated, data visibilities using the definition of the gains.

Definition of the gain: (visibility for baseline of tile i and tile j) = (gain of tile i) (gain of tile j) (model visibility for baseline of tile i and tile j)

If only two othogonal polarizations were used to calibrate, calculate the phase offset between the two orthogonal dipoles to solve for a degeneracy in the cross polarizations. The formula can be derived by multiplying the gains by an equal and opposite phase in the linear least square solver and solving for the phase when the partial derivative w.r.t the offset is 0.

Parameters:

vis_arr (NDArray[np.complex128]) – Uncalibrated data visiblities
obs (dict) – Observation metadata dictionary
cal (dict) – Calibration dictionary
vis_model_arr (NDArray[np.complex128]) – Simulated model visibilites
vis_weights (NDArray[np.float64]) – Weights (flags) of the visibilities
logger (Logger) – pyfhd’s logger for displaying errors and info to the log files

Returns:

vis_arr (NDArray[np.complex128]) – Calibrated visibilities
cal (dict) – Calibration dictionary

pyfhd.calibration.calibration_utils.vis_calibration_flag(obs: dict, cal: dict, pyfhd_config: dict, logger: Logger) → dict[source]

Flag tile and frequency outliers based on the calibration solutions. First, iteratively flag a maximum of three times on amplitude with three tests: 1) flag frequencies above 5 sigma, 2) flag tiles above 5 sigma, and 3) flag tiles either 2x lower or 2x higher than average. Second, iteratively flag a maximum of three times on phase with two tests: 1) flag tiles with slopes above 5 sigma, and 2) flag tiles with per-frequency deviations from their slope above 5 sigma.

Parameters:

obs (dict) – Observation metadata dictionary
cal (dict) – Calibration dictionary
pyfhd_config (dict) – pyfhd’s configuration dictionary containing all the options set for a pyfhd run
logger (Logger) – pyfhd’s logger for displaying errors and info to the log files

Returns:

obs – Observation metadata dictionary

Return type:

dict

pyfhd.calibration.calibration_utils.vis_extract_autocorr(obs: dict, vis_arr: ndarray[tuple[Any, ...], dtype[complex128]], pyfhd_config: dict, auto_tile_i: ndarray[tuple[Any, ...], dtype[integer]] | None = None) → tuple[ndarray[tuple[Any, ...], dtype[float64]], ndarray[tuple[Any, ...], dtype[integer]]][source]

Extract the auto-correlations if they exist from the full visibility array.

Parameters:

obs (dict) – Observation metadata dictionary
vis_arr (NDArray[np.complex128]) – Uncalibrated data visiblities
pyfhd_config (dict) – pyfhd’s configuration dictionary containing all the options set for a pyfhd run
auto_tile_i (NDArray[np.integer] | None, optional) – Index array for auto-correlation visibilities, by default None

Returns:

autocorr (NDArray[np.float64]) – The auto-correlation visibilities
auto_tile_i (NDArray[np.integer]) – The index array for auto-correlation visibilities

vis_calibrate_subroutine

pyfhd.calibration.vis_calibrate_subroutine.vis_calibrate_subroutine(vis_arr: ndarray[tuple[Any, ...], dtype[complex128]], vis_model_ptr: ndarray[tuple[Any, ...], dtype[complex128]], vis_weight_ptr: ndarray[tuple[Any, ...], dtype[float64]], obs: dict, cal: dict, params: dict, pyfhd_config: dict, logger: Logger, calibration_weights=False, no_ref_tile=False)[source]

Perform a linear least-squares regression between the data visilbities and the simulated model visiblities to solve for an amplitude and phase for each tile for each frequency channel.

Parameters:

vis_arr (NDArray[np.complex128]) – Uncalibrated data visiblities
vis_model_ptr (NDArray[np.complex128]) – Simulated model visibilites
vis_weight_ptr (NDArray[np.float64]) – Weights (flags) of the visibilities
obs (dict) – Observation metadata dictionary
cal (dict) – Calibration dictionary
pyfhd_config (dict) – pyfhd’s configuration dictionary containing all the options set for a pyfhd run
calibration_weights (bool, optional) – Weight the visibilities at the minimum and maximum baseline by a soft taper for calibration purposes only, by default False
no_ref_tile (bool, optional) – Do not reference calibration phases at this stage, by default False

Returns:

cal – Calibration dictionary

Return type:

dict

Raises:

ValueError – Should almost never happen, only gets raised for max_cal_iter has been set to less than 5, however since it’s currently hardcoded in here it’s more of an exception for any developers of pyfhd

Gridding

Gridding Utilities

pyfhd.gridding.gridding_utils.baseline_grid_locations(obs: dict, psf: dict, params: dict, vis_weights: ndarray[tuple[Any, ...], dtype[float64]], logger: Logger, bi_use: ndarray[tuple[Any, ...], dtype[integer]] | None = None, fi_use: ndarray[tuple[Any, ...], dtype[integer]] | None = None, fill_model_visibilities: bool = False, interp_flag: bool = False, mask_mirror_indices: bool = False) → dict[source]

Calculate the histogram of baseline grid locations in units of pixels whilst also returning the minimum pixel number that an unflagged baseline contributes to (depending on the size of the kernel). Optionally return the 2D derivatives for bilinear interpolation and the indices of the unflagged baselines/frequencies.

Parameters:

obs (dict) – Observation metadata dictionary
psf (dict) – Beam metadata dictionary
params (dict) – Visibility metadata dictionary
vis_weights (NDArray[np.float64]) – Weights (flags) of the visibilities
logger (Logger) – pyfhd’s logger
bi_use (NDArray[np.integer] | None, optional) – Baseline index array for gridding, i.e even vs odd time stamps, by default None
fi_use (NDArray[np.integer] | None, optional) – Frequency index array for gridding, i.e. gridding all frequencies for continuum images, by default None
fill_model_visibilities (bool, optional) – Calculate baseline grid locations for all baselines, regardless of flags, by default False
interp_flag (bool, optional) – Calculate derivatives for bilinear interpolation of the kernel to pixel locations, by default False
mask_mirror_indices (bool, optional) – Exclude baselines mirrored along the v-axis, by default False

Returns:

baselines_dict – Histogram of baseline grid locations, associated derivatives, and minimum contributing pixel location, arranged in a dictionary

Return type:

dict

pyfhd.gridding.gridding_utils.conjugate_mirror(image: ndarray[tuple[Any, ...], dtype[complex128 | float64]]) → ndarray[tuple[Any, ...], dtype[complex128 | float64]][source]

Mirror an image about the origin and take the complex conjugate. The origin is considered to be the first row, and thus is not repeated in the conjugate mirror.

Parameters:: image (NDArray[np.complex128 | np.float64]) – A 2D array of real or complex numbers
Returns:: conj_mirror_image – The 2D conjugate mirror of the input without the origin
Return type:: NDArray[np.complex128 | np.float64]

pyfhd.gridding.gridding_utils.crosspol_reformat(image_uv: ndarray[tuple[Any, ...], dtype[complex128]]) → ndarray[tuple[Any, ...], dtype[complex128]][source]

Reformat the cross-polarizations (i.e. XY and YX) as pseudo Stokes Q and U. This helps to avoid complex numbers in creating images – however, this is an imperfect assumption.

Parameters:: image_uv (NDArray[np.complex128]) – A 2D {u,v} plane in four linear polarizations.
Returns:: image_uv – A 2D {u,v} plane in two linear polarizations, pseudo Stokes Q, and pseudo Stokes U.
Return type:: NDArray[np.complex128]

pyfhd.gridding.gridding_utils.dirty_image_generate(dirty_image_uv: ndarray[tuple[Any, ...], dtype[complex128]], pyfhd_config: dict, logger: Logger, uniform_filter_uv: ndarray[tuple[Any, ...], dtype[float64]] | None = None, mask: ndarray[tuple[Any, ...], dtype[integer]] | None = None, baseline_threshold: int | float | None = None, normalization: float | ndarray[tuple[Any, ...], dtype[float64]] | None = None, resize: int | None = None, width_smooth: int | float | None = None, degpix: float | None = None, not_real: bool = False, pad_uv_image: int | float | None = None, weights: ndarray[tuple[Any, ...], dtype[float64]] | None = None, filter: ndarray[tuple[Any, ...], dtype[float64]] | None = None, beam_ptr: ndarray[tuple[Any, ...], dtype[complex128]] | None = None) → tuple[ndarray[tuple[Any, ...], dtype[complex128]], ndarray[tuple[Any, ...], dtype[float64]], float | ndarray[tuple[Any, ...], dtype[float64]] | None][source]

Generate a projected image from an input {u,v} plane through a 2D FFT. Optionally apply padding, masking, filtering, and more.

Parameters:

dirty_image_uv (NDArray[np.complex128]) – A 2D {u,v} plane which generally includes the beam via a gridding kernel
pyfhd_config (dict) – pyfhd’s configuration dictionary containing all the options set for a pyfhd run
logger (Logger) – pyfhd’s logger
uniform_filter_uv (NDArray[np.float64] | None, optional) – A 2D {u,v} gridded visibility number, by default None
mask (NDArray[np.integer] | None, optional) – A 2D {u,v} mask to apply before image creation, by default None
baseline_threshold (int | float, optional) – The maximum baseline length to include in units of pixels, default None
normalization (float | NDArray[np.float64] | None, optional) – A value by which to normalize the image by, by default None
resize (int | None, optional) – Increase the number of pixels by a factor and rebin the input {u,v} plane, by default None
width_smooth (int | float | None, optional) – Smooth out the harsh baseline threshold by the given number of pixels, by default None
degpix (float | None, optional) – Degrees per pixel, by default None
not_real (bool, optional) – Flag to return a complex image, by default False
pad_uv_image (int | float | None, optional) – Pad the {u,v} plane by this pixel amount to increase perceived resolution of the image, by default None
weights (NDArray[np.float64] | None, optional) – Gridded {u,v} plane of visibility weights, necessary in some filtering schemes, by default None
filter (NDArray[np.float64] | None, optional) – Image filter to apply, by default None
beam_ptr (NDArray[np.complex128] | None, optional) – Weight by an additional factor of the beam for optimal weighting, by default None

Returns:

dirty_image (NDArray[np.complex128]) – A 2D {l,m} directional-cosine image plane
filter (NDArray[np.float64]) – The filter applied to the dirty image
normalization (float | NDArray[np.float64] | None) – The normalization (if any) applied to the dirty image

pyfhd.gridding.gridding_utils.grid_beam_per_baseline(psf: dict, pyfhd_config: dict, logger: Logger, uu: ndarray[tuple[Any, ...], dtype[float64]], vv: ndarray[tuple[Any, ...], dtype[float64]], ww: ndarray[tuple[Any, ...], dtype[float64]], l_mode: ndarray[tuple[Any, ...], dtype[float64]], m_mode: ndarray[tuple[Any, ...], dtype[float64]], n_tracked: ndarray[tuple[Any, ...], dtype[float64]], frequency_array: ndarray[tuple[Any, ...], dtype[float64]], x: ndarray[tuple[Any, ...], dtype[float64]], y: ndarray[tuple[Any, ...], dtype[float64]], xmin_use: int, ymin_use: int, freq_i: ndarray[tuple[Any, ...], dtype[integer]], bt_index: ndarray[tuple[Any, ...], dtype[integer]], polarization: int, image_bot: int, image_top: int, psf_dim3: int, box_matrix: ndarray[tuple[Any, ...], dtype[complex128]], vis_n: int, beam_int: ndarray[tuple[Any, ...], dtype[complex128]] | None = None, beam2_int: ndarray[tuple[Any, ...], dtype[complex128]] | None = None, n_grp_use: ndarray[tuple[Any, ...], dtype[integer]] | None = None, degrid_flag: bool = False)[source]

Calculate the contribution of each baseline to the static {u,v} grid using the corrective phases in image-space, as detailed in J. Line’s thesis “PUMA and MAJICK: cross-matching and imaging techniques for a detection of the epoch of reionisation”

Parameters:

psf (dict) – Beam metadata dictionary
pyfhd_config (dict) – pyfhd’s configuration dictionary containing all the options set for a pyfhd run
logger (Logger) – pyfhd’s logger
uu (NDArray[np.float64]) – 1D array of the u-coordinate of selected baselines in light travel time
vv (NDArray[np.float64]) – 1D array of the v-coordinate of selected baselines in light travel time
ww (NDArray[np.float64]) – 1D array of the w-coordinate of selected baselines in light travel time
l_mode (NDArray[np.float64]) – Directional-cosine l of pixel centers of the hyperresolved beam
m_mode (NDArray[np.float64]) – Directional-cosine m of pixel centers of the hyperresolved beam
n_tracked (NDArray[np.float64]) – Directional-cosine n of pixel centers of the phase-tracked hyperresolved beam
frequency_array (NDArray[np.float64]) – Array of selected frequencies in Hz
x (NDArray[np.float64]) – 1D array of gridding extent and resolution in the x-direction in wavelengths
y (NDArray[np.float64]) – 1D array of gridding extent and resolution in the y-direction in wavelengths
xmin_use (int) – The minimum x-pixel that each selected baseline contributes to
ymin_use (int) – The minimum y-pixel that each selected baseline contributes to
freq_i (NDArray[np.integer]) – The current frequency index
bt_index (NDArray[np.integer]) – The current baseline/time index
polarization (int) – The current polarization index
image_bot (int) – The bottom-most pixel index that the image-space hyperresolved beam contributes to
image_top (int) – The top-most pixel index that the image-space hyperresolved beam contributes to
psf_dim3 (int) – The pixel area of the psf footprint on the {u,v} grid
box_matrix (NDArray[np.complex128]) – A 2D array of the number of visibilities to grid and the area of each visibility on the static {u,v} grid
vis_n (int) – The number of visibilities to grid
beam_int (NDArray[np.complex128] | None, optional) – The integral of the beam sensitivity in {u,v} space, by default None
beam2_int (NDArray[np.complex128] | None, optional) – The integral of the squared beam sensitivity in {u,v} space, by default None
n_grp_use (NDArray[np.integer] | None, optional) – The number of baselines in the current grouping, by default None
degrid_flag (bool, optional) – Perform degridding instead of gridding, by default False

Returns:

box_matrix – The kernel values on the static {u,v} grid for each visibility

Return type:

NDArray[np.complex128]

pyfhd.gridding.gridding_utils.holo_mapfn_convert(map_fn, psf_dim, dimension, elements=None, norm=1, threshold=0)[source]

Convert pointer array holographic map function to a sparse matrix. May need to be deprecated. The mapping functions were not translated into visibility_grid at the time of translation as it wasn’t clear what pyfhd was going to use for sparse/large arrays at the time. If you wish to implement the mapping functions, I suggest using HDF5 chunk loading for the mapping function.

Parameters:

map_fn (np.ndarray) – Pointer array holographic map function
psf_dim (int, float) – The number of pixels in one direction of the psf {u,v} footprint
dimension (int) – The number of pixels in the u-direction
elements (None, optional) – The number of pixels in the v-direction
norm (int) – The normalization of the holographic mapping function (i.e. the number of visibilities)
threshold (int, float, optional) – Include values from the holographic map function which are above a provided threshold

Returns:

map_fn – Sparse array holographic map function

Return type:

np.recarray

pyfhd.gridding.gridding_utils.interpolate_kernel(kernel_arr: ndarray[tuple[Any, ...], dtype[complex128]], x_offset: ndarray[tuple[Any, ...], dtype[integer]], y_offset: ndarray[tuple[Any, ...], dtype[integer]], dx0dy0: ndarray[tuple[Any, ...], dtype[float64]], dx1dy0: ndarray[tuple[Any, ...], dtype[float64]], dx0dy1: ndarray[tuple[Any, ...], dtype[float64]], dx1dy1: ndarray[tuple[Any, ...], dtype[float64]]) → ndarray[tuple[Any, ...], dtype[complex128]][source]

Perform a bilinear interpolation given the 2D derivatives of the baseline center to the nearest hyperresolved pixel edge in {u,v} space. This will provide a quadratic estimate at the sample location to smooth out the dependence on hyperresolved pixel size.

Parameters:

kernel_arr (NDArray[np.complex128]) – The 2D kernel for bilinear interpolation
x_offset (NDArray[np.integer]) – The nearest pixel offset in the x (u) direction
y_offset (NDArray[np.integer]) – The nearest pixel offset in the y (v) direction
dx0dy0 (NDArray[np.float64]) – (1 - derivative to pixel edge in x) * (1 - derivative to pixel edge in y) for selected baselines
dx1dy0 (NDArray[np.float64]) – (derivative to pixel edge in x) * (1 - derivative to pixel edge in y) for selected baselines
dx0dy1 (NDArray[np.float64]) – (1 - derivative to pixel edge in x) * (derivative to pixel edge in y) for selected baselines
dx1dy1 (NDArray[np.float64]) – (derivative to pixel edge in x) * (derivative to pixel edge in y) for selected baselines

Returns:

kernel – The interpolated 2D kernel

Return type:

NDArray[np.complex128]

pyfhd.gridding.gridding_utils.visibility_count(obs: dict, psf: dict, params: dict, vis_weights: ndarray[tuple[Any, ...], dtype[float64]], logger: Logger, fi_use: ndarray[tuple[Any, ...], dtype[integer]] | None = None, bi_use: ndarray[tuple[Any, ...], dtype[integer]] | None = None, mask_mirror_indices: bool = False, no_conjugate: bool = False, fill_model_visibilities: bool = False) → ndarray[tuple[Any, ...], dtype[float64]][source]

Calculate the number of contributing visibilities per pixel on the static {u,v} grid

Parameters:

obs (dict) – Observation metadata dictionary
params (dict) – Visibility metadata dictionary
vis_weights (NDArray[np.float64]) – Weights (flags) of the visibilities
pyfhd_config (dict) – pyfhd’s configuration dictionary containing all the options set for a pyfhd run
logger (Logger) – pyfhd’s logger
fi_use (NDArray[np.integer] | None, optional) – Frequency index array for gridding, i.e. gridding all frequencies for continuum images, by default None
bi_use (NDArray[np.integer] | None, optional) – Baseline index array for gridding, i.e even vs odd time stamps, by default None
mask_mirror_indices (bool, optional) – Exclude baselines mirrored along the v-axis, by default False
no_conjugate (bool, optional) – Do not perform the conjugate mirror of the {u,v} plane, by default False
fill_model_visibilities (bool, optional) – Calculate baseline grid locations for all baselines, regardless of flags, by default False

Returns:

uniform_filter – 2D array of number of contributing visibilities per pixel on the {u,v} grid

Return type:

NDArray[np.float64]

Filters

pyfhd.gridding.filters.filter_uv_uniform(image_uv: ndarray[tuple[Any, ...], dtype[complex128]], vis_count: ndarray[tuple[Any, ...], dtype[float64]] | None, obs: dict | None = None, params: dict | None = None, pyfhd_config: dict | None = None, logger: Logger | None = None, weights: ndarray[tuple[Any, ...], dtype[float64]] | None = None, fi_use: ndarray[tuple[Any, ...], dtype[integer]] | None = None, bi_use: ndarray[tuple[Any, ...], dtype[integer]] | None = None, mask_mirror_indices: bool = False) → tuple[ndarray[tuple[Any, ...], dtype[complex128]], ndarray[tuple[Any, ...], dtype[float64]]][source]

Perform uniform weighting in {u,v} space.

Parameters:

image_uv (NDArray[np.complex128]) – A 2D {u,v} gridded plane to be filtered
vis_count (NDArray[np.float64] | None) – 2D array of number of contributing visibilities per pixel on the {u,v} grid
obs (dict | None, optional) – Observation metadata dictionary, by default None
params (dict | None, optional) – Visibility metadata dictionary, by default None
pyfhd_config (dict | None, optional) – Run option dictionary, by default None
logger (Logger | None, optional) – pyfhd’s logger, by default None
weights (NDArray[np.float64] | None, optional) – The weights array (aka vis_weights), by default None
fi_use (NDArray[np.integer] | None, optional) – Frequency index array for gridding, i.e. gridding all frequencies for continuum images, by default None
bi_use (NDArray[np.integer] | None, optional) – Baseline index array for gridding, i.e even vs odd time stamps, by default None
mask_mirror_indices (bool, optional) – Exclude baselines mirrored along the v-axis, by default False

Returns:

image_uv_filtered (NDArray[np.complex128]) – The filtered 2D {u,v} plane
filter_use (NDArray[np.float64]) – The filter used

Raises:

TypeError – In the case obs or params is None and vis_count is also None

Visibility Grid

pyfhd.gridding.visibility_grid.visibility_grid(visibility: ndarray[tuple[Any, ...], dtype[complex128]], vis_weights: ndarray[tuple[Any, ...], dtype[float64]], obs: dict, psf: dict | File, params: dict, polarization: int, pyfhd_config: dict, logger: Logger, uniform_flag: bool = False, no_conjugate: bool = False, model: ndarray[tuple[Any, ...], dtype[complex128]] | None = None, fi_use: ndarray[tuple[Any, ...], dtype[integer]] | None = None, bi_use: ndarray[tuple[Any, ...], dtype[integer]] | None = None, verbose_logging=True) → dict[source]

Put visibilities on a discrete, hyperresolved 2D plane in {u,v}-space with the Fourier-transform of the beam sensitivity as the kernel (or spreading function). This can done per frequency to create 3D {u,v,f} cubes that can generate power spectrum statistics, or this can be done once for all frequencies to create a single 2D {u,v} plane for continuum images. These {u,v} planes are the slant-orthographic projection of the sky when Fourier transformed.

Gridding is done for calibrated data visibilities, (optionally) simulated model visibilities, weights, and variances. Weights and variances are crucial for propogating uncertainty estimates in the power spectrum space and for properly weighted images. The model is crucial for subtraction.

The kernel is a extremely hyperresolved look-up table, which is (optionally) interpolated even further. Since the {u,v} pixels are discrete and the baseline locations are not, the kernel will populate the pixels in a unique way for each individual baseline. This code is optimized to provide the best estimate for each baseline whilst maintaining speed.

Parameters:

visibility (NDArray[np.complex128]) – Calibrated and flagged data visibilities
vis_weights (NDArray[np.float64]) – Weights (flags) of the visibilities
obs (dict) – Observation metadata dictionary
psf (dict | h5py.File) – Beam metadata dictionary
params (dict) – Visibility metadata dictionary
polarization (int) – Index of the current polarization
pyfhd_config (dict) – pyfhd’s configuration dictionary containing all the options for a run
logger (Logger) – pyfhd’s logger
uniform_flag (bool, optional) – Grid a number count for contributing baselines per pixel, by default False
no_conjugate (bool, optional) – Do not perform the conjugate mirror to fill half of the {u,v} plane, by default False
model (NDArray[np.complex128] | None, optional) – Simulated model visibilites, by default None
fi_use (NDArray[np.integer] | None, optional) – Frequency index array for gridding, i.e. gridding all frequencies for continuum images, by default None
bi_use (NDArray[np.integer] | None, optional) – Baseline index array for gridding, i.e even vs odd time stamps, by default None
verbose_logging (bool, optional) – If True, will log the gridding process, by default True Set to False if you’re doing gridding per frequency (such as when creating HEALPIX files).

Returns:

gridding_dict – A dictionary with all the gridded {u,v} planes, updated observation metadata dic, and the number of visibilties that where gridded.

Return type:

dict

Raises:

ValueError – Raised in the case the model provided was not a NumPy Array when a model is not None

Visibility Degrid

pyfhd.gridding.visibility_degrid.visibility_degrid(image_uv: ndarray[tuple[Any, ...], dtype[complex128]], vis_weights: ndarray[tuple[Any, ...], dtype[float64]], obs: dict, psf: dict | File, params: dict, polarization: int = 0, fill_model_visibilities: bool = False, vis_input: ndarray[tuple[Any, ...], dtype[complex128]] | None = None, spectral_model_uv_arr: ndarray[tuple[Any, ...], dtype[float64]] | None = None, beam_per_baseline: bool = False, uv_grid_phase_only: bool = True, conserve_memory: bool = False, memory_threshold: float | int = 100000000.0)[source]

Generate visibilities from a 2D hyperresolved {u,v} plane using the Fourier transform of the beam sensitivity as the kernel (or integration function). The input {u,v} plane is the slant-orthographic projection of the sky when Fourier transformed with no instrumental effects. The integration kernel adds the instrumental effects to the visiblities.

Degridding is performed only on simulated model {u,v} planes. Simulated sources towards the edge of the sky image will be distorted (smeared) due to the projection of the 2D {u,v} plane.

The kernel is a extremely hyperresolved look-up table, which is (optionally) interpolated even further. Since the {u,v} pixels are discrete and the baseline locations are not, the kernel will integrate the pixels in a unique way for each individual baseline. This code is optimized to provide the best estimate for each baseline whilst maintaining speed.

Parameters:

image_uv (NDArray[np.complex128]) – A simulated {u,v} plane with no instrumental effects
vis_weight (NDArray[np.float64]) – Weights (flags) of the visibilities
obs (dict) – Observation metadata dictionary
psf (dict | h5py.File) – Beam metadata dictionary
params (dict) – Visibility metadata dictionary
polarization (int) – Index of the current polarization, by default 0
fill_model_visibilities (bool, optional) – Create all model visibilities disregarding flags, by default False
vis_input (NDArray[np.complex128] | None, optional) – Extra model visibilities to add to the degridded products, by default None
spectral_model_uv_arr (NDArray[np.float64] | None, optional) – Additional {u,v} planes to degrid for complicated source spectral dependencies, by default None
beam_per_baseline (bool, optional) – Generate beams with corrective phases given the baseline location, by default False
uv_grid_phase_only (bool, optional) – Generate beams with only {u,v} corrective phases, disregarding w phases, by default True
conserve_memory (bool, optional) – Reduce memory load by running loops, by default False

Returns:

visibility_array (NDArray[np.complex128]) – A simulated visibility array from degridding the input {u,v} plane with the instrumental kernel
obs (dict) – Updated observation metadata dictionary

Input & Output

pyfhd.io.pyfhd_io.convert_sav_to_dict(sav_path: str, logger: Logger, tmp_dir='temp_pyfhd')[source]

Given a path to an IDL style .sav file, load into a python dictionary using scipy.io.readsav.

If the file was saved with the IDL /compress option, the readsav function has to save a decompressed version of the file. By default this uses the tempfile module to find a location, but this usually finds a bad location with little storage when called on a super cluster. So explicitly make our own temp dir tmp_pyfhd where the code is being called. It is assumed many files are to be converted, so tmp_pyfhd should be deleted after all calls.

Mostly used just for testing, if you;re not a developer you can safely ignore this function

Parameters:

sav_path (str) – Filepath for an IDL .sav file
logger (Logger) – The logger to output any error messages to
tmp_dir (str) – Dir to place temporary files, creates the directory if doesn’t exist. Default: “tmp_pyfhd”.

Returns:

sav_dict – Dictionary containing whatever was in the .sav file

Return type:

dict

Converts a dictionary to a HDF5 group. This is called in the event a dictionary is found inside a dictionary that is being saved in a HDF5 file. Creates a subgroup for the hdf5 file with everything turning into individual datasets.

Parameters:

group (h5py.Group) – The created group to save the dictionary in
to_convert (dict) – The dictionary to save into the group
to_chunk (dict[str, dict]) – The chunking dictionary, see save for more information
variable_lengths (dict[str, DTypeLike]) – The variable length dictionary, see save for more information
logger (Logger) – pyfhd’s Logger

See also

pyfhd.io.pyfhd_io.save: Save a HDF5 file

Picks the double precision type for the given dtype for saving the hdf5 file to ensure everything is saved without losing information.

Parameters:: dtype (type) – The numpy dtype of an array
Returns:: The corresponding double precision type
Return type:: type

pyfhd.io.pyfhd_io.format_array(array: ndarray[tuple[Any, ...], dtype[Any]]) → ndarray[tuple[Any, ...], dtype[Any]][source]

Find any None values in an object array and replaces them with empty strings if we’re dealing with a string array, or NaNs if we’re dealing with a Number array. If complex, the NaN will be nan + nanj. If a string array is found, convert the string array to a bytes array, in all other cases leave the array alone as it should be ready to save into a HDF5 file.

Parameters:: array (NDArray[Any]) – The array to find None in and if so convert from object array
Returns:: array – Array without None objects and in the correct dtype
Return type:: NDArray[Any]

pyfhd.io.pyfhd_io.group_to_dict(group: Group) → dict[source]

When loading a hdf5 file into a dictionary, this turns a group into a dictionary, and then returns the dictionary.

Parameters:: group (h5py.Group) – A h5py group to turn into a dictionary
Returns:: return_dict – The group turned into a dictionary
Return type:: dict

pyfhd.io.pyfhd_io.load(file_name: Path, logger: Logger | None = None, lazy_load: bool = False) → dict[str, object] | ndarray[tuple[Any, ...], dtype[Any]] | File[source]

Loads a HDF5 file into pyfhd, it reads the HDF5 into an array if the HDF5 file contains a single dataset, while a HDF5 which contains multiple datasets will load them into a dictionary. Any groups will be convered to sub dictionaries using group_to_dict

Parameters:

file_name (Path) – The /path/to/the/hdf5.h5
logger (Logger) – pyfhd’s Logger
lazy_load (bool, optional) – Set to true if you wish to lazy load the file, currently the only file that will be supported to do this in pyfhd will be the beam/psf file, but support for other files can be done easily enough, by default False

Returns:

return_dict | array | h5_file – Returns a dict in the case the HDF5 file contains multple datasets, An array if the HDF5 contains one dataset or h5py File object if the file is lazy loaded to conserve memory.

Return type:

dict[str, object] | NDArray[Any] | h5py.File

See also

pyfhd.io.pyfhd_io.save: Save a HDF5 file
pyfhd.io.pyfhd_io.group_to_dict: Converts a h5py Group object to a dictionary

pyfhd.io.pyfhd_io.load_dataset(h5py_obj: File | Group, key: str, dataset: Dataset) → Any[source]

Loads a single dataset from a HDF5 File or Group, the key here is the dataset name from the file or group and is only used to check the attributes of said file or group. If the attribute associated with the key is True, then we assume the value saved is an empty dataset and we should return None. If this is False, load the value and check if this value should be a single value. There are special checks for byte arrays, if there is byte arrays, pyfhd assumes these are meant to be strings.

Parameters:

h5py_obj (h5py.File | h5py.Group) – A HDF5 file or group
key (str) – The dataset name
dataset (h5py.Dataset) – The dataset we are loading

Returns:

The value stored in the HDF5 Dataset

Return type:

Any

See also

pyfhd.io.pyfhd_io.load: Load a HDF5 file

pyfhd.io.pyfhd_io.recarray_to_dict(data: recarray | dict) → dict[source]

Turns a record array into a dict, but does it as a deep convert. This was needed due to scipy’s readsav returning an inception like experience of record arrays. This would mean to access values from something like the obs structure for a test, the code had to be obs[0][‘baseline_info’][0][‘tile_a’], which was became untenable as the full python translation won’t require these leaving us two codebases for IDL compatible and Python compatible. Instead, this function turns all record arrays into dictionaries, which are easier to understand and are faster.

This was made specifically to work with the readsav function, to get compatibility with general recarrays remove the zero index, as readsav for some reason adds a single dimension to all recarrays.

This was updated later to also take a dictionary which may contain record arrays too.

This was also updated later to turn object arrays into multidimensional arrays if they can be one. In the case the object array couldn’t be turned into a multidimensional array it was turned into a list

Parameters:: data (np.recarray or dict) – A record array or dictionary maybe containing nested record arrays
Returns:: data – A potentially nested dictionaries of dictionaries
Return type:: dict

pyfhd.io.pyfhd_io.save(file_name: Path, to_save: ndarray[tuple[Any, ...], dtype[Any]] | dict, dataset_name: str, logger: Logger | None = None, to_chunk: dict[str, dict] = {}, variable_lengths: dict[str, type[Any] | dtype[Any] | _SupportsDType[dtype[Any]] | tuple[Any, Any] | list[Any] | _DTypeDict | str | None] = {}) → None[source]

Saves a numpy array or dictionary into a hdf5 file using h5py, with compression applied to all arrays/datasets. An array will be saved as a single dataset, while a dictionary will be saved where each key will be a dataset unless the key points a dictionary in which case a group will be created and dict_to_group called to turn each key in that sub dict into a dataset (or another group if it’s another sub dictionary). This function should be kept as general as possible, if something needs formatting for saving, format it before calling this function. If you are converting a sav file to hdf5 with this function, use recarray_to_dict which converts the sav output from readsav into a proper python dictionary (rather than recarrays or weird array shapes, objects arrays etc.)

Parameters:

file_name (Path) – The file to save as hdf5 should be /path/to/file_name.h5 (or .hdf5)
to_save (NDArray[Any] | dict) – The dictionary or numpy array to save into the hdf5 file
dataset_name (str) – Used in the case that the to_save variable is an array, this name will be used as the key for the dataset in the hdf5 file.
logger (Logger, optional) – pyfhd’s Logger, by default None (in case you don’t want to use the logger for testing)
to_chunk (dict[str, dict], optional) – A dictionary where each key-value pair represents a key in the to_save dictionary, and the value is a dictionary which should contain two key-value pairs, shape which should be the shape of the array and chunk which tells hdf5 how to chunk the dataset when it’s being read/written. If you’re not sure how to chunk the dataset, set chunk to True which enables h5py to guess the chunk size for you. By default {}
variable_lengths (dict[str, DTypeLike], optional) – A dictionary where each key-value pair represents a key in the to_save dictionary, and the value is a dtype. This is for special cases where you must save an array of variable length arrays. H5Py does support variable length arrays, but you must use a special type, using the h5py.vlen_dtype() you can create a dtype which accepts object arrays of variable lengths. For example if you wish to have variable integer array called ija, you would use h5py.vlen_dtype(np.int64), and save use it in the variable_lengths dictionary like so, {‘ija’: h5py.vlen_dtype(np.int64)}, which will set the dtype appropriately during a create_dataset call. By default {}

See also

pyfhd.io.pyfhd_io.load: Load a HDF5 file
pyfhd.io.pyfhd_io.dict_to_group: Converts a dictionary to a h5py Group Object
pyfhd.io.pyfhd_io.recarray_to_dict: Turns any record arrays into dicts, also formats object arrays into the correct dtype array
pyfhd.io.pyfhd_io.save_dataset: Saves a single dataset based off a dictionary key-value pair
pyfhd.io.pyfhd_io.format_array: Finds any None is an array and replaces them appropriately

pyfhd.io.pyfhd_io.save_dataset(h5py_obj: ~h5py._hl.files.File | ~h5py._hl.group.Group, key: str, value: ~typing.Any, to_chunk: dict[str, dict], variable_lengths: [<class 'str'>, typing.Union[type[typing.Any], numpy.dtype[typing.Any], numpy._typing._dtype_like._SupportsDType[numpy.dtype[typing.Any]], tuple[typing.Any, typing.Any], list[typing.Any], numpy._typing._dtype_like._DTypeDict, str, NoneType]], logger: ~logging.Logger | None) → bool[source]

A general function for saving a dataset inside a HDF5 File or Group. It’s used exclusively for saving a dictionary into a HDF5 file, hence why we take a key and value pair. The to_chunk parameter is explained in the save function, please look there for explanation. In the case of finding a None object an Empty Dataset is saved and the is_none is returned as True, so the attribute associated with the key can also be set to True to indicate to pyfhd later that the value is meant to be None when reading in the dataset again.

Parameters:

h5py_obj (h5py.File | h5py.Group) – A h5py object that has access to the create_dataset and create_group methods
key (str) – The key from the dictionary we’re saving
value (Any) – The value from the dictionary
to_chunk (dict[str, dict]) – A dictionary where each key-value pair represents a key in the to_save dictionary, and the value is a dictionary which should contain two key-value pairs, shape which should be the shape of the array and chunk which tells hdf5 how to chunk the dataset when it’s being read/written. If you’re not sure how to chunk the dataset, set chunk to True which enables h5py to guess the chunk size for you. By default {}
variable_lengths (dict[str, DTypeLike]) – A dictionary where each key-value pair represents a key in the to_save dictionary, and the value is a dtype. This is for special cases where you must save an array of variable length arrays. H5Py does support variable length arrays, but you must use a special type, using the h5py.vlen_dtype() you can create a dtype which accepts object arrays of variable lengths. For example if you wish to have variable integer array called ija, you would use h5py.vlen_dtype(np.int64), and save use it in the variable_lengths dictionary like so, {‘ija’: h5py.vlen_dtype(np.int64)}, which will set the dtype appropriately during a create_dataset call. By default {}
logger (Logger | None) – pyfhd’s Logger

Returns:

is_none – True if the value is None, False otherwise

Return type:

bool

See also

pyfhd.io.pyfhd_io.save: Save a HDF5 file
pyfhd.io.pyfhd_io.dict_to_group: Converts a dictionary to a h5py Group Object

pyfhd.io.pyfhd_quickview.get_image_renormalization(obs: dict, weights: ndarray[tuple[Any, ...], dtype[float64]], beam_base: ndarray[tuple[Any, ...], dtype[complex128]], filter_arr: ndarray[tuple[Any, ...], dtype[float64]], pyfhd_config: dict, logger: Logger) → ndarray[source]

Use the weights to renormalize the image for Jy/beam to Jy/sr. While Jy/beam is more common in radio astronomy, Jy/str is a physical brightness unit, rather than an instrumental brightness unit.

Parameters:

obs (dict) – Observation metadata dictionary.
weights (NDArray[np.float64]) – Visibility weights array.
beam_base (NDArray[np.complex128]) – Beam orthoslant image per polarization.
filter_arr (NDArray[np.float64]) – u-v array of filter weights, relevant for a uniform filter.
pyfhd_config (dict) – pyfhd configuration settings.
logger (Logger) – pyfhd’s Logger.

Returns:

Conversion in image space from Jy/beam to Jy/sr per pixel.

Return type:

NDArray[np.float64]

pyfhd.io.pyfhd_quickview.quickview(obs: dict, psf: dict, params: dict, cal: dict, vis_arr: ndarray[tuple[Any, ...], dtype[complex128]], vis_weights: ndarray[tuple[Any, ...], dtype[float64]], image_uv: ndarray[tuple[Any, ...], dtype[complex128]], weights_uv: ndarray[tuple[Any, ...], dtype[complex128]], variance_uv: ndarray[tuple[Any, ...], dtype[float64]], uniform_filter_uv: ndarray[tuple[Any, ...], dtype[float64]], model_uv: ndarray[tuple[Any, ...], dtype[complex128]], pyfhd_config: dict, logger: Logger) → None[source]

Generate continuum images from all gridded u-v planes, and save as FITS files and optionally PNG files.

Parameters:

obs (dict) – Observation metadata dictionary.
psf (dict) – Beam dictionary.
params (dict) – Visibility metadata dictionary.
cal (dict) – Calibration dictionary.
vis_arr (NDArray[np.complex128]) – Calibrated visibilities array.
vis_weights (NDArray[np.float64]) – Visibility weights array.
image_uv (NDArray[np.complex128]) – Continuum uv-plane of the calibrated data.
weights_uv (NDArray[np.complex128]) – Continuum uv-plane of the weights (the sampling map).
variance_uv (NDArray[np.float64]) – Continuum uv-plane of the variance (the variance map).
uniform_filter_uv (NDArray[np.float64]) – Continuum uv-plane of the uniform filter (if used).
model_uv (NDArray[np.complex128]) – Continuum uv-plane of the model data.
pyfhd_config (dict) – pyfhd configuration settings.
logger (Logger) – pyfhd’s Logger.

HEALPix

pyfhd.healpix.export.healpix_snapshot_cube_generate(obs: dict, psf: dict | File, cal: dict, params: dict, vis_arr: ndarray[tuple[Any, ...], dtype[complex128]], vis_model_arr: ndarray[tuple[Any, ...], dtype[complex128]], vis_weights: ndarray[tuple[Any, ...], dtype[float64]], pyfhd_config: dict, logger: Logger) → None[source]

Generate and save HEALPix images per polarization and per frequency channel for each of the gridded visibility products, creating calibrated data, model data, sampling map, and variance map cubes. General settings split up the outputs into even and odd interleaved time samples. Interpolation is performed from the orthoslant projection to the HEALPix grid.

Parameters:

obs (dict) – Observation metadata dictionary.
psf (dict | h5py.File) – Beam dictionary
cal (dict) – Calibration dictionary
params (dict) – Visibility metadata dictionary
vis_arr (NDArray[np.complex128]) – The visibility array
vis_model_arr (NDArray[np.complex128]) – The model visibility array
vis_weights (NDArray[np.float64]) – The visibility weights
pyfhd_config (dict) – pyfhd configuration settings
logger (Logger) – pyfhd’s Logger

pyfhd.healpix.healpix_utils.beam_image_cube(obs: dict, psf: dict | File, logger: Logger, freq_i_arr: ndarray[tuple[Any, ...], dtype[integer]] | None = None, pol_i_arr: ndarray[tuple[Any, ...], dtype[integer]] | None = None, n_freq_bin: int | None = None, square: bool = True, beam_threshold: float | None = None) → tuple[ndarray[tuple[Any, ...], dtype[complex128]], ndarray[tuple[Any, ...], dtype[float64]]][source]

Calculate the beam image per unflagged frequency channel to build a cube. Build a contiguous mask in x and y based off of a threshold value.

Parameters:

obs (dict) – Observation metadata dictionary
psf (dict | h5py.File) – Beam dictionary
logger (Logger) – pyfhd’s Logger
freq_i_arr (np.ndarray | None, optional) – Index array of the beam frequency binning in the observation metadata dictionary, by default None
pol_i_arr (np.ndarray | None, optional) – Index array of the polarizations in the observation metadata dictionary, by default None
n_freq_bin (int | None, optional) – Number of frequency channels to divide the full observation bandwidth into for the beam image cube, by default None
square (bool, optional) – Square the beam image, by default True
beam_threshold (float | None, optional) – Fractional threshold in which to build a mask for all smaller beam values, by default None

Returns:

Beam image frequency cube and mask threshold array.

Return type:

tuple[NDArray[np.complex128], NDArray[np.float64]]

pyfhd.healpix.healpix_utils.healpix_cnv_apply(image: ndarray[tuple[Any, ...], dtype[integer | floating | complexfloating]], hpx_cnv: dict) → ndarray[tuple[Any, ...], dtype[float64]][source]

healpix_cnv_apply creates a map based off the array/image and healpix convention dictionary given. In FHD the healpix_cnv_apply was mainly used as a wrapper for sprsax2, as such I will put the code for sprsax2 in here as pyfhd will only use sprsax2 here as we don’t have the holographic mapping function at this time.

Vectors ‘sa’ (interpolation) and ‘ija’ (index) follow the row-index sparse storage mode as described in section 2.7 of Numerical Recipes in C, 2nd edition.

Parameters:

image (NDArray[np.integer | np.floating | np.complexfloating]) – An orthoslant image array that is to be converted to a HEALPix map.
hpx_cnv (dict) – The HEALPix index/interpolation dictionary

Returns:

hpx_map – A flattened 1D HEALPix array with the interpolated values from the image.

Return type:

NDArray[np.float64]

pyfhd.healpix.healpix_utils.healpix_cnv_generate(obs: dict, mask: ndarray[tuple[Any, ...], dtype[int64]] | None, hpx_radius: float, pyfhd_config: dict, logger: Logger, nside: int = None) → dict[source]

Generate the HEALPix index/interpolation dictionary that is used to convert between orthoslant image pixelization and the RING index HEALPix pixelization scheme. Binlinear interpolation is performed to HEALPix pixels centers. All angles are in degrees.

Parameters:

obs (dict) – Observation metadata dictionary
mask (np.ndarray) – If HEALPix indices are not provided in the config file, this boolean mask map of the orthoslant image can be used to mask out areas from the HEALPix conversion.
hpx_radius (float) – If HEALPix indices are not provided in the config file, this selects a radius in degrees, centred on the RA/Dec of the observation, to be converted to HEALPix pixels.
pyfhd_config (dict) – pyfhd configuration settings
logger (Logger) – pyfhd’s Logger
nside (int) – The HEALPix nside parameter, calculated based on the observation metadata if none provided. Defaults to None.

Returns:

A dictionary containing the orthoslant to HEALPix pixel mapping and interpolation. Vectors ‘sa’ (bilinear interpolation calculated from fractional offsets) and ‘ija’ (flattened index) follow the row-index sparse storage mode as described in section 2.7 of Numerical Recipes in C, 2nd edition.

Return type:

dict

pyfhd.healpix.healpix_utils.phase_shift_uv_image(obs: dict) → ndarray[tuple[Any, ...], dtype[complex128]][source]

Phase shift the entire u-v plane to the original phase center.

Parameters:: obs (dict) – Observation metadata dictionary
Returns:: A 2D u-v phase shift array that will shift the current phase centre to the original phase centre.
Return type:: NDArray[np.float64]

pyfhd.healpix.healpix_utils.vis_model_freq_split(obs: dict, psf: dict | File, params: dict, vis_weights: ndarray[tuple[Any, ...], dtype[float64]], vis_model_arr: ndarray[tuple[Any, ...], dtype[complex128]], vis_arr: ndarray[tuple[Any, ...], dtype[complex128]], polarization: int, pyfhd_config: dict, logger: Logger, fft: bool = True, save_uvf: bool = True, uvf_name: str = '', bi_use: ndarray[tuple[Any, ...], dtype[integer]] = None) → dict[source]

Grid as a function of frequency, which can be split into larger channels. This uvf cube can be saved as is or used to create an orthoslant image as a function of frequency. Optional bi_use argument allows for the separate gridding of even/odd interleaved time samples for error propagation in the power spectrum.

Parameters:

obs (dict) – Observation metadata dictionary
psf (dict | h5py.File) – Beam dictionary
params (dict) – Visibility metadata dictionary
vis_weights (NDArray[np.float64]) – Visibility weights array
vis_model_arr (NDArray[np.complex128]) – Model visibility array
vis_arr (NDArray[np.complex128]) – Calibrated visibility array
polarization (int) – The polarization index
pyfhd_config (dict) – pyfhd configuration settings
logger (Logger) – pyfhd’s Logger
fft (bool, optional) – Calculate the orthoslant image instead of the u-v plane, by default True
save_uvf (bool, optional) – Save the uvf cube, by default True
uvf_name (str, optional) – The name after the obsid in the uvf cube filename, by default ‘’
bi_use (NDArray[np.integer], optional) – The visibility indices to use in gridding, i.e. even/odd interleaved time samples, by default None

Returns:

cube_split – Dictionary of the gridded u-v planes or orthoslant images as a function of frequency for the calibrated data, model (if present), residual data (if present), sampling map, and variance map, along with the observation metadata dictionary.

Return type:

dict

Plotting

Calibration Plots

pyfhd.plotting.calibration.plot_cals(obs: dict, cal: dict, pyfhd_config: dict)[source]

Plot the calibration solutions, the calibration residuals, and the raw calibration solutions in a grid of 128 per page.

Parameters:

obs (dict) – Observation metadata dictionary
cal (dict) – Calibration dictionary
pyfhd_config (dict) – Run option dictionary

Gridding Plots

pyfhd.plotting.gridding.plot_gridding(obs: dict, image_uv: ndarray[tuple[Any, ...], dtype[complex128]], weights_uv: ndarray[tuple[Any, ...], dtype[complex128]], variance_uv: ndarray[tuple[Any, ...], dtype[float64]], pyfhd_config: dict, model_uv: ndarray[tuple[Any, ...], dtype[complex64]] | None, logger: Logger) → None[source]

Plot the continuum uv-planes for data, model, and weights.

Parameters:

obs (dict) – Observation metadata dictionary
image_uv (NDArray[np.complex128]) – UV-plane of the gridded data
weights_uv (NDArray[np.complex128]) – UV-plane of the gridded weights
variance_uv (NDArray[np.float64]) – UV-plane of the gridded variances
model_uv (NDArray[np.complex128], optional) – UV-plane of the gridded model
pyfhd_config (dict) – Run option dictionary
logger (Logger) – pyfhd’s logger for displaying errors and info to the log files

Plotting Utils and FITS imaging

pyfhd.plotting.image.color_range(count_missing: int = None) → tuple[source]

Define the color range for the image data.

Parameters:: count_missing (int, optional) – Count of missing values, by default None
Returns:: A tuple containing the color range and the number of colors.
Return type:: tuple

pyfhd.plotting.image.log_color_calc(data: ndarray[tuple[Any, ...], dtype[integer | floating | complexfloating]], data_range: ndarray[tuple[Any, ...], dtype[integer | floating]] = None, color_profile: str = 'log_cut', log_cut_val: float = None, min_abs: float = None, count_missing: int = None, wh_missing: ndarray[tuple[Any, ...], dtype[integer]] = None, missing_color: int = None, invert_colorbar: bool = False) → tuple[source]

Translated version of log_color_calc from IDL to Python.

Parameters:

data (NDArray[np.integer | np.floating | np.complexfloating]) – A 2D array of data to be displayed as an image. The data can be of type int, float, or complex.
data_range (NDArray[np.integer | np.floating], optional) – Min/max color bar range, by default [np.nanmin(image), np.nanmax(image)]
color_profile (str, optional) – Color bar profiles for logarithmic scaling. “log_cut”, “sym_log”, “abs”, by default “log_cut”
log_cut_val (int | float, optional) – Minimum log value to cut at, by default None
data_min_abs (int | float, optional) – The minimum absolute value for the color bar, by default None
count_missing (int, optional) – The number of missing values, by default None
wh_missing (NDArray[np.integer], optional) – The location of the missing values, by default None
missing_color (int, optional) – The index of the color bar for missing values, by default None
invert_colorbar (bool, optional) – Invert the color bar, by default False

Returns:

data_log_norm (NDArray[np.int | np.float64]) – The normalized data array.
cb_ticks (NDArray[np.int | np.float64]) – The color bar ticks.
cb_ticknames (NDArray[np.int | np.float64]) – The color bar tick names.

pyfhd.plotting.image.plot_fits_image(fits_file: str, output_path: str, logger: Logger, title: str = 'FITS Image') → None[source]

Plot a FITS image using Astropy and save it to the specified output directory.

Parameters:

fits_file (str) – Path to the FITS file.
output_path (str) – Path to output image file.
title (str, optional) – Title of the plot, by default “FITS Image”.
logger (Logger) – pyfhd’s logger for displaying errors and info to the log files

Returns:

The function saves the plot to the specified output path.

Return type:

None

pyfhd.plotting.image.quick_image(image: ndarray[tuple[Any, ...], dtype[integer | floating | complexfloating]], xvals: ndarray[tuple[Any, ...], dtype[integer | floating]] = None, yvals: ndarray[tuple[Any, ...], dtype[integer | floating]] = None, data_range: ndarray[tuple[Any, ...], dtype[integer | floating]] = None, data_min_abs: float = None, xrange: ndarray[tuple[Any, ...], dtype[integer | floating]] = None, yrange: ndarray[tuple[Any, ...], dtype[integer | floating]] = None, data_aspect: float = None, log: bool = False, color_profile: str = 'log_cut', xtitle: str = None, ytitle: str = None, title: str = None, cb_title: str = None, note: str = None, charsize: int = None, xlog: bool = False, ylog: bool = False, multi_pos: list = None, start_multi_params: dict = None, alpha: float = None, missing_value: int | float | complex = None, savefile: str = None, png: bool = False, eps: bool = False, pdf: bool = False) → None[source]

General function to display and/or save a 2D data array as an image with an appropriately scaled color bar.

Parameters:

image (NDArray[np.integer | np.floating | np.complexfloating]) – A 2D array of data to be displayed as an image. The data can be of type int, float, or complex.
xvals (NDArray[np.integer | np.floating], optional) – An array of x-axis values, by default None
yvals (NDArray[np.integer | np.floating], optional) – An array of y-axis values, by default None
data_range (NDArray[np.integer | np.floating], optional) – Min/max color bar range, by default [np.nanmin(image), np.nanmax(image)]
data_min_abs (float, optional) – The minimum absolute value for the color bar, by default None
xrange (NDArray[np.integer | np.floating], optional) – The indices (or xvals, if provided) to zoom the image, by default None
yrange (NDArray[np.integer | np.floating], optional) – The indices (or yvals, if provided) to zoom the image, by default None
data_aspect (int | float, optional) – The aspect ratio of y to x, by default None
log (bool, optional) – Color bar on logarithmic scale, by default False
color_profile (str, optional) – Color bar profiles for logarithmic scaling. “log_cut”, “sym_log”, “abs”, by default “log_cut”
xtitle (str, optional) – The title of the x-axis, by default None
ytitle (str, optional) – The title of the x-axis, by default None
title (str, optional) – The title of the image, by default None
cb_title (str, optional) – The title of the color bar, by default None
note (str, optional) – A small note to place on the bottom right of the image, by default None
charsize (int, optional) – The size of the font, by default None
xlog (bool, optional) – Use logarithmic scale for the x-axis, by default False
ylog (bool, optional) – Use logarithmic scale for the y-axis, by default False
multi_pos (list, optional) – A list of 4 elements defining the position of the plot in a multi-panel layout, by default None
start_multi_params (dict, optional) – Parameters for starting a multi-panel layout, by default None
alpha (float, optional) – Transparency for the image, by default None
missing_value (int | float | complex, optional) – Exclude value from the color bar, by default None
savefile (str, optional) – The save file name, by default None
png (bool, optional) – Create a png of the image, by default False
eps (bool, optional) – Create an eps of the image, by default False
pdf (bool, optional) – Create a pdf of the image, by default False

Returns:

Displays the image and/or saves it to disk.

Return type:

None

Tools & Utilities

pyfhd.pyfhd_tools.pyfhd_utils.angle_difference(ra1: float, dec1: float, ra2: float, dec2: float, degree: bool = False, nearest: bool = False) → float[source]

Calculates the angle difference between two given celestial coordinates.

Parameters:

ra1 (float) – Right Ascension of a coordinate 1 in radians or degrees
dec1 (float) – Declination of a coordinate 1 in radians or degrees
ra2 (float) – Right Ascension of a coordinate 2 in radians or degrees
dec2 (float) – Declination of a coordinate 2 in radians or degrees
degree (bool, optional) – If True, then we treate the coordinates given as degrees, by default False
nearest (bool, optional) – Calculate implied angle instead, by default False

Returns:

relative_angle – The angle difference in degrees

Return type:

float

pyfhd.pyfhd_tools.pyfhd_utils.array_match(array_1: ndarray[tuple[Any, ...], dtype[integer | floating | complexfloating]], value_match: ndarray[tuple[Any, ...], dtype[integer | floating | complexfloating]], array_2: ndarray[tuple[Any, ...], dtype[integer | floating | complexfloating]] | None = None) → ndarray[tuple[Any, ...], dtype[int64]][source]

Find the indices of the input array which match the array of values provided. If a second input array is provided, then find the indices where the values provided match in both input arrays. This matching can only be done between integer values.

Parameters:

array_1 (NDArray[np.integer | np.floating | np.complexfloating]) – The input array to match the values in
value_match (NDArray[np.integer | np.floating | np.complexfloating]) – The values to find in the input array
array_2 (NDArray[np.integer | np.floating | np.complexfloating] | None, optional) – A second array to match the values in via a logical AND, by default is None

Returns:

match_indices – The indices of the input array(s) which match the values provided

Return type:

NDArray[np.int64]

Raises:

ValueError – Gets raised if value_match is None

pyfhd.pyfhd_tools.pyfhd_utils.crosspol_split_real_imaginary(image: ndarray[tuple[Any, ...], dtype[complex128]], pol_names: list[str] | None = None) → tuple[ndarray[tuple[Any, ...], dtype[complex128]], list[str] | None][source]

Reformat the input full polarisation image, containing PP, PQ, QP, and QQ, into PP, real(PQ), imaginary(PQ), and QQ. PQ and QP (i.e. XY and YX) are complex and conjugate mirrors of one another. To make a understandable image, we can plot the real and imaginary parts of PQ separately without loss of information.

Parameters:

image (NDArray[np.complex128]) – Image array ordered in polarisation by PP, PQ, QP, and QQ.
pol_names (list[str] | None, optional) – The name of the polarisations, by default None

Returns:

image (NDArray[np.complex128]) – Image array ordered in polarisation by PP, real(PQ), imaginary(PQ), and QQ
pol_names (list[str]|None]) – New polarisation name array to reflect real(PQ) and imaginary(PQ)

pyfhd.pyfhd_tools.pyfhd_utils.deriv_coefficients(n: int, divide_factorial: bool = False) → ndarray[tuple[Any, ...], dtype[float64]][source]

Computes an array of coefficients resulting in taking the n-th derivative of a function of the form x^a (a must not be a positive integer less than n)

Parameters:

n (int) – Decides the length of coefficients
divide_factorial (bool, optional) – Determine if we need to divide by the factorial, by default is False

Returns:

coeff – An array of coefficients

Return type:

NDArray[np.float64]

pyfhd.pyfhd_tools.pyfhd_utils.get_bins(min: int | float, max: int | float, bin_size: int) → ndarray[tuple[Any, ...], dtype[float64 | int64]][source]

Calculates the bins for the histogram and reverse indices based on a minimum and maximum value, plus a given bin size. It mirrors IDL’s way of calculating the bin edges. In IDL bins seem to be right hand side of the bin open like IDL, even at the end. However, in comparison to NumPy, the last bin is always the maximum value. This does utilize numba and parallelization to make it faster.

Parameters:

min (int | float) – The minimum chosen for the histogram
max (int | float) – The maximum chosen for the histogram
bin_size (int) – The bin size chosen for the histogram. This histogram always uses bins of equal widths.

Returns:

bins – A NumPy array of all the bins ranging from min (bin[0]) to the max (bin[-1]). The step is bin_size.

Return type:

NDArray[np.float64 | np.int64]

See also

histogram: Calculates the bins, histogram and reverse indices
get_hist: Calculates histogram only
get_ri: Calculates the reverse indices only

pyfhd.pyfhd_tools.pyfhd_utils.get_hist(data: ndarray[tuple[Any, ...], dtype[floating | integer | complexfloating]], bins: ndarray[tuple[Any, ...], dtype[float64 | int64]], min: int | float, max: int | float) → ndarray[tuple[Any, ...], dtype[int64]][source]

Calculates the histogram based on the given bins and data, taking into account the minimum and maximum

Parameters:

data (NDArray[np.floating | np.integer | np.complexfloating]) – A NumPy array that is of one dtype float, int, or complex. Cannot be object
bins (NDArray[np.float64 | np.int64]) – A NumPy array of bins for the histogram
min (int | float) – The minimum set for the data
max (int | float) – The maximum set for the data

Returns:

hist – A histogram of the data using the bins found in the bins array.

Return type:

NDArray[np.int64]

See also

histogram: Calculates the bins, histogram and reverse indices.
get_bins: Calculates the bins only
get_ri: Calculates the reverse indices only

pyfhd.pyfhd_tools.pyfhd_utils.get_ri(data: ndarray[tuple[Any, ...], dtype[floating | integer | complexfloating]], bins: ndarray[tuple[Any, ...], dtype[float64 | int64]], hist: ndarray[tuple[Any, ...], dtype[int64]], min: int | float, max: int | float) → ndarray[tuple[Any, ...], dtype[int64]][source]

Calculates the reverse indices of a data and histogram. The function replicates IDL’s REVERSE_INDICES keyword within their HISTOGRAM function. The reverse indices array which is returned by this function can be hard to understand at first, I will explain it here and also link JD Smith’s famous article on IDL’s HISTOGRAM.

The reverse indices array is two vectors concatenated together. The first vector contains indexes for the second vector, this vector should be the size of bins + 1. The second vector contains indexes from the data itself, and should be the size of data. The justification for having such an array is to quickly make adjustments to certain bins without having to search the array multiple times, thus avoiding multiple O(data.size) loops.

The first vector indexes contain the starting positions of each bin in the second vector. For example, between the indexes given by first_vector[0] and first_vector[1] of the second vector should be all the indexes of bins[0] from inside the data. So if I wanted to make adjustments to the entire first bin, and only the first bin I can use the reverse indices array, ri to do this. Let’s say I wanted to flag all values of bins[0] with -1 for some reason to make them invalid in other calculations with the data, then I could do this:

data[ri[ri[0] : ri[1]]] = -1

Or more generally

data[ri[ri[i] : ri[i + 1]]] = -1

Where i is 0 <= i <= bins.size.

If you wish to gain a better understanding of how this get_ri function works, and the associated histogram function I have created here, please use the link given in the Notes section. This link will take you JD Smith’s article on IDL’s HISTOGRAM, it is an article which explains the IDL HISTOGRAM function better than IDL’s own documentation. If you must gain a deeper understanding, read it once, gasp and get your shocks and many cries of why out of your system, then read it again. And keep reading it till you understand, as per the editor’s note on the article:

“…If you read it enough, the secrets of the command will be revealed to you. Stranger things have happened”

Parameters:

data (NDArray[np.floating | np.integer | np.complexfloating]) – A NumPy array of the data
bins (NDArray[np.float64 | np.int64]) – A NumPy array containing the bins for the histogram
hist (NDArray[np.int64]) – A NumPy array containing the histogram
min (int | float) – The minimum for the dataset
max (int | float) – The maximum for the dataset

Returns:

ri – An array containing the reverse indices, which is two vectors, the first vector containing indexes for the second vector. The second vector contains indexes for the data.

Return type:

NDArray[np.int64]

See also

histogram: Calculates the bins, histogram and reverse indices.
get_bins: Calculates the bins only
get_hist: Calculates histogram only

Notes

‘HISTOGRAM: The Breathless Horror and Disgust’ : http://www.idlcoyote.com/tips/histogram_tutorial.html

The histogram function combines the use of the get_bins, get_hist and get_ri functions into one function. For the descriptions and docs of those functions look in See Also. This function will return the histogram, bin/bin_edges and the reverse indices.

Parameters:

data (NDArray[np.floating | np.integer | np.complexfloating]) – A NumPy array containing the data we want a histogram of
bin_size (int, optional) – Sets the bin size for this histogram, by default 1
num_bins (int | None, optional) – Set the number of bins this does override bin_size completely, by default None
min (int | float | None, optional) – Set a minimum for the dataset, by default None
max (int | float | None, optional) – Set a maximum for the dataset, by default None

Returns:

hist (NDArray[np.int64]) – The histogram of the data
bins (NDArray[np.float64 | np.int64]) – The bins of the histogram
ri (NDArray[np.int64]) – The reverse indices array for the histogram and data

See also

get_bins: Calculates the bins only
get_hist: Calculates histogram only
get_ri: Calculates the reverse indices only

pyfhd.pyfhd_tools.pyfhd_utils.idl_argunique(arr: ndarray[tuple[Any, ...], dtype[integer | floating | complexfloating]]) → ndarray[tuple[Any, ...], dtype[int64]][source]

In IDL the UNIQ function returns the indexes of the unique values within an array (that is assumed to be sorted beforehand). In NumPy they use the first unique index when using return index, where as in IDL they use the last index.

Parameters:: arr (NDArray[np.integer | np.floating | np.complexfloating]) – A sorted numpy array of any type.
Returns:: An array containing indexes of the last occurence of the unique value in arr
Return type:: NDArray[np.int64]

Examples

>>> test = np.array([10, 30, 5, 100, 200, 75, 200, 100, 30])
>>> idl_argunique(test)
array([0, 1, 3, 4, 6, 8])

pyfhd.pyfhd_tools.pyfhd_utils.idl_median(x: ndarray[tuple[Any, ...], dtype[integer | floating | complexfloating]], width: int = 0, even: bool = False) → float[source]

The IDL Median function doesn’t always work as you’d expect, as generally you need to use the EVEN keyword to get the median of an even number of elements, otherwise it returns the maximum element of the two numbers in the middle of an even sorted array. This function replicates the IDL median function, in case you need that functionality. Typically though, we recommend using numpy.median or scipy.ndimage.median_filter.

Parameters:

x (NDArray[np.integer | np.floating | np.complexfloating]) – Data to perform median on
width (int) – If set, perform a type of median filtering.
even (bool, optional) – In this case it will just run numpy.median, by default False

When width is set. unfortunately the edge conditions when using cannot be replicated soley with scipy.ndimage.median_filter so use median_filter and set the edge cases manually

Returns:: The median of the array
Return type:: float

See also

numpy.median: Computes the median of an array
scipy.ndimage.median_filter: Applies a median filter to an array

pyfhd.pyfhd_tools.pyfhd_utils.l_m_n(obs: dict, psf: dict | File, obsdec: float | None = None, obsra: float | None = None, declination_arr: ndarray[tuple[Any, ...], dtype[floating]] | None = None, right_ascension_arr: ndarray[tuple[Any, ...], dtype[floating]] | None = None) → tuple[ndarray[tuple[Any, ...], dtype[float64]], ndarray[tuple[Any, ...], dtype[float64]], ndarray[tuple[Any, ...], dtype[float64]]][source]

Calculate the directional cosines l,m,n given an RA, Dec, and phase centre for each pixel in the input arrays given, or from the hyperresolved RA/Dec array created during beam creation.

Parameters:

obs (dict) – Observation metadata dictionary.
psf (dict | h5py.File) – Beam dictionary.
obsdec (float | None, optional) – The Dec phase centre for the observation, by default None and set to the phase centre saved in the observation metadata dicitonary (obs[‘obsdec’])
obsra (float | None, optional) – The RA phase centre for the observation, by default None and set to the phase centre saved in the observation metadata dicitonary (obs[‘obsra’])
declination_arr (NDArray[np.floating] | None, optional) – An array of declinations, by default None and set to the hyperresolved Dec array in psf[‘image_info’][‘dec_arr’]
right_ascension_arr (NDArray[np.floating] | None, optional) – An array of right ascensions, by default None and set to the hyperresolved RA array in psf[‘image_info’][‘ra_arr’]

Returns:

l_mode (NDArray[np.float64]) – Directional cosine l array, or the cosine of the vector that contributes to the x-axis
m_mode (NDArray[np.float64]) – Directional cosine m array, or the cosine of the vector that contributes to the y-axis
n_tracked (NDArray[np.float64]) – Directional cosine n array, or the cosine of the vector that contributes to the z-axis, dependent on the phase centre.

pyfhd.pyfhd_tools.pyfhd_utils.meshgrid(dimension: int, elements: int, axis: int | None = None, return_integer: bool = False) → ndarray[tuple[Any, ...], dtype[int64 | float64]][source]

Generates a 2D array of X or Y values. Could be replaced by a another function

Parameters:

dimension (int) – Sets the column size of the array to return
elements (int) – Sets the row size of the array to return
None (axis = int |) – Set axis = 1 for X values, set axis = 2 for Y values, by default is None
optional – Set axis = 1 for X values, set axis = 2 for Y values, by default is None
integer (return) – dtype is implied by dimension and/or elements by default. If True, sets return array to int, by default is False

Returns:

result – A numpy array of shape (elements, dimension) that is a modified np.arange(elements * dimension).

Return type:

NDArray[np.int64 | np.float64]

pyfhd.pyfhd_tools.pyfhd_utils.parallactic_angle(latitude: float, hour_angle: float, dec: float) → float[source]

Calculates the parallactic angle given latitude (usually a declination), hour_angle and another declination

Parameters:

latitude (float) – An angle in degrees, usually a declination in this package
hour_angle (float) – The hour angle in degrees
dec (float) – A declination in degrees

Returns:

parallactic_angle – The angle between the great circle through a celestial object and the zenith, and the hour circle of the object

Return type:

float

Resizes a 2D array by averaging or repeating elements, new dimensions must be integral factors of original dimensions.

In the case of expanding an existing array, rebin will interpolate between the original values with a linear function. In the case of compressing an existing array, rebin will average

Parameters:

a (NDArray[np.integer | np.floating | np.complexfloating]) – Input array.
new_shape (ArrayLike) – Shape of the output array in (rows, columns) Must be a factor or multiple of a.shape
sample (bool, optional) – Use this to get samples using rebin, rather than interpolation, by default False.

Returns:

rebinned – If the new shape is smaller of the input array, the data are averaged, if the new shape is bigger array elements are repeated and interpolated

Return type:

NDArray[np.integer | np.floating | np.complexfloating]

Examples

>>> test = np.array([0,10,20,30])
>>> rebin(test, (1,8)) # Expand Columns
array([ 0,  5, 10, 15, 20, 25, 30, 30])
>>> rebin(test, (2,8)) # Expand Rows and Columns
array([[ 0,  5, 10, 15, 20, 25, 30, 30],
       [ 0,  5, 10, 15, 20, 25, 30, 30]])
>>> data = np.array([[ -5,   4,   2,  -8,   1],
                     [  3,   0,   5,  -5,   1],
                     [  6,  -7,   4,  -4,  -8],
                     [ -1,  -5, -14,   2,   1]])
>>> rebin(data, (8,10)) # 2D Array example
array([[ -5,  -1,   4,   3,   2,  -3,  -8,  -4,   1,   1],
       [ -1,   0,   2,   2,   3,  -2,  -7,  -3,   1,   1],
       [  3,   1,   0,   2,   5,   0,  -5,  -2,   1,   1],
       [  4,   0,  -4,   0,   4,  -1,  -5,  -5,  -4,  -4],
       [  6,  -1,  -7,  -2,   4,   0,  -4,  -6,  -8,  -8],
       [  2,  -2,  -6,  -6,  -5,  -3,  -1,  -3,  -4,  -4],
       [ -1,  -3,  -5, -10, -14,  -6,   2,   1,   1,   1],
       [ -1,  -3,  -5, -10, -14,  -6,   2,   1,   1,   1]])
>>> rebin(data, (2,5)) # Compression
array([[-1,  2,  3, -6,  1],
       [ 2, -6, -5, -1, -3]])
>>> to_compress = np.array([[3, 9, 7, 0, 1, 5],
                            [7, 7, 1, 9, 7, 3],
                            [9, 2, 2, 3, 1, 1],
                            [0, 3, 5, 0, 4, 3],
                            [5, 7, 7, 1, 9, 1],
                            [7, 2, 1, 1, 3, 0]])
>>> rebin(to_compress, (3,3)) # Compression
array([[6, 4, 4],
       [3, 2, 2],
       [5, 2, 3]])

References

[1] https://stackoverflow.com/a/8090605

See also

pyfhd.pyfhd_tools.pyfhd_utils.rebin_rows: Expand the number of rows through interpolation
pyfhd.pyfhd_tools.pyfhd_utils.rebin_columns: Expand the number of columns through interpolation

pyfhd.pyfhd_tools.pyfhd_utils.rebin_columns(a: ndarray[tuple[Any, ...], dtype[integer | floating | complexfloating]], ax: int, shape: tuple, col_sizer: int) → ndarray[tuple[Any, ...], dtype[integer | floating | complexfloating]][source]

Performs expansion on the columns of a 1D or 2D array using interpolation to fill in the values that are created by expanding in the space between existing values. This function assumes the rows have already been expanded to the required number.

Parameters:

a (NDArray[np.integer | np.floating | np.complexfloating]) – An array to be expanded
ax (int) – The axis we’re expanding, almost always set to 1
shape (tuple) – The shape of the original array
col_sizer (int) – The number of columns we’re adding in

Returns:

rebinned – An interpolated array of a containing shape[1] * col_sizer columns

Return type:

NDArray[np.integer | np.floating | np.complexfloating]

See also

pyfhd.pyfhd_tools.pyfhd_utils.rebin_rows: Expand the number of rows through interpolation
pyfhd.pyfhd_tools.pyfhd_utils.rebin: Expand or Contract an array based on a given shape

pyfhd.pyfhd_tools.pyfhd_utils.rebin_rows(a: ndarray[tuple[Any, ...], dtype[integer | floating | complexfloating]], ax: int, shape: tuple, old_shape: tuple, row_sizer: int) → ndarray[tuple[Any, ...], dtype[integer | floating | complexfloating]][source]

Performs expansion on the rows of array a to the number of rows in shape[0] using interpolation to fill between any new values added when adding new rows between existing values.

Parameters:

a (NDArray[np.integer | np.floating | np.complexfloating]) – The array to be rebinned
ax (int) – The axis we’re expanding, almost always set to 0
shape (tuple) – The new shape we’re expanding to
old_shape (tuple) – The shape of the array a
row_sizer (int) – The number of rows we’re expanding by

Returns:

row_rebinned – The interpolated array with row_sizer extra columns between existing rows

Return type:

NDArray[np.integer | np.floating | np.complexfloating]

See also

pyfhd.pyfhd_tools.pyfhd_utils.rebin_columns: Expand the number of columns through interpolation
pyfhd.pyfhd_tools.pyfhd_utils.rebin: Expand or Contract an array based on a given shape

Replicates IDL’s Region Grow, where a region of interest will grow based upon a given threshold within a 2D array. It finds all the pixels within the array that are connected neighbors via the threshold and blob detection using SciPy’s label function. In this case, the standard deviation form of this function hasn’t been implemented as pyfhd will only use this function once with a threshold.

If you want to use standard deviation region growing adjusting the function can be done by potentially implementing skimage’s blob detection algorithms for the labelling and keeping the rest the same.

Parameters:

image (NDArray[np.integer | np.floating | np.complexfloating]) – A 2D array of pixels
roiPixels (NDArray[np.integer]) – The region of interest given as FLAT indexes i.e. array.flat
low (int | float | None, optional) – The low threshold, any number below this is considered background, If left as None, this will be the lowest value of the region of interest, by default None
high (int | float | None, optional) – The high threshold, any number higher than this is considered background, If left as None, this will be the highest value of the region of interest, by default None

Returns:

growROIPixels – The grown region of interest that has connected neighbours by using the threshold

Return type:

NDArray[np.integer | np.floating | np.complexfloating] | None

See also

scipy.ndimage.label: Labels an image based off a given kernel

Notes

‘scikit-image Blob Detection’ : https://scikit-image.org/docs/stable/auto_examples/features_detection/plot_blob.html

pyfhd.pyfhd_tools.pyfhd_utils.reshape_and_average_in_time(vis_array: ndarray[tuple[Any, ...], dtype[complex128]], n_freq: int, n_time: int, n_baselines: int, vis_weights: ndarray[tuple[Any, ...], dtype[float64]]) → ndarray[tuple[Any, ...], dtype[complex128]][source]

Given a single polarisation 2D vis_array of shape (n_freq, n_time*n_baselines), reshape into (n_freq, n_time, n_baselines), and then average in time, weighting by vis_weights (must be of shape (n_freq, n_time, n_baselines)) Returns the averaged array in shape (n_freq, n_baselines)

Parameters:

vis_array (NDArray[np.complex128]) – The visibility array
n_freq (int) – Number of frequencies
n_time (int) – Number of time steps
n_baselines (int) – Number of baselines
vis_weights (NDArray[np.float64]) – The visibility weights array

Returns:

reshape_array – The reshaped visibility array the same shape as the visibility weights array

Return type:

NDArray[np.complex128]

pyfhd.pyfhd_tools.pyfhd_utils.resistant_mean(array: ndarray[tuple[Any, ...], dtype[integer | floating | complexfloating]], deviations: int, mad_scale: float = 0.6745, sigma_coeff: ndarray[tuple[Any, ...], dtype[float64]] = array([0.020142, -0.23584, 0.90723, -0.15405])) → int | float | complex | number[source]

The resistant_mean function translate the IDLAstro function resistant_mean from IDL to Python using NumPy. The values mad_scale and sigma_coeff are also retrieved from the same IDLAstro function when running in Double Precision Mode.

The resistant_mean gets the mean of an array which has had a median absolute deviation threshold applied to the absolute deviations of the array to exclude outliers.

If resistant_mean needs to be optimized, it can be vectorized easily enough

Parameters:

array (NDArray[np.integer | np.floating | np.complexfloating]) – A 1 dimensional array of values, multidimensional arrays should be flattened before use
deviations (int) – The number of median absolute deviations from the median we want use to exclude outliers
mad_scale (float, optional) – The scale factor for the median absolute deviation, by default 0.67449999999999999
sigma_coeff (NDArray[np.float64], optional) – The coefficients applied to the polynomial equation to the standard deviation of the points excluded by the outliers for additional exclusion, by default np.array([0.020142000000000000, -0.23583999999999999 , 0.90722999999999998 , -0.15404999999999999])

Returns:

resistant_mean – The mean of the array with outliers excluded using median absolute deviation

Return type:

Number

References

pyfhd.pyfhd_tools.pyfhd_utils.run_command(cmd: str, dry_run=False)[source]

Runs the command string cmd using subprocess.run. Returns any text output to stdout

Parameters:

cmd (str) – The command to run on the command line
dry_run (bool) – If True, don’t actually run the command. Defaults to False (so defaults to running the command)

pyfhd.pyfhd_tools.pyfhd_utils.simple_deproject_w_term(obs: dict, params: dict, vis_arr: ndarray[tuple[Any, ...], dtype[complex128]], direction: float, logger: Logger) → ndarray[tuple[Any, ...], dtype[complex128]][source]

Applies a w-term deprojection to the visibility array

Parameters:

obs (dict) – The observation data structure
params (dict) – The data from the UVFITS file
vis_arr (NDArray[np.complex128]) – The visibility array
direction (float) – The direction we apply to the phase

Returns:

vis_arr – The visibility array with the deprojection applied

Return type:

NDArray[np.complex128]

pyfhd.pyfhd_tools.pyfhd_utils.split_vis_weights(obs: dict, vis_weights: ndarray[tuple[Any, ...], dtype[float64]]) → tuple[ndarray[tuple[Any, ...], dtype[float64]], ndarray[tuple[Any, ...], dtype[int64]]][source]

Separate the indices in the visibility array (data/model/res/weights) into interleaved time samples, generally called “even” and “odd” depending on whether the time index is even or odd. Interleaved data can be used to generate cross power spectra and propagate uncertainties, see eq 2 of Jacobs et al. 2016. Ensures that the same flagging is applied to both sets.

Parameters:

obs (dict) – Observation metadata dictionary
vis_weights (NDArray[np.float64]) – Visibility weights array

Returns:

vis_weights (NDArray[np.float64]) – Full visibility weights array with the same flagging applied to both interleaved sets.
bi_use NDArray[np.int64] – Baseline index arrays for interleaved time samples, separated by “even” and “odd” indices.

pyfhd.pyfhd_tools.pyfhd_utils.vis_noise_calc(obs: dict, vis_arr: ndarray[tuple[Any, ...], dtype[complex128]], vis_weights: ndarray[tuple[Any, ...], dtype[float64]], bi_use: ndarray[tuple[Any, ...], dtype[int64]] | None = None) → ndarray[tuple[Any, ...], dtype[float64]][source]

Calculate the noise from the calibrated visibilities by taking the difference between the imaginary parts of the interleaved time samples. A factor of sqrt(2) is required because there is half as many independent noise samples when calculated from just the imaginary part.

Parameters:

obs (dict) – The observation metadata dictionary
vis_arr (NDArray[np.complex128]) – The vsisibility array
vis_weights (NDArray[np.float64]) – The visibility weights array
bi_use (NDArray[np.int64] | None, optional) – Baseline index arrays for interleaved time samples, separated by “even” and “odd” indices, by default None

Returns:

noise_arr – Calculated variance of the noise as a function of polarisation and frequency.

Return type:

NDArray[np.float64]

pyfhd.pyfhd_tools.pyfhd_utils.vis_weights_update(vis_weights: ndarray[tuple[Any, ...], dtype[float64]], obs: dict, psf: dict | File, params: dict) → tuple[ndarray[tuple[Any, ...], dtype[float64]], dict][source]

Update the visibility weights array to match any updates to the observation metadata dictionary, including flagged times, frequencies, tiles, and min/max baseline length.

Parameters:

vis_weights (NDArray[np.float64]) – Visibility weights array.
obs (dict) – Observation metadata dictionary
psf (dict | h5py.File) – Beam dictionary
params (dict) – Visibility metadata dictionary

Returns:

vis_weights (NDArray[np.float64]) – Updated vis_weights given the modified observation metadata dictionary
obs (dict) – The observation metadata dictionary, now containing the correct summary statistics of the new flagging

The weights invert function cleans the weights given by removing the values that are 0, NaN or Inf ready for additional calculation. If a threshold is set, then we check the values that match the threshold instead of checking for zeros.

Parameters:

weights (NDArray[np.integer | np.floating | np.complexfloating] | int | float | np.number) – An array of values of some dtype
threshold (float | None, optional) – A real number set as the threshold for the array. By default its set to None, in this case function checks for zeros, by default None
use_abs (bool, optional) – If True, take the absolute value (sometimes useful for complex numbers) By default this is False, so will leave as a complex number and invert, by default False

Returns:

result – The weights array that has had NaNs and Infinities removed, and zeros OR values that don’t meet the threshold.

Return type:

Testing Utilities

pyfhd.pyfhd_tools.test_utils.convert_to_h5(test_path: Path, save_path: Path, *args: list[Path]) → None[source]

For every file specified as an arg, read the file from the test_path into a python dictionary. If it’s a dict or recarray that contaisn recarrays, convert all the recarrays using recarray_to_dict. The files can be .npy or .sav files. The python dict will then be written into a HDF5 file for testing purposes.

This function was made to convert many of the .npy and .sav files into something that can be read and written more easily by other packages other than numpyt or scipy.

Parameters:

test_path (Path) – The path to a directory with all the files inside it
save_path (Path) – The path to the file for saving the HDF5
*args (list[Path]) – A list of file names to be read in, can be .npy or .sav files

pyfhd.pyfhd_tools.test_utils.get_data(data_dir: Path, data_filename: str, *args: list[str]) → list[source]

This function is designed to read npy or sav files in a data directory inside test_fhd_*. Ensure the data file has been made with the scripts inside the scripts directory. Use splitter.py to put the files and directories in the right format if you have used histogram runner and rebin runner. Paths are expected to be of data_dir/data/function_name/[data,expected]_filename.npy data_dir is given by pytest-datadir, it should be the directory where the test file is in.

Parameters:

data_dir (Path) – This should be the dir passed through from pytest-datadir
data_filename (atr) – The name of the file for the input
*args (list[str]) – If given, is expected to be more filenames

Returns:

return_list – Contains just the input if only one file given, otherwise, it also gives the output if other files given

Return type:

list

pyfhd.pyfhd_tools.test_utils.get_data_items(data_dir: Path, data_with_item_path: Path, *args: list[str]) → list[source]

Takes all the path inputs from tests and processes them so they’re ready for use.

Parameters:

data_dir (Path) – Path to the data directory
data_with_item_path (Path) – Path to the data that contains only an item
*args (Paths) – Give more paths to more data with items that need to be extracted

Returns:

return_list – Variable(s) required to do the test

Return type:

list

pyfhd.pyfhd_tools.test_utils.get_data_sav(data_dir: Path, sav_file: Path, *args: list[Path]) → list[source]

Takes all the path inputs from tests and processes them so they’re ready for use. More specifically takes in sav files

Parameters:

data_dir (Path) – Path to the data directory
sav_file (Path) – Path to the sav file, which will load a python dictionary
args (list[Path]) – If given, is expected to be more filenames

Returns:

return_list – Contains just the data if only one file given, otherwise, it also gives the output if other files given

Return type:

list

pyfhd.pyfhd_tools.test_utils.get_savs(data_dir: Path, sav_file: Path, *args: list[Path]) → dict | list[dict][source]

Takes in the path for many sav files and reads them without reading their keys. Assumes the sav files here have more than one key. If you use one sav_path only then the function acts as a wrapper for scipy’s readsav.

Parameters:

data_dir (Path) – Path to the data directory
sav_file (Path) – Path to the sav file, which will load a python dictionary
args (Paths) – If given, is expected to be more filenames

Returns:

data – Either a dict of one sav file or the dicts of multiple sav files

Return type:

dict | list[dict]

pyfhd.pyfhd_tools.test_utils.print_types(dictionary: dict, dict_name: str, indent_level: int = 1) → None[source]

When generating the tests, Sometimes I’d find it useful to see the types of all the keys and value pairs inside the dictionary I’m manipulating. The Debug mode is helpful for this too, but this can be easily used inside a notebook if experimenting in there too.

Parameters:

dictionary (dict) – The dictionary to print the types of
dict_name (str) – The name of the dict
indent_level (int) – Sets the indent levels for printing as it’s a recursive function, by default 1

pyfhd.pyfhd_tools.test_utils.sav_file_vis_arr_swap_axes(sav_file_vis_arr: ndarray[tuple[Any, ...], dtype[_ScalarT]]) → ndarray[tuple[Any, ...], dtype[_ScalarT]][source]

After saving arrays from IDL like vis_arr and vis_model_arr into and IDL .sav file, and subsequently loading in via scipy.io.readsav, they come out in a shape/format unsuitable for pyfhd. Use this function to reshape into shape = (n_pol, n_freq, n_baselines)

Parameters:: sav_file_vis_arr (NDArray) – Array as read in by scipy.io.readsav, if n_pol = 2 should have shape=(2,)
Returns:: Returns the array with shape=(n_pol, n_freq, n_baselines)
Return type:: NDArray

pyfhd.pyfhd_tools.test_utils.try_assert_all_close(actual: ndarray[tuple[Any, ...], dtype[_ScalarT]], target: ndarray[tuple[Any, ...], dtype[_ScalarT]], name: str, tolerance=1e-08) → None[source]

Uses the numpy testing assert_all_close but uses a try and except wrapper around it to print the error instead of doing an AssertionError which stops the running of the program. This is helpful when doing testing with expected precision errors, but wanting to avoid stopping the program or constantly setting the tolerances on multiple assert statements.

Parameters:

actual (NDArray) – The array we calculated
target (NDArray) – The array we actually want to calculate
name (str) – The name of the variable we are testing
tolerance (float, optional) – This is the tolerance for the error in absolute values, by default 1e-8