Outputs¶
This page documents every output artifact produced by the pipeline, with particular emphasis on the final merged catalog columns and their physical units.
Directory Layout¶
With default settings, outputs.work_dir contains:
work_dir/
├── EPSFs/ # ePSF artifacts
│ ├── <band>/
│ │ └── r00_c00/
│ │ ├── epsf.npy # ePSF array (normalized)
│ │ ├── epsf.png # ePSF stamp visualization
│ │ ├── epsf_growth_curve.png # Encircled-energy diagnostic
│ │ ├── epsf_residual_diagnostics.png # Star minus ePSF residual diagnostics
│ │ ├── meta.json # ePSF build metadata
│ │ ├── psfstars.png # PSF star selection overlay
│ │ ├── psfstars_bgsub.png # PSF star overlay on background-subtracted patch
│ │ ├── background_diagnostics.png # Raw/background/bg-sub patch panels
│ │ ├── star_local_background_diagnostics.png # Per-star raw/annulus/bg-sub diagnostics
│ │ ├── extracted_stars.png # Extracted star cutout montage
│ │ └── used_stars.png # Fitted star cutout montage
│ └── epsf_summary.json # Summary across all bands and cells
├── patches/ # Patch geometry definitions
│ ├── patches.csv
│ └── patches.json
├── patch_payloads/ # Compressed per-patch input data
│ └── r00_c00_pr00_pc00.pkl.gz
├── outputs/ # Per-patch Tractor fit results
│ └── r00_c00_pr00_pc00/
│ ├── r00_c00_pr00_pc00.log # Patch fit log
│ ├── r00_c00_pr00_pc00.runner.log # Subprocess stdout/stderr
│ ├── r00_c00_pr00_pc00_cat_fit.csv # Patch fit catalog
│ ├── meta.json # Patch metadata + PSF audit
│ ├── patch_overview.png # Data/Model/Residual overview
│ └── cutouts/
│ ├── src_{ID}.png # Per-source montage (named by source ID)
│ └── ...
├── cropped_images/ # Crop diagnostic plots
│ ├── white_before_crop.png
│ └── white_after_crop.png
├── bright_mask/ # Bright unmatched-source mask diagnostics
│ ├── white_bright_mask.fits # Raw boolean mask (1 = masked)
│ └── white_bright_mask_overlay.png # White-stack overlay with mask and detections
├── overlay/ # White-stack overlay plots
│ ├── white_overlay.png
│ └── white_overlay_zoom.png
├── wcs.fits # Runtime WCS snapshot (working pixel frame)
├── ZP/ # Zero-point calibration (if zp.enabled)
│ ├── input_catalog_*_with_Gaia.csv # Augmented input catalog
│ ├── gaia_augmentation_overlay.png # Source selection overlay
│ ├── zp_vs_mag__<band>.png # Per-band ZP diagnostic plots (constrained y-range)
│ ├── zp_vs_mag__<band>_fullrange.png # Per-band ZP diagnostic plots (adaptive y-range)
│ ├── zp_summary.csv # Per-band ZP summary
│ └── zp_all_bands__per_object.csv # Per-star ZP values
├── config_used_{timestamp}.yaml # Snapshot of the config used for this run
├── {command}.log # Per-command log file (e.g. run.log, merge.log)
└── output_catalog.csv # Final merged catalog
ePSF Patch Diagnostics¶
Each EPSFs/<band>/<epsf_tag>/ directory contains ePSF QA artifacts:
| File | Description |
|---|---|
psfstars.png |
PSF-star selection overlay on the raw ePSF cell image. |
psfstars_bgsub.png |
PSF-star selection overlay on the background-subtracted ePSF cell image. |
background_diagnostics.png |
Three panels: raw patch, estimated background map, raw-minus-background. Written only when epsf.save_patch_background_diagnostics: true. |
star_local_background_diagnostics.png |
Per-star diagnostics (one row per used star): raw stamp, annulus sample pixels, background-subtracted stamp. Written only when epsf.save_star_local_background_diagnostics: true and local-background subtraction is enabled. If id_label is missing/duplicated, this file is skipped with a warning. |
epsf_growth_curve.png |
Encircled-energy growth curve of the final ePSF stamp. |
epsf_residual_diagnostics.png |
Median star/model/residual panel plus normalized residual histogram. |
Per-cell meta.json includes local_bkg_diagnostics.local_bkg_diag_status to indicate whether star_local_background_diagnostics.png was written (ok) or skipped (skipped_*).
Per-Patch Output Directory¶
Each patch produces a subdirectory under outputs/ named by its patch tag (e.g. r05_c02_pr04_pc00). Contents:
| File | Description |
|---|---|
<tag>.log |
Tractor optimization log: iteration-by-iteration dlnp values, convergence status, PSF choices. |
<tag>.runner.log |
Full subprocess command line and captured stdout/stderr. Useful for debugging crashes. |
<tag>_cat_fit.csv |
Per-patch fitted catalog. Contains all input columns plus fit result columns. |
meta.json |
Patch metadata including PSF audit (which bands used ePSF vs fallback), patch geometry, fitting mode, and source counts (n_sources for base sources, n_halo_sources for halo neighbors included in fitting but not in the output CSV). In multi-band mode, also includes optimizer summary (niters, converged, hit_max_iters). When staged fitting is enabled (patch_run.enable_staged_fit: true), includes staged_fit_enabled and staged_fit_stages (or staged_fit_stages_by_band for single-band) with per-stage iteration counts and convergence status. |
patch_overview.png |
Three-panel plot: Data (white stack), Model+Sky, Residual. Source positions shown as crosshairs. |
cutouts/src_{ID}.png |
Per-source cutout montage for base sources (one row per panel: Data, Model+Sky, Residual; one column per band). Crosshairs show original (lime, dashed) and fitted (magenta, solid) positions. Filename uses the source ID (e.g. src_00019.png, src_gaia_12345.png). If ID is absent, falls back to zero-padded index (src_000000.png). Filesystem-unsafe characters in the ID are replaced with underscores. |
cutouts/src_halo_{ID}.png |
Per-source cutout montage for halo sources. Same layout as base source montages, but crosshairs use cyan (fitted) and yellow (original) instead of magenta/lime. Halo sources are fitted as part of the patch model but their results are not included in the output CSV. |
Final Merged Catalog¶
File: output_catalog.csv (or as set by outputs.final_catalog)
The merge stage performs a left join from the base catalog onto the concatenated patch fit results, using the merge key (ID if present, otherwise RA+DEC). The base catalog is the augmented catalog (when zp.enabled) or the original input catalog (otherwise). Either way, it includes all sources — both active and excluded — with their flag columns.
As a result:
- Every row in the base catalog appears in the output, even if the source was excluded from fitting.
- Fit columns are empty (
NaN) for sources that were excluded or not assigned to any patch.
Input Catalog Columns (preserved)¶
All columns from the base catalog are preserved in their original form. These include ID, RA, DEC, TYPE, and any FLUX_*, ELL, THETA, Re, POS_ERR columns you provided, plus gaia_source_id and exclusion flag columns added during the load and augmentation stages. When zp.gaia_pos_err_pix is set, newly injected Gaia rows in the augmented base catalog also carry POS_ERR.
Exclusion Tracking Columns¶
These columns are added by the pipeline to indicate why a source was excluded from fitting.
| Column | Type | Description |
|---|---|---|
excluded_crop |
bool | True if the source was flagged by crop filtering (fell outside the crop margin). |
excluded_saturation |
bool | True if the source was flagged by saturation-cut filtering (near saturated pixels). |
excluded_any |
bool | True if excluded by any reason (logical OR of all exclusion flags). |
excluded_reason |
string | Human-readable exclusion reason. Values: "crop", "saturation", "crop+saturation", or "" (not excluded). |
affected_by_bright_mask |
bool | True if the source's position falls inside a bright-source mask ellipse. The source is still fitted, but its surrounding pixels may be partially or fully masked (invvar = 0), potentially degrading fit quality. Only present when bright_mask.enabled: true. |
Fitted Position Columns¶
The column names depend on the fitting mode (enable_multi_band_simultaneous_fitting).
Multi-band simultaneous fitting (default, true): position is shared across bands.
| Column | Type | Unit | Description |
|---|---|---|---|
x_pix_patch_fit |
float | pixels | Fitted x position in the local patch coordinate frame. |
y_pix_patch_fit |
float | pixels | Fitted y position in the local patch coordinate frame. |
x_pix_white_fit |
float | pixels | Fitted x position in the full (post-crop) white-stack coordinate frame. Computed as x_pix_patch_fit + x0_roi. |
y_pix_white_fit |
float | pixels | Fitted y position in the full (post-crop) white-stack coordinate frame. Computed as y_pix_patch_fit + y0_roi. |
RA_fit |
float | degrees (ICRS) | Fitted right ascension, computed from (x_pix_white_fit, y_pix_white_fit) via WCS. |
DEC_fit |
float | degrees (ICRS) | Fitted declination, computed from (x_pix_white_fit, y_pix_white_fit) via WCS. |
Single-band independent fitting (false): position is fitted independently per band. For each band {band}:
| Column | Type | Unit | Description |
|---|---|---|---|
x_pix_patch_{band}_fit |
float | pixels | Fitted x position in the local patch frame for this band. |
y_pix_patch_{band}_fit |
float | pixels | Fitted y position in the local patch frame for this band. |
x_pix_white_{band}_fit |
float | pixels | Fitted x position in the white-stack frame for this band. |
y_pix_white_{band}_fit |
float | pixels | Fitted y position in the white-stack frame for this band. |
RA_{band}_fit |
float | degrees (ICRS) | Fitted right ascension for this band. |
DEC_{band}_fit |
float | degrees (ICRS) | Fitted declination for this band. |
Fitted Flux Columns¶
For each band {band} present in the images (matching FILTER header values). Column names are the same in both fitting modes:
| Column | Type | Unit | Description |
|---|---|---|---|
FLUX_{band}_fit |
float | scaled counts (nominal ZP = zp_ref) |
Fitted flux in the given band. Images are scaled so that the nominal ZP equals zp_ref, but see note below. |
FLUXERR_{band}_fit |
float | scaled counts (nominal ZP = zp_ref) |
Flux uncertainty (1-sigma), derived from the Tractor's parameter variance at the optimized solution. |
Flux system and approximate vs calibrated magnitudes
All fitted fluxes are in the scaled system defined by image_scaling.zp_ref (default 25.0). The formula m_approx = -2.5 * log10(FLUX_{band}_fit) + zp_ref gives only an approximate AB magnitude. This approximation assumes ZP_AUTO in the FITS header is the exact zero-point, but in practice ZP_AUTO has errors, and the Tractor photometry method differs from whatever method was used to derive ZP_AUTO. For calibrated AB magnitudes, use the MAG_{band}_fit column (when zp.enabled), which applies the ZP derived from Gaia-matched stars: MAG_{band}_fit = ZP_median - 2.5 * log10(FLUX_{band}_fit). The offset ZP_median - zp_ref reflects primarily the error in ZP_AUTO. When ZP calibration is well-behaved, this offset is small. Note that ZP_median and ZP_AUTO live in different flux systems (scaled vs original), so comparing them directly is not meaningful — see Pipeline Behavior — ZP Calibration.
Fitted Morphology Columns¶
Multi-band simultaneous fitting (default, true): morphology is shared across bands.
| Column | Type | Unit | Description |
|---|---|---|---|
stype_fit |
string | — | Source model used for fitting: "star", "exp", "dev", or "sersic". |
sersic_n_fit |
float | dimensionless | Fitted Sersic index (only for sersic type; NaN otherwise). |
re_pix_fit |
float | pixels | Fitted effective (half-light) radius in the Tractor's internal parameterization. |
ab_fit |
float | dimensionless | Fitted axis ratio b/a (range 0–1). |
phi_deg_fit |
float | degrees | Fitted position angle in the Tractor's internal convention (modulo 180). |
ELL_fit |
float | dimensionless | Fitted ellipticity: 1 - ab_fit. Comparable to the input ELL column. |
Re_fit |
float | pixels | Fitted effective radius (same as re_pix_fit; provided for naming symmetry with input Re). |
THETA_fit |
float | degrees | Fitted position angle, approximately converted back to SExtractor convention: phi_deg_fit - 90. Comparable to the input THETA column. |
Single-band independent fitting (false): morphology is fitted independently per band. For each band {band}:
| Column | Type | Unit | Description |
|---|---|---|---|
stype_fit |
string | — | Source type (same across bands, from input TYPE column). |
sersic_n_{band}_fit |
float | dimensionless | Fitted Sersic index for this band. |
re_pix_{band}_fit |
float | pixels | Fitted effective radius for this band. |
ab_{band}_fit |
float | dimensionless | Fitted axis ratio for this band. |
phi_deg_{band}_fit |
float | degrees | Fitted position angle for this band. |
ELL_{band}_fit |
float | dimensionless | Fitted ellipticity for this band: 1 - ab_{band}_fit. |
Re_{band}_fit |
float | pixels | Fitted effective radius for this band. |
THETA_{band}_fit |
float | degrees | Fitted position angle for this band (SExtractor convention). |
Morphology columns for point sources
For stype_fit = "star", all morphology columns are NaN (in both fitting modes).
Optimizer Diagnostic Columns¶
Multi-band simultaneous fitting (default, true): one set of optimizer diagnostics per source.
| Column | Type | Description |
|---|---|---|
opt_converged |
bool | True if the optimizer converged (dlnp dropped below patch_run.dlnp_stop). |
opt_hit_max_iters |
bool | True if the optimizer did NOT converge and effectively exhausted iteration budget (iterations ≥ n_opt_iters - flag_maxiter_margin). |
opt_niters |
int | Number of optimizer iterations actually performed. |
opt_last_dlnp |
float | Last delta-log-probability value from the optimizer. |
flag_bound_stalled |
bool | True if any hard-bounded parameter for this source is at or near its limit after optimization. This indicates the optimizer may have been stuck at a constraint boundary rather than converging to a true optimum. |
flag_bound_stalled_which_parameter |
string | Comma-separated list of parameters at their bounds, encoded as {param}@{side}. Examples: pos.x@upper, shape.logre@lower, sersicindex@upper. Empty string if flag_bound_stalled is False. |
Single-band independent fitting (false): per-band optimizer diagnostics. For each band {band}:
| Column | Type | Description |
|---|---|---|
opt_converged_{band} |
bool | Whether the optimizer converged for this band. |
opt_hit_max_iters_{band} |
bool | Whether the optimizer exhausted iterations for this band. |
opt_niters_{band} |
int | Number of iterations for this band. |
opt_last_dlnp_{band} |
float | Last dlnp for this band. |
flag_bound_stalled_{band} |
bool | Whether any bounded parameter is at its limit for this band. |
flag_bound_stalled_which_parameter_{band} |
string | Parameters at their bounds for this band (same encoding as multi-band). |
Patch and PSF Audit Columns¶
| Column | Type | Description |
|---|---|---|
patch_tag |
string | Patch identifier (e.g. r05_c02_pr04_pc00). |
epsf_tag |
string | Parent ePSF cell identifier (e.g. r05_c02). |
psf_min_epsf_nstars_for_use |
int | Quality gate threshold used for this patch. |
psf_used_epsf_band_count |
int | Number of bands that used a real ePSF. |
psf_fallback_band_count |
int | Number of bands that used a fallback PSF. |
psf_low_star_band_count |
int | Number of bands where ePSF existed but was rejected due to low star count. |
psf_fallback_bands |
string | Comma-separated list of bands that used fallback PSF. |
psf_low_star_bands |
string | Comma-separated list of bands with low-star ePSF rejection. |
psf_frozen_bands_for_optimizer |
string | Comma-separated list of bands where GaussianMixturePSF params were frozen for optimizer stability. |
psf_fallback_reasons_json |
string (JSON) | JSON dict mapping band → fallback reason string. |
Reliability Summary Columns¶
These columns provide a single-column summary of source fit reliability, computed during the merge stage from the exclusion, optimizer, bound-stall, and bright-mask flags above.
| Column | Type | Description |
|---|---|---|
fit_is_reliable |
bool | True if the fit is considered trustworthy. False if any hard-fail condition applies (see below). |
fit_quality_issues |
string | Comma-separated list of all quality concerns for this source (both hard and soft). Empty string if none. |
Hard-fail conditions (set fit_is_reliable = False):
| Tag | Trigger |
|---|---|
excluded |
excluded_any is True. |
no_fit_result |
Source is not excluded but has no fitted position (patch crash or unassigned). |
hit_max_iters |
opt_hit_max_iters is True (any band in single-band mode). |
bound_stalled |
flag_bound_stalled is True (any band in single-band mode). |
bright_mask |
affected_by_bright_mask is True. |
Soft warnings (recorded in fit_quality_issues only, do not affect fit_is_reliable):
| Tag | Trigger |
|---|---|
not_converged |
Optimizer did not formally converge but did not hit max iterations either. |
all_psf_fallback |
All bands used an analytic fallback PSF instead of ePSF. |
Internal Coordinate Columns¶
These columns are added during patch input building and carried through to the output. They are useful for debugging spatial assignments.
| Column | Type | Unit | Description |
|---|---|---|---|
x_pix_white |
float | pixels | Source x position in the white-stack frame (from WCS projection of input RA/DEC). |
y_pix_white |
float | pixels | Source y position in the white-stack frame. |
x_pix_patch |
float | pixels | Source x position in the local patch frame (= x_pix_white - x0_roi). |
y_pix_patch |
float | pixels | Source y position in the local patch frame. |
Gaia Augmentation Column¶
When zp.enabled: true, the augmented catalog adds this column:
| Column | Type | Description |
|---|---|---|
gaia_source_id |
string | Gaia DR3 source_id for sources matched to (or injected from) the GaiaXP synphot catalog. Empty string for non-Gaia sources. |
Zero-Point Calibrated Magnitude Columns¶
When zp.enabled: true, the ZP computation stage adds these columns for each band {band}:
| Column | Type | Unit | Description |
|---|---|---|---|
MAG_{band}_fit |
float | mag (AB) | AB magnitude: ZP_median - 2.5 * log10(FLUX_{band}_fit). NaN for sources with non-positive flux. |
MAGERR_{band}_fit |
float | mag | Magnitude error: sqrt((2.5/ln10 * FLUXERR/FLUX)^2 + ZP_err^2). Propagates both flux error and per-band ZP uncertainty. ZP_err is either zp_err_mad or zp_err_std depending on zp.zp_err_method (default "all_mad"). NaN if flux error is missing. |
Diagnostic Plots¶
Bright-Source Mask Diagnostics¶
When bright_mask.enabled: true (default), the bright_mask/ directory contains:
| File | Description |
|---|---|
white_bright_mask.fits |
Boolean FITS mask (1 = masked pixel). Combines bright unmatched-source ellipses with any-band saturation mask. |
white_bright_mask_overlay.png |
White-stack image with the mask shown as a semi-transparent red overlay. Four marker categories are shown (see below). A legend with counts is included. |
The overlay plot uses two independent matching passes with bright_mask.match_radius_pix:
| Marker | Color/Shape | Meaning |
|---|---|---|
| Open circle | Lime | SEP matched — SEP detection that has a matching active (non-excluded) catalog source nearby. These are known sources; not masked. |
| X | Red | SEP unmatched — SEP detection with no catalog match. These bright contaminants generate the mask ellipses. |
| Diamond | Blue (deepskyblue) | Cat matched — input catalog source (including excluded sources) that has a nearby SEP detection. |
| Triangle | Orange | Cat unmatched masked — catalog source with no nearby SEP detection whose position falls inside the bright mask (affected_by_bright_mask = True). |
Note: "SEP matched" and "Cat matched" are computed from opposite directions and differ subtly. The SEP-to-catalog match only considers active sources (excluded_any = False), so a SEP detection near an excluded source appears as "SEP unmatched" even though the excluded source appears as "Cat matched". Multiple SEP detections can match the same catalog source, and vice versa.
White-Stack Diagnostics¶
| File | Description |
|---|---|
cropped_images/white_before_crop.png |
White-stack image with crop box overlaid (green rectangle). Generated before crop is applied. Only when crop.enabled: true and crop.plot_pre_crop: true. |
cropped_images/white_after_crop.png |
White-stack image after crop. Only when crop.enabled: true and crop.plot_post_crop: true. |
overlay/white_overlay.png |
Current working white stack with source positions color-coded by TYPE: cyan=STAR, magenta=GAL/EXP/DEV/SERSIC, yellow squares=UNKNOWN, red X=saturation-excluded, gray=crop-excluded (legend only). Generated when overlay.enabled: true, regardless of crop.enabled. |
overlay/white_overlay_zoom.png |
Zoomed region of the overlay. Generated when overlay.zoom_enabled: true. Out-of-bounds areas are blank-filled. |
wcs.fits |
Runtime WCS snapshot in the working pixel frame (post-crop if crop enabled, full-frame otherwise). Used by merge to compute RA_fit/DEC_fit. |
Patch Overview¶
patch_overview.png in each patch output directory shows a three-panel view (Data, Model+Sky, Residual) of the white-combined patch images. When per-filter patch data is available, the white combination uses initial (pre-mask) inverse variance weights derived from the sky sigma, so that saturated and bright-masked pixels receive physically correct weights rather than falling back to equal-weight. The red overlay and display scaling use the final Tractor mask (non-finite | saturation | bright mask): pixels with invvar = 0 in all bands are shown with a semi-transparent red overlay, and the color stretch is computed from unmasked pixels only. Source positions are marked:
Base sources:
- Orange X — original (input catalog) positions
- Deepskyblue X — fitted positions
- Cyan line — connecting line from each original position to its fitted position(s)
Halo sources (fitted for neighbor modeling but not output to CSV):
- Yellow diamond — original positions
- Lime diamond — fitted positions
- Lime line — connecting line from original to fitted position
The title reports both base and halo counts (e.g. n_sources=2 + 3 halo).
In single-band mode, each source has multiple blue markers (one per band, with reduced alpha/linewidth to avoid clutter), each connected to the single orange marker by a cyan line. In multi-band mode, there is one blue marker per source.
Source Cutout Montages¶
cutouts/src_{ID}.png (or cutouts/src_halo_{ID}.png for halo sources) shows a grid of panels: one column per band, three rows (Data, Model+Sky, Residual). The filename uses the source's ID value (e.g. src_00019.png, src_gaia_2518065009826205440.png, src_halo_gaia_12345.png). If ID is absent, a zero-padded index is used as fallback.
Markers for the center source (base):
- Magenta solid crosshair — fitted position
- Lime dashed crosshair — original (input) position
Markers for the center source (halo):
- Cyan solid crosshair — fitted position
- Yellow dashed crosshair — original (input) position
Markers for other base sources in the cutout:
- Deepskyblue X — fitted position
- Orange X — original position
- Cyan line — connecting line
Markers for other halo sources in the cutout:
- Lime diamond — fitted position
- Yellow diamond — original position
- Lime line — connecting line
Masked pixels (where invvar = 0, including bright-mask and bad-pixel regions) are shown as a semi-transparent red overlay on all three panels (Data, Model+Sky, Residual). The color stretch is computed from unmasked pixels only, so faint sources near masked bright contaminants are displayed with full dynamic range.
In single-band mode, each band column shows that band's own independently fitted position for the crosshairs and markers.
ZP Diagnostics¶
When zp.enabled: true, the ZP/ directory contains:
| File | Description |
|---|---|
input_catalog_*_with_Gaia.csv |
Augmented input catalog with Gaia-matched gaia_source_id column and (if inject_gaia_sources: true) injected Gaia source rows. When zp.gaia_pos_err_pix is not null, newly injected Gaia rows also include POS_ERR. |
gaia_augmentation_overlay.png |
White-stack overlay showing original sources (cyan circles), Gaia-matched originals (lime squares), injected Gaia sources (magenta circles, only when inject_gaia_sources: true), saturation-excluded Gaia sources (red X), saturation-excluded input sources (red X), crop-excluded input sources (orange X), and the selection bounding box (yellow dashed). |
zp_vs_mag__{band}.png |
Per-band ZP diagnostic scatter plot (constrained y-range): ZP vs GaiaXP synphot AB magnitude. Shows unreliable (gray diamonds), sigma-clipped (red X), and kept sources with error bars. When zp_err_method: "bright_std", kept sources are split into high-SNR (black circles) and low-SNR (grey circles) based on the active SNR threshold; the shaded band shows the std error. When zp_err_method: "all_mad", all kept sources are shown as black circles and the shaded band shows MAD. Pale grey vertical bands mark magnitude ranges outside the GaiaXP catalog coverage. x-axis: plot_mag_lo..plot_mag_hi; y-axis: median +/- plot_zp_offset. |
zp_vs_mag__{band}_fullrange.png |
Same as above but with adaptive y-axis range (based on data points, not error bars). |
zp_summary.csv |
Per-band summary: band, n_raw, n_kept, n_clipped, n_unreliable, clip_sigma, zp_median, zp_err_mad (MAD of all kept sources), zp_err_std (std of high-SNR kept sources), n_zp_err_src (number of sources used for std), zp_err_snr_used (actual SNR threshold used after any fallback). Both error columns are always populated regardless of zp_err_method. |
zp_all_bands__per_object.csv |
Per-star ZP values for all reliable Gaia-matched sources across all bands, with is_kept/is_clipped flags from sigma clipping. Unreliable sources are not included. |
Merge Behavior Details¶
The merge stage:
- Reads the base catalog (augmented catalog if ZP is enabled, otherwise the original input catalog). Exclusion flag columns (
excluded_crop,excluded_saturation) are already present from the load stage and are used directly. - Computes
excluded_anyandexcluded_reasonfrom the flag columns. - Collects all
*_cat_fit.csvfiles from per-patch output directories. - Concatenates fit results and handles duplicate merge keys (drops duplicates with a warning; rare boundary sources may appear in two patches).
- Performs a left join:
base_catalog.merge(fit_results, how="left", on=key_cols). - Computes sky coordinates from fitted pixel positions using the runtime WCS snapshot (
wcs.fits) generated duringload_inputs(). In multi-band mode:RA_fit/DEC_fit. In single-band mode:RA_{band}_fit/DEC_{band}_fitfor each band.
Merge Key Selection¶
- If
IDcolumn exists → useIDas the sole merge key. - Otherwise → use
(RA, DEC)as a composite merge key.
What "merge-missing" Means¶
The merge log reports merge-missing (left join rows without fit): N. This count includes:
- Sources excluded by crop filtering.
- Sources excluded by saturation filtering.
- Sources with
NaNcoordinates that could not be assigned to patches. - Sources in patches that failed or crashed.
Check the excluded_* columns to distinguish intentional exclusions from unexpected failures.
Merge Logging¶
The merge stage logs:
- Output file path
- Row and column count
- Exclusion counts (if exclusion columns are present): crop, saturation, any
- Merge-missing count (left-join rows without fit results)