Known Issues and Limitations
Please pay particular attention to the following problems, and especially their status as currently unresolved or fixed:
Corrupted files on disk38
Following a hardware isssue a small fraction (10 percent) of FITS files
on disk38 are showing as corrupt. We are working to identify and fix the files
but in the meantime please be aware of the issue if you are downloading files
from disk38 (the disk name shows in the URL of the download file).
Affected image files will have some extension(s) that will
fail to uncompress or open. We are not sure of the effect on catalogue files.
Issues affecting earlier releases but fixed in more recent release(s):
- UKIDSS DR7+ LAS and GCS and DR6+ GPS have a small number of frames (around 0.1%, or a few multiframes in each passband)
missing in the source merging process; corresponding rows are missing from the merge log and source tables. For a given
passband and survey, the following SQL will show the detector frames missed - in this case, 4 Y detector frames missed in the LAS:
SELECT d.multiframeID, d.extnum
FROM Multiframe m, MultiframeDetector d LEFT OUTER JOIN lasMergeLog l
on l.ymfID=d.multiframeID AND l.yenum=d.extnum
WHERE m.filterID=2 AND project LIKE '%las%' AND frametype='stack'
AND m.multiframeID=d.multiframeID AND m.deprecated=0
AND d.deprecated=0 AND (l.yenum IS NULL)
The problem is fixed for LAS and GCS in DR8 and all subsequent releases; it did not occur in DR6 and before; the problem
is fixed for the GPS in DR7 and all subsequent releases; it did not occur in DR5 and before.
- UKIDSS DR5+ contained an inconsistency between the dxsDetection photometry
and the zeropoints in MultiframeDetector for intermediate (not deep) stacks.
This was caused by an updating error on MultiframeDetector. This has been
fixed and updated at the same time as the release of the UDS-DR5 data so this
is no longer a problem for UKIDSS DR5.
- UKIDSS DR4+ contained a small amount of deprecated data in the *Source
tables due to (we believe) reprocessed data arriving at the archive end
after source merging for a data release had been done. For example,
the LAS contained 12 H-band and 6 J-band detector frames which were
reprocessed and automatically deprecated so their detections did not
appear in lasDetection; nonetheless the merged sources containing those
detections slipped through into the release. We thank Richard McMahon
for bringing this to our attention. The problem has been fixed in
DR5+ and all subsequent releases; moreover it did not affect DR3+ and
- 206 fields (204 GPS, 1 LAS and 1 GCS) have astrometric errors of order
1 arcsec. This problem is mainly confined to some of the most crowded star fields
at Galactic longitudes l < 30 degrees, and is due to a minor procedural
problem at the pipeline processing stage in the data flow system.
A list of multiframe files affected is
available here (kindly provided by Mike Irwin
at CASU). As a consequence, sources that were
detected in multiple passbands in those fields may not have been
successfully paired up in the gpsSource catalogue, since the merging
process uses a source matching radius of 1 arcsec in the GPS. The affected fields
will be fixed in DR4, but they remain in DR3 and earlier releases. Thanks are due
to Phil Lucas and Reba Bandyopadhyay for identifying this problem.
- Error in dither offset calculation - In the DR3, DR2 releases the
dither offset calculation that defines the underexposed region of stacked
images was incorrect in some LAS and DXS passbands. This resulted in fewer
sources, in these surveys, being flagged with
bit 22 and to a
lesser extent also
bit 19 in the
- Error in Gaussian sigma calculation - In all releases prior to DR4
there was an error in the Gaussian sigma calculation in the udsDetection
catalogues from deep stacks (i.e from SExtractor output). In previous releases,
we calculated sigma=sqrt(A_IMAGE**2+B_IMAGE**2). While testing a seeing and
average stellar ellipticity calculator, this was found to be incorrect. The
correct equation, derived from the equations in the SExtractor manual, is
sigma=0.5*sqrt(A_IMAGE**2+B_IMAGE**2). This has been updated in the curation
software and will be corrected in future releases. Anyone using the gaussian
sigma in the UDS deep stack catalogues should correct this. In future releases,
the UDS deep stacks will have seeing and average stellar ellipticity calculated
Incorrect values for variableClass in DXS Data Release 6
8th January 2010 fixed - DR6 was updated and the problem fixed
Adding new attributes to the Variability table led to an indexing error, which
caused all the variableClass values to be incorrectly calculated, so that all
have values of 0. The jvarClass and kvarClass values are correct though, so
these can be used to discriminate between variables and non-variables. This
problem will be fixed and the correct values released when the GPS and UDS are
appended to Data Release 6.
Photometry issue with processed WFCAM data, observation dates from 12/03/08 (08A/B)
23rd March 2009 fixed - The affected files (stack images
and catalogues) have been deprecated and replaced by version 2 (V2).
An error in flat-file catalogue photometry has been discovered that
affects all version 1 processed data having observation dates from
12th March 2008 (semester 08A) onwards. The error has been traced to
a bug in the standard nightly pipeline source extraction software.
Any users, of both survey and non-survey (i.e. private PI-led
programmes), who have downloaded data observed from 12th March 2008
should be aware that there are systematic errors of 10% or more
in various flux measures (e.g. aperture fluxes) in all catalogues
in all wavebands. Image file pixel values are not affected by the
problem, but there will be small (1% to 2%) systematic errors in
the catalogue-derived zeropoints copied into the image headers.
Please note that UKIDSS releases up to and including DR4, and PI
programmes up to and including only data to 11/03/08 in Semester
08A, are NOT affected by the problem in any way.
VDFS operations at CASU and WFAU apologise for the error and the
delay in delivery of UKIDSS Data Release 5, which will now slip
by around 6 weeks. Over the first 3 weeks, the flat file products
will be reprocessed, transfered and ingested as version 2 to
replace the affected (version 1) products.
13th October 2009 (DR6) fixed for DXS, LAS, GCS
- we have applied a
correction written by CASU to redistribute the fluxes of the components
according to their isophotal fluxes.
Erroneous extended source fluxes/radii/mags for deblends
A pipeline processing bug has resulted in all CASU processed catalogue
deblend components having erroneous values of Petrosian, Kron and Hall
fluxes, radii and magnitudes. The problem does not affect those
attributes for isolated (un-deblended) catalogue detections. Deblended
objects can be identified via the ppErrBits attribute: the
SQL construct WHERE ... AND (ppErrBits&0x10) > 0 can be used to filter
such sources. Users interested in extended source fluxes for deblended
components should use the isophotal values for the time being; at the time
of writing (November 2008) we are planning to fix this problem in DR6
Badly stacked DXS intermediate stack.
A problem has been noted with a DXS intermediate stack in the ELIAS-N1 region.
This stack was incorrectly stacked in extensions 4 and 5, for reasons unknown
and has been missed by quality control so far. In extension 4, there are 3
images for each object (of similar brightness), and in extension 4 there are
two images (one bright and one faint). Extensions 2 and 3 appear to be fine.
This image has multiframeID=232788. The intermediate stack is one of several
that are components of deep stack with productID=95, pointing EN011_1. This
deep stack has been released in all data releases apart from DR1, where
tougher than optimal seeing constraints rejected it.
These deep stack images look fine, and any additional objects have
probably been clipped out. Since the zeropoints are calculated on an
extension by extension basis, most objects in these frames will have the
correct magnitude, as the incorrect weighting will be compensated for by a
slight change in the zeropoint from the expected value. For instance in the
DR4 deep stack there are 17 input intermediate stacks. Extension 4 of
multiframeID=232788 has 3 objects for each real source, and the brightest of
these is ~1 magnitude fainter than expected. Thus the total flux will be 16.4
times a single input rather than 17 times, so each object would be ~0.04 mag
fainter than expected. Instead the zeropoint will be ~0.04 mag brighter than
expected. However, some objects at the edges of the deep stack which do not
lie on multiframeID=232788 because of the dither pattern will be
~0.04 mag brighter.
The deep stacks which are affected are:
UKIDSS-EDR: multiframeID=249019, filename="20060123_v1/e20060123_000953_dp_sf_st.fit". This deep stack contains 9 intermediate stacks.
UKIDSS-DR2: multiframeID=1104105, filename="20070131_v2/e20070131_0010400095_dp_sf_st.fit". This deep stack contains 17 intermediate stacks.
UKIDSS-DR3: multiframeID=1624039, filename="20071108_v3/e20071108_0010400095_dp_sf_st.fit". This deep stack contains 17 intermediate stacks.
UKIDSS-DR4: multiframeID=2012258, filename="20080612_v4/e20080612_0010400095_dp_sf_st.fit". This deep stack contains 17 intermediate stacks.
The intermediate stack will be deprecated in UKIDSS-DR5 and a new deep stack will be created excluding this frame.
Illumination correction bug, 08A - 10A
An error in the application of the illumination corrections has been
spotted, affecting data from 08A until 10A inclusive. Also reprocessed data that
has been ingested between the beginning of 2009 and 2011/03/15 is affected.
All the data has been corrected with the H band illumination correction
instead of the illumination correction of the respective band which might
lead to an error of ~0.04 mag in the worst case but should be much smaller
in general. Please note that the resulting systematic errors only really affect a
small percentage of photometric measurements for brighter objects where
the random errors do not dominate the effect.
This error will be fixed in the final data release.
GPS Photometric Calibration - Fixed in DR8
Improvements to the GPS photometric calibration have been made
and the calibration procedures are now reliable in all parts of the survey. Analysis of the overlap regions between independently
calibrated adjacent arrays shows that ~90% of adjacent fields have calibrations consistent to within 0.03 mag. We find that <0.1% of
adjacent arrays differ by >0.1 mag. This is turn implies a high degree
of uniformity in the calibration of 2MASS in the Galactic plane (since
UKIDSS calibration derives from 2MASS) which has not previously been
documented. Any remaining very rare instances of bad calibration are
more likely to be due to bad data that slipped through quality control
rather than problems in the calibration procedures.
A calibration problem in a small fraction of GPS fields was previously noted on this page, affecting DR7 and all previous
releases. Sources in fields with anomalous zero points were flagged by setting a ppErrbits parameter to 131072 (or higher) in the gpsSource
in DR7. Further investigation showed that there were two calibration
problems. (1) The photometric transformation from the 2MASS system to
the UKIDSS system contains a term to describe the effect of interstellar
extinction on the transformation of the 2MASS calibrator stars (see
Hodgkin et al.(2009, MNRAS 394, 675). This empirically derived term was
not accurate for J band observations in fields with high extinction
throughout, owing to the limited amount of data available in such fields
at the time the paper was written. An improved extinction term has now
been established, following the procedures described in that paper. The
selection of 2MASS calibrator stars has also been modified slightly
to ensure that relatively blue dwarf stars are used whenever
possible. These changes improve the accuracy of the zero points and
source magnitudes in all three passbands. (2) The second problem was a
modest (approx. 0.05 mag) underestimation of the aperture corrections in
very crowded fields, which affected most fields in the GPS southern strip
at longitudes l<15 degrees). The aperture corrections are an additive
term in the computation of the zero points, so this led to an underestimation of the zero points at the same level. Fortunately, the
aperture corrections are subtracted when computing the aperture magnitudes of sources, causing the error to cancel out almost perfectly.
The net effect of this problem was therefore negligible for most users. Only users who were using the erroneous zero points in their own
photometry would have been affected.
To put these changes into perspective, the typical change in source
magnitudes between DR7 and DR8 was <0.01 mag. We find that the calibration
of 99.8% of GPS fields has changed by <0.1 mag, and 98.6% of GPS fields
have changed by <0.05 mag. In the southern fields at longitudes l<15
degrees the problems were a little more serious, leading to typical
absolute changes of 0.016, 0.015 and 0.006 mag in the J, H and K
calibrations respectively. In this southern portion of the survey the
calibration of 1.3% of fields has changed by >0.1 mag, these significant
changes occurring only in the J and H bands.
Default values for the photometric statistics in non-surveys
A software upgrade to numpy led to an error in the calculation of the best aperture
for photometric statistics in the Variability tables. Therefore all the photometric statistics in
Variability and the noise properties in VarFrameSetInfo are all set to default values. This affects all
datasets released from 25th April 2009 until 7th May 2009.
Incomplete Best-Match tables for multi-epoch data sets
An error in matching the Source table to Detecion tables to create the
SourceXDetectionBestMatch (BM) table has been discovered (5th May 2009).
For sources where
there are less than 3 good matches (3 different epochs), there are no entries
in the BM table (except occasionally in dense regions). This was implemented
to avoid calculating variability statistics for sources with too few
detections. However, the side affect of this is to miss important links
between frames, particularly for faint objects near the detection limit of
individual epoch images, or datasets with a small number of epochs.
We have modified the code and the BM table will include links
between the Source and Detection tables for sources with fewer than 3 epochs
but only calculate the astrometric statistics with 3 or more observations.
We will apply this to the DXS in DR6, and the UDS in the DR5.
PI programmes with multi-epoch observations will have this fix if they were
released after 5th May 2009, and won't if they were released before.
They will be fixed in future re-releases of data sets.
Under estimate of Kron and Petrosian fluxes for
The apertures that for Kron and Petrosian magnitudes have an upper limit
of 12 arcsec to avoid incorrect background subtraction for galaxies where the
size is greater than the background estimation. However this will lead to an
underestimate of the flux of large galaxies. The very brightest galaxies,
where background subtraction would not lead to significant errors, do
have larger apertures.
Error in Kron radii
A bug has been identified in the Kron radii produced by the standard
pipeline extractor. Therefore all catalogues except
the UDS catalogues are affected. The exact nature of the bug has
not yet been ascertained, only that the Kron radii are not being reliably
measured. This will affect the Kron magnitudes, which should not be used until
this bug has been fixed. The same tests have shown that the Petrosian radii
are reliable and so for the foreseeable future, Petrosian magnitudes should be
used for total magnitude estimates in extragalactic work.
The cross-talk artefacts produced by bright stars result in spurious souces in the
object catalogues and tables.
see CASU for description and examples.
Releases prior to DR1 all suffer from this problem. Semester 05B data in DR1 (but not 05A data) are
less affected as improvements in the pipeline processing have reduced the effect.
Starting with DR2, during post-processing quality error bit flags
were set that should help with some of the issues e.g. cross-talk
(see quality error bits page).
Can result in spurious souces in the
object catalogues and tables.
see CASU for description and examples; a more recent report is available
Concerning image processing, the CASU pipeline processing pages contain much
information on known issues and image artifacts in WFCAM data - see, for
here. For general help, support and any bug reports concerning CASU
pipeline processing, please email
General archive modification history:
- February 21st 2008: problem with FIRST cross-matches
We discovered a problem with the neighbour table,
lasSourceXfirstSource whereby the pairing had been done incorrectly.
The cross-matching was re-run and the table has been corrected in all
UKIDSS release databases. Prior to this any cross matches queries executed in
the EDR, DR1, DR2 or DR3 with FIRST making use of this table would have been in
- January 7th - 12th 2008: missing pixel data
One of the disks holding pixel data was lost, consequently there will be no
access to images that the disk held. See the
list of affected dates.
The data are being re-transferred from CASU. Note this only affects
pixel access (eg getImage, multiGetImage) and downloads of flat files
from the affected nights. Database queries are unaffected.
- October 31st 2005: USNO-B database corrupted
The USNO-B database held in the archive was found to be corrupted and will
remain unavailable until further notice.
WFAU, Institute for Astronomy,
Royal Observatory, Blackford Hill
Edinburgh, EH9 3HJ, UK
Tel +44 131 668 8356 (office)
or +44 131 668 8100 (switchboard)