If you are me, you spend a couple of weeks deriving stuff, and then finally figure out the easy way.

But hopefully you are not me, so here is the easy way, to save you the bother.

First, find the peak of your function, and put a hat on the value(s) or the parameter(s) at this point. Now your (N-dimensional) function can be approximated as

\(f(x) \simeq f(\hat{x})\exp \left[-\frac{1}{2} (x-\hat{x})^\mathrm{T} \Sigma^{-1} (x-\hat{x}) \right]\)

where \(\Sigma\) is the covariance matrix, and our task it to figure out what it is.

But fear not, help is at hand. If we find the following Hessian matrix, and if we assume that the covariance matrix is symmetric, we have

\(
\begin{align} H_{ij} &\equiv \frac{\partial^2}{\partial x_i \partial x_j} \left( - \ln f(x) \right) \\ &\simeq \frac{\partial^2}{\partial x_i \partial x_j} \left( - \ln f(\hat{x}) + \frac{1}{2} (x-\hat{x})^\mathrm{T} \Sigma^{-1} (x-\hat{x}) \right) \\ &= \frac{1}{2} (\Sigma^{-1}_{ij} + \Sigma^{-1}_{ji}) \\ &= \Sigma^{-1}_{ij} \end{align}
\)

That's it. Easy. Now you can even integrate the function

\(\int f(x) \, \mathrm{d}x = (2\pi)^{N/2} |\Sigma|^{1/2} f(\hat{x}) = (2\pi)^{N/2} |H|^{-1/2} f(\hat{x})\)

Some of the catalogues are available here, if you're interested.

The biggest challenge was confusion (about which there is a great deal of confusion). For example, in this HerMES image, everything you can see is light from galaxies, and each galaxy produces a small round blob in the image. The number of galaxies is so high, that light from any one galaxy is confused with light from numerous neighbouring galaxies.

So when the computer looks at the image and spots a blob with brightness 30 mJy, what does that mean? Does it mean there is one galaxy there with brightness 30 mJy? Or perhaps there are 10 galaxies, with brightnesses of 15 mJy or lower?

It's difficult to tell. But what you can do (and what we did in the paper) is to stick fake galaxies (blobs) into the image, and see what difference that makes. So if we have a fake galaxy with brightness 30 mJy, and we put it in at a random position, what effect will that have, on average? What is the probability that it will be detected? (This is approaching what we mean by the completeness of the catalogue, although it is far from easy to know what is meant by those simple words "it" and "detected"!) What brightness will it be measured to have, on average? 20 mJy? 30 mJy? 40 mJy? 100 mJy?

For very bright galaxies, it's relatively straightforward. But for fainter things, it can do your head in...

First, a small utility:

def getSortedInstituteCodes(authorList, authorInstituteCodes):
    """Return a list of institute codes, in the correct order for authorList.
 
    authorList = ["J.~Bloggs", "A.N.~Other"]
    authorInstituteCodes = {"J.~Bloggs":1, "A.N.~Other":[1,"Sussex"]}
    """
    instituteCodes = []
    for author in authorList:
        codes = authorInstituteCodes[author]
        if not hasattr(codes, '__iter__'):
            codes = [codes]
        for code in codes:
            if not code in instituteCodes:
                instituteCodes.append(code)
    return instituteCodes

then the function for MNRAS:

def getAuthorsStringMNRAS(authorList, authorInstituteCodes, institutesTex,
                          authorEmails):
    """Return a string containing LaTeX for MNRAS authors list.
 
    authorList = ["J.~Bloggs", "A.N.~Other"]
    authorInstituteCodes = {"J.~Bloggs":1, "A.N.~Other":[1,"Sussex"]}
    institutesTex = {1:"Somewhere, Earth", "Sussex":"University of Sussex"}
    authorEmails = {"J.~Bloggs": "j.bloggs@no.where"}
    """
    instituteCodes = getSortedInstituteCodes(authorList, authorInstituteCodes)
 
    # Make a list of strings:
    # "Author Name$^{1,2}$\thanks{E-mail: \texttt{j.bloggs@no.where}}"
    authorInstituteList = []
    for author in authorList:
        try:
            codes = authorInstituteCodes[author]
            if not hasattr(codes, '__iter__'):
                codes = [codes]  # Only one institute code for this author
            authorInstituteList.append(
                author
                + ',$^{' + ','.join([str(instituteCodes.index(code) + 1)
                                     for code in codes])
                + '}$')
        except KeyError:
            authorInstituteList.append(author + ',')
        try:
            email = authorEmails[author]
            authorInstituteList[-1] += '\\thanks{E-mail: \\texttt{'
            authorInstituteList[-1] += email + '}}'
        except KeyError:
            pass # No email address for this author
    # Sort out the punctuation (won't work if the penultimate or ultimate
    # author has no institute)
    authorInstituteList[-2] = authorInstituteList[-2].replace(',$', '$')
    authorInstituteList[-2] += ' and'
    authorInstituteList[-1] = authorInstituteList[-1].replace(',$', '$')
 
    # Put it all together
    authorsString = '\\author[' + authorList[0] + ' et al.]'
    authorsString += '\n{\\parbox{\\textwidth}{'
    authorsString += '\n'.join(authorInstituteList)
    authorsString += '}\\vspace{0.4cm}\\\\\n'
    authorsString += '\\parbox{\\textwidth}{'
    authorsString += '\\\\\n'.join(['$^{' + str(iInst + 1) + '}$'
                                    + institutesTex[instituteCodes[iInst]]
                                    for iInst in range(len(instituteCodes))])
    authorsString += '}}\n'
 
    return authorsString

and the function for A&A:

def getAuthorsStringAA(authorList, authorInstituteCodes, institutesTex,
                       authorEmails):
    """Return a string containing LaTeX for A&A authors list.
 
    authorList = ["J.~Bloggs", "A.N.~Other"]
    authorInstituteCodes = {"J.~Bloggs":1, "A.N.~Other":[1,"Sussex"]}
    institutesTex = {1:"Somewhere, Earth", "Sussex":"University of Sussex"}
    authorEmails = {"J.~Bloggs": "j.bloggs@no.where"}
    """
    instituteCodes = getSortedInstituteCodes(authorList, authorInstituteCodes)
 
    # Associate institute numbers with authors and emails with institutes
    authorInstituteList = []  # A list of "Author Name\inst{1,2}" strings
    instituteEmails = {}  # A dictionary of email addresses for each institute
    for author in authorList:
        try:
            codes = authorInstituteCodes[author]
            if not hasattr(codes, '__iter__'):
                codes = [codes]  # Only one institute code for this author
            authorInstituteList.append(
                author + '\inst{'
                + ','.join([str(instituteCodes.index(code) + 1)
                            for code in codes])
                + '}')
            try:
                email = authorEmails[author]
                try:
                    instituteEmails[codes[0]].append(email)
                except KeyError:  # First email address for this institute
                    instituteEmails[codes[0]] = [email]
            except KeyError:
                pass # No email address associated with author
        except KeyError:
            authorInstituteList.append(author)
 
    # Create a list of "Institute Name, Address\\
    #                   \email{j.bloggs@no.where}" strings
    instituteEmailList = []
    for instituteCode in instituteCodes:
        try:
            emails = instituteEmails[instituteCode]
            instituteEmailList.append(
                institutesTex[instituteCode]
                + ''.join(['\\\\\n\email{' + email + '}' for email in emails]))
        except KeyError:
            instituteEmailList.append(institutesTex[instituteCode])
 
    # Put it all together
    authorsString = '\\author{'
    authorsString += '\n\\and '.join(authorInstituteList)
    authorsString += '}\n'
    authorsString += '\\institute{'
    authorsString += '\n\\and '.join(instituteEmailList)
    authorsString += '}\n'
 
    return authorsString

Anyone still reading?

\author[J.~Bloggs et al.]
{\parbox{\textwidth}{J.~Bloggs,$^{1}$\thanks{E-mail: \texttt{j.bloggs@no.where}}
J.~Bloggs,$^{2,3}$
J.~Bloggs,$^{3}$
J.~Bloggs,$^{4}$
J.~Bloggs,$^{5}$
J.~Bloggs,$^{6}$
%...
J.~Bloggs$^{99}$ and
J.~Bloggs$^{100}$}\vspace{0.4cm}\\
\parbox{\textwidth}{$^{1}$Department of Something, Somewhere\\
$^{2}$Department of Something, Somewhere\\
$^{3}$Department of Something, Somewhere\\
$^{4}$Department of Something, Somewhere\\
$^{5}$Department of Something, Somewhere\\
$^{6}$Department of Something, Somewhere\\
%...
$^{99}$Department of Something, Somewhere\\
$^{100}$Department of Something, Somewhere}}

Next: a Python script to generate the above from a simple list of authors and affiliations.

\[P(\lambda|k) = \frac{P(k|\lambda)P(\lambda)}{P(k)}\]

Now, for the prior, \(P(\lambda)\), we assume a prior which is flat on a logarithmic scale. That is, we guess (before making the observation) that the expected number is as likely to lie between 1 and 10 as it is to lie between 1000 and 10,000. (The alternative, a flat prior on a linear scale, would mean that we guess the true density is just as likely to lie between 10,001 and 10,010 as it is to lie between 1 and 10, which is ridiculous.) So \(P(\lambda) \propto 1/\lambda\). The likelihood, \(P(k|\lambda)\) is given by the Poisson distribution. So, ignoring the normalizing factor of \(P(k)\),

\[P(\lambda|k) \propto \frac{\lambda^k}{k!} e^{-\lambda} \frac{1}{\lambda}\]

\[ \propto \lambda^{k-1}e^{-\lambda}\]

And this is the Gamma distribution. Easy peasy.

• Herschel Science Archive
• Herschel Data Processing
• HIPE software: stable release, or development builds (installation instructions)

Research in astronomy includes the analysis of astronomical images, parsing and manipulation of large catalogs, statistical yet often visual inference, and the creation of data visualizations for publication and dissemination of results.

The purpose of this web site is to act as a community knowledge base for performing this research with the open source Python language. It provides a forum for general discussion, advice, or relevant news items, collecting lists of useful resources, users' code snippets or scripts, and longer tutorials on specific topics. The topics within these pages are presented in a list view with the ability to sort by date or topic. A traditional "blog" view of the most recently posted topics is visible from the site Home page.

Data volumes from multiple sky surveys have grown from gigabytes into terabytes during the past decade, and will grow from terabytes into tens (or hundreds) of petabytes in the next decade. ... For astronomy to effectively cope with and reap the maximum scientific return from existing and future large sky surveys, facilities, and data-producing projects, we need our own information science specialists. We therefore recommend the formal creation, recognition, and support of a major new discipline, which we call Astroinformatics. ... Now is the time for the recognition of Astroinformatics as an essential methodology of astronomical research. The future of astronomy depends on it.

Astroinformatics: A 21st Century Approach to Astronomy

Here's a Python function to accompany the previous post. It's not maximally efficient, but should make sense...

from scipy import stats
def gaussian_pixel(minxy, maxxy, sigma, meanxy=(0.,0.), norm=None):
    """Return the value of a pixel sampling a 2D Gaussian,
    normalized such that the area under the Gaussian is 1
    (default) or such that the peak is given by norm."""
    x1, y1 = minxy
    x2, y2 = maxxy
    x0, y0 = meanxy
    if norm is None:
        norm = 1. / 2 / math.pi / sigma ** 2
    return norm * 2 * math.pi * sigma ** 2 / (x2 - x1) / (y2 - y1) * (
        (1 - stats.erfc((x2 - x0) / math.sqrt(2) / sigma)) / 2.
        - (1 - stats.erfc((x1 - x0) / math.sqrt(2) / sigma)) / 2.) * (
        (1 - stats.erfc((y2 - y0) / math.sqrt(2) / sigma)) / 2.
        - (1 - stats.erfc((y1 - y0) / math.sqrt(2) / sigma)) / 2.)

If the pixels are larger than infinitesimal, as is generally the case, then the maximum value of the pixelated PRF will be the average value over the pixel, which will be less than 1.

For example, if the PRF is a two-dimensional Gaussian, centred on \((x_0, y_0)\), with standard deviation \(\sigma\), then the value in a pixel with \(x_1 < x < x_2\) and \(y_1 < y < y_2\) will be

\[A = \frac{1}{(x_2 - x_1)(y_2 - y_1)} \int_{x_1}^{x_2} \int_{y_1}^{y_2} e^{- \left( \frac{(x-x_0)^2 + (y-y_0)^2}{2\sigma^2} \right)} \mathrm{d}x \mathrm{d}y.\]

\[A = \frac{2 \pi \sigma^2}{(x_2 - x_1)(y_2 - y_1)} \left( \frac{1}{2} \mathrm{erf} \left( \frac{x_2 - x_0}{\sqrt{2}\sigma} \right) - \frac{1}{2} \mathrm{erf} \left( \frac{x_1 - x_0}{\sqrt{2}\sigma} \right) \right)\]

\[ \times \left( \frac{1}{2} \mathrm{erf} \left( \frac{y_2 - y_0}{\sqrt{2}\sigma} \right) - \frac{1}{2} \mathrm{erf} \left( \frac{y_1 - y_0}{\sqrt{2}\sigma} \right) \right).\]

Ugh. Let's make that simpler. For a PRF centred on \((0,0)\), and a pixel \((\pm r, \pm r)\), this is

\[A = \frac{\pi \sigma^2}{2 r^2} \left( \mathrm{erf} \left( \frac{r}{\sqrt{2}\sigma} \right) \right)^2.\]

As an example, the fairly-Gaussian beam for the Herschel Space Observatory SPIRE instrument has an FWHM of around 18", which corresponds to a standard deviation of around \(18"/2\sqrt{2 \ln 2} = 7.64"\). If we make a Jy/beam map with pixel size 6", then the peak value for a 1 Jy point source in the centre of a pixel will be

\[A = \frac{\pi (7.64")^2}{2 (3")^2} \left( \mathrm{erf} \left( \frac{3"}{\sqrt{2}(7.64")} \right) \right)^2 = 0.950 \,\text{Jy/beam}.\]

Easy.

First, find \(P_i\), which is the point response function (PRF), telling you what a point source of flux 1 will look like in the map. This may be normalized so that the peak is 1 (if your map is in Jy/beam or similar), or so that \(\sum P_i = 1\) (if your map is in Jy/pixel or similar). If your map is in MJy/sr ... well, figure it out and add a comment below. Basically, if you normalize your PRF correctly, you won't need to worry about the map units in what follows. Phew.

Now the measured value of each pixel around the point source, \(d_i\), will be

\[d_i = f P_i + n_i,\]

Now the badness of the fit is measured by the \(\chi^2\), which is given by

\[\chi^2 = \sum_i \left( \frac{d_i - fP_i}{\sigma_i} \right)^2.\]

\[\frac{\mathrm{d}\chi^2}{\mathrm{d}f} = 0.\]

\[\sum_i (-2d_i + 2 f P_i^2) / \sigma_i^2 = 0.\]

\[f = \frac{\sum_i d_i P_i / \sigma_i^2}{\sum_i P_i^2 / \sigma_i^2}.\]

Now just do this for each pixel in the map (corresponding to a point source centred on each pixel) and you're done.

Worked example. \(P_i\) is 0.5, 1.0 and 0.5, for three adjacent pixels (you'll have realised that the map is in Jy/beam or similar), and \(d_i\) is 1, 2 and 1 Jy/beam, for three adjacent pixels, with the same (tiny!) value of \(\sigma_i\) for each pixel (in this case, we can ignore the value of \(\sigma_i\) in what follows). So the flux at the central pixel is estimated to be

\[f = (0.5 \times 1 + 1.0 \times 2 + 0.5 \times 1) / (0.5^2 + 1.0^2 + 0.5^2) = 2 \,\mathrm{Jy},\]

This is an example of a matched filter (I haven't read the page, but hopefully including the link will make me look clever). And, given that point sources are under no particular obligation to align themselves with the centres of the pixels of your map, \(P_i\) can easily be re-estimated for a source with a certain offset from the pixel centre.

Here's the Python code

import numpy as np
from matplotlib import pyplot as plt
 
class FlickerImage(object):
    def __init__(self, im, err):
        self.im = im.copy()
        self.err = err.copy()
        finite = np.isfinite(self.im + self.err)
        self.vmin = (self.im - 2 * self.err)[finite].min()
        self.vmax = (self.im + 2 * self.err)[finite].max()
        self.im[np.invert(finite)] = self.vmax
        self.err[np.invert(finite)] = 0
    def flicker(self):
        fg = plt.imshow(np.zeros(self.im.shape),
                        interpolation='nearest',
                        vmin=self.vmin,
                        vmax=self.vmax)
        while True:
            ran = np.random.normal(size=im.shape)
            fg.set_data(im + err * ran)
            plt.draw()

And here's an example script:

import pyfits
f = pyfits.open('file.fits')
im = f["IMAGE"].data
err = f["ERROR"].data
flicker_image = FlickerImage(im, err)
flicker_image.flicker()

import numdisplay
import pyfits
arr = pyfits.getdata('file.fits')
numdisplay.display(arr)

Easy!

Alternatively, the Kapteyn package seems excellent, and uses Python's matplotlib for displaying images. It requires WCSLIB to run, though, so the installation process is a bit longer [update: apparently that bit is no longer true - 17 Oct 2011].

A third option is to use python-sao:

import pysao
import pyfits
ds9 = pysao.ds9()
f = pyfits.open('file.fits')
ds9.view(f[0])

Easy again! And the WCS information is preserved, which doesn't seem to be the case with Numdisplay.

1. StarFinder seems to do a great job at finding point sources in crowded fields. It includes a routine for generating the Airy pattern. For a 51 x 51 array, with the peak at [25, 25], and an FWHM of 8.0 pixels, this is the command:

psf = airy_pattern(51, 51, 25, 25, 2./8.0)
isurface, psf

The output looks something like this:

2. The IDL Astronomy User's Library contains a routine, psf_gaussian, that produces a Gaussian PSF. To produce the same as above, the command would be:

psf = psf_gaussian(npix=51, fwhm=8.0, /double)
isurface, psf

... which produces something like this:

• IDL Workbench rocks! (This is because it is basically Eclipse, which is a proper development environment, unlike that hideous old IDLDE.)
• Another reason for using IDL Workbench (for me at least, and for now) is that IDL help doesn't seem to work if Java 6 is the default (as it is on my Mac), but the help does work if launched through the IDL Workbench.

To transition to IDL Workbench:

1. Import your code as described on David Fanning's page - fret not, it's easy and harmless
2. Preferences -> IDL -> Startup file, if you have one, and
3. Preferences -> IDL -> Paths -> Insert... for me it was just my idl folder, including all sub-folders, to mimic my $IDL_PATH environment variable.

The appropriate scientific methodology with which to address such questions is itself problematic: how does one apply what many consider the “traditional scientific method”, involving objective analysis of independent repeated experiments as a test of theory, when the Universe does not allow us to experiment, in the traditional laboratory physics sense; when we have no useful predictive theory for much of astrophysics; and when the nature of the Universe may restrict our observation to only a very small part of an unobservable larger whole? More specifically, is the observational test of prediction how science actually operates? Is that how astrophysics operates?

Good stuff. But the most cutting remarks come in his assessment of the current approach to modelling galaxy formation:

Such a long list of observations all inconsistent with apparently fundamental features of galaxy formation models suggests two approaches. In one approach, new complex physics (“feedback”) must be added, to “improve” agreement with observation. The appearances are to be saved. In another, common assumptions in the galaxy simulations could be examined further.

With the reference to the saving of appearances, the allusion is to Ptolemy's epicycles: making a misguided model seem more plausible by making it more contrived.

The specific problem Gilmore sees with cosmological simulations is the suppression of the "ultraviolet divergence", i.e., small-scale perturbations, by "numerical smoothing (‘finite resolution')": "It is unlikely that Nature does it that way." He suggests that many of the inconsistencies between galaxy formation models and observations could be a result of this poor handling of the small-scale power spectrum.

(*) Disclaimer: I will not be held responsible for any damage sustained to your eyes as a result of following links on this page.

• keep track of PDFs of academic papers,
• search for papers using Google Scholar, ADS, arXiv, ...,
• search your personal library in an instant,
• read papers full-screen,
• add notes to papers,
• organise the papers using collections and smart collections,
• interact with BibTeX databases and citation keys,
• and do all the above in something that looks and feels like iTunes?

Here it is: Papers by mekentosj.com.

Actually, no.

At least, not according to this Galaxy Zoo paper, on the independence of morphology and colour (or here).

First of all, there's a sizeable population of galaxies that blatantly refuse to allow their colour to determine what shape they should be. There are red galaxies with beautiful spiral morphology and blue galaxies with plain old elliptical morphology.

Okay, but we know that red galaxies like to hang out in crowded places, and that elliptical galaxies are similarly gregarious, so clearly there's some connection between being red and being well-rounded?

Nope, wrong again!

The main reason that we see more red galaxies in dense environments is that the fraction of spiral galaxies that are red changes, and the fraction of elliptical galaxies that are blue changes. So in sparsely populated bits of the universe, most of the spiral galaxies are blue, but in densely populated regions, most of the spiral galaxies are red. It's similar for elliptical galaxies. In low-density regions, a large fraction (not quite half) of the elliptical galaxies are blue, whereas in dense environments the vast majority of elliptical galaxies are red.

So the morphology-density relation has really very little (directly) to do with the colour-density relation.

Moral: "elliptical/spiral" doesn't mean "red/blue"!

This is the K-band luminosity function: the number of galaxies per volume as a function of their luminosity, with low luminosity at the left and high luminosity at the right. It's far from perfect, but hopefully a step in the right direction. There's quite a bit of incompleteness (missing galaxies) and uncertainty (due to small numbers of galaxies and large-scale structure) at the faint end (left-hand side of the plot). But perhaps more interesting is the disagreement at the bright end (right-hand side). All of the previous results shown on the plot used 2MASS imaging, so this might explain the different results we have found. Specifically, it could be that (1) we use Petrosian magnitudes rather than Kron or total magnitudes, (2) UKIDSS photometry is better than 2MASS photometry, (3) the evolution corrections are different, (4) something else or (5) any combination of the above.

This is some work in progress: K-band luminosity function from the UKIDSS Large Area Survey (LAS, black dots), showing the number of galaxies per unit volume depending on the luminosity of the galaxies, from faint (left) to bright (right). I.e., there are lots more small galaxies than big galaxies.

I've fit several Schechter functions to the data. This is a convenient way of describing the luminosity function in terms of three numbers: the slope of the faint end (alpha), the luminosity brighter than which the number of galaxies drops off rapidly (M-star) and the number of galaxies per unit volume at M-star (phi-star). To fit the Schechter functions I've used only a portion of the data, as shown in the figure. For example, for the green curve, I've used only the black points brighter than (to the right of) absolute magnitude -21.

Now here's the point. At high redshift, it is possible to see only the brightest galaxies. So we would be able to plot only the black points towards the right-hand side of the figure. But what effect would this have on the Schechter function? Even if we assume the luminosity function does not vary with redshift, our Schechter function fits would! In fact, if we relied on the Schechter function fit to tell us how the galaxy population varied with redshift (a silly thing to do, but people do it all the time), we would infer that the high-redshift galaxy population was (1) brighter (2) more dominated by small galaxies and (3) less abundant than the low-redshift galaxy population.

(Now (1) and (3) are probably true, but we don't need the Schechter function to tell us. Not so sure about (2).)

Moral: don't rely on the Schechter function!