Star Chart Verification: An Independent Review
The goal of this website is to investigate underlying beauty and order. To check our accuracy, we engaged an astronomer to review our work. He was asked to validate our methodology, then to take a set of representative chart outputs and compare them against independent references: NASA JPL Horizons and Stellarium Web.
Dr. Juan Manuel Salerno of the Instituto de Astronomía Teórica y Experimental reviewed the star, planet, and Moon positions and the Moon’s phase and illuminated fraction. His signed statement is below, in full and unedited, along with his verification reports. He signed as an individual astronomer. The institute is named to identify him, not to endorse us.
What he reviewed
Along with our declared methodology and representative outputs, he was given a written account of how the engine works, including what it deliberately leaves out, and an extensive set of computed positions to check.
Dr. Salerno’s signed methodology review statement. The full document is linked at the foot of this page.
What we tested before we sent it out
Our engine computes positions from the HYG 4.2 star catalog and Astronomy Engine 2.1.19, accounting for proper motion, precession, nutation, and atmospheric refraction, then projects them with a zenithal stereographic projection.
Before we sent anything out, we ran our own verification: 28 star cases and 15 planet and Moon cases, grouped so that each group tests a specific way a star chart can go wrong. Many of the cases test the edges. In the southern hemisphere, an engine that fails to flip the sky draws Sydney’s view upside down. We tested latitudes above 60 degrees north and below 60 degrees south, out to Alert, Canada, at 82.5 degrees and McMurdo Station at 77.8 degrees south. At the equator, both celestial poles sit on the horizon at once, and a careless calculation divides by zero. We also tested the date line, midnight at the exact epoch boundary, and a catalog index check for a phantom star at coordinates zero, zero.
Dates run from 2 BCE through to the present, including Tycho’s supernova night in 1572, the evening Pluto was discovered in 1930, and the 2003 Mars opposition.
Some of those cases are checks on the drawing, not the numbers. A planet below the horizon has to be absent from the rendered chart, not merely negative in a table. The Moon’s lit limb has to face west on a waxing crescent and east on a waning one. Seen from Sydney, the whole Moon has to turn over.
Every case passed but one. Case 24 puts Vega over Bethlehem in 2 BCE, and the precession model we use is not validated that far back. We marked it DOCUMENT instead of PASS in our own grid, before any outside review, because we could not verify it and did not want to record it as verified.
Why we asked anyway
Our verification compared our engine’s output against PyEphem. Both libraries implement the same underlying planetary theory, VSOP87, which is why the two agreed to four decimal places on most cases. That agreement shows the implementation is sound, but a check between two descendants of one theory cannot tell you whether the theory was applied wrong for your particular observer.
So we sent the whole set to Dr. Salerno. He rebuilt the calculation independently in R and recomputed every case. Then, where the tools allowed, he checked both results by hand against NASA JPL Horizons and Stellarium Web. He could not confirm five of the star cases we had recorded as passing.
How the star cases came out
We ran 28 star cases. One of them, Polaris over Quito, had been failing on our own grid, and the outside check traced it to an elevation handling error in our verification code, not to any disagreement about the sky. It was repaired and it passes now, but because the case was rewritten during the review, we have withdrawn it from the count. That leaves 27 cases evaluated. Twenty-two came in inside the 0.1 degree tolerance, and five did not.
In the 22 that passed, the average angular separation between our positions and Dr. Salerno’s was about 0.000954 degrees, roughly 3.4 arcseconds. The worst passing case was about 20 arcseconds. For scale, the full Moon is about 1,800 arcseconds across, so the typical disagreement is a small fraction of the width of the Moon as you see it.
The five that did not confirm
Cases 8, 21, 24, 25, and 26 did not confirm. Their average angular separation was about 0.423 degrees, and the largest was 0.584 degrees. The full case notes are in the manual results report below.
In three of them, case 8, Polaris over Sydney, case 21, Canopus over Singapore, and case 25, Canopus over Samoa, the hand check in Stellarium Web landed closer to Dr. Salerno’s recalculation than to our value. His report concludes that the discrepancy is probably in our expected value, not in his calculation. We are treating our altitude as the error.
Case 24 is the Bethlehem case we had already set aside. Stellarium Web would not accept the negative year either, so it is now unverified from both directions and stays that way until we find a tool that handles BCE dates properly.
Case 26, Canopus over Tonga, is not a clean test. The coordinates entered into Stellarium Web did not match the coordinates in our case, because Stellarium Web accepts city names, not exact coordinates. A shifted observer moves a low star’s altitude on its own, so that case stays open.
What that means for a chart you order
The largest of the five, 0.584 degrees, is a little over the width of the full Moon as you see it in the sky. On a printed chart, the full Moon at this scale is .5 millimeters across (depending on chart size selected), and a star is a dot considerably smaller than that. Half a degree of altitude error moves a star by roughly the width of the Moon and no further, and it moves it only in the vertical, so no constellation changes shape. Three of the five are also concentrated in one situation, a bright star very low over a southern hemisphere horizon, which is exactly where refraction is hardest to model and where the smallest error in observer elevation shows up most.
What the planet and Moon cases show
The fifteen planet and Moon cases cover the Moon, Mercury, Venus, Mars, Jupiter, and Saturn. Three things decide whether a chart like this is right: where an object sits, how much of the Moon is lit, and what phase it is showing. A wrong Moon is the one error a viewer will catch without instruments.
Variable Largest disagreement Tolerance Result
Azimuth 0.0054° 0.5° Pass
Altitude 0.4191° 0.5° Pass
Illuminated fraction 0.0458% 1% Pass
Phase angle 0.1344° 1° Pass
Apparent magnitude (not a position, see below) 0.42 mag 0.1 mag Fail, 9 of 15 cases
Worst case across all fifteen planet and Moon cases, compared against the tolerance for each variable. The first four are what place an object on the chart and draw the Moon’s phase. The fifth is brightness, which is discussed below.
Dr. Salerno’s own recalculation sat extremely close to Horizons, averaging about 0.00078 degrees in azimuth and 0.00056 degrees in altitude. Our values averaged about 0.086 degrees from Horizons in altitude. That is larger than his by a wide factor and still well inside the tolerance for the product.
Apparent magnitude, and why it sits outside the review
Nine of the fifteen planet and Moon cases are marked FAIL. Every one failed on apparent magnitude alone, with position, illumination, and phase inside tolerance.
Apparent magnitude is not a position. It is a photometric estimate, and it depends on how a given library models the way a planet’s disc reflects sunlight at a given phase angle. Saturn is especially sensitive because of the rings, and Mercury and Venus swing with phase angle. Two tools can put a planet in exactly the same place and still disagree about how bright it is.
We did not ask Dr. Salerno to focus on magnitude, because it does not change any positions, and his analysis sets it aside for the same reason. Our renderer sizes a planet’s drawn disc from computed magnitude, so even the largest disagreement, 0.42 magnitude, changes only the diameter of one dot on a chart carrying hundreds of them. The conclusion the review supports is about position, illumination, and phase, which is what a printed chart is judged on.
What the engine does not do
The declared methodology that was reviewed includes what we leave out.
We do not draw the Sun. We do not apply aberration to star positions. We do not model daylight or atmospheric extinction, so the chart shows the sky as though it were dark and clear, whatever the real conditions were. We draw constellation lines for 39 constellations, not all 88. This was a deliberate choice, and the reasoning is on our star chart calculation page.
The documents
All four are reproduced in full. Nothing edited, summarized, or excerpted. The figures Dr. Salerno produced, comparing results against Horizons and Stellarium Web case by case, are inside them.
Endorsement Astronomical methodology review statement. Dr. Salerno’s signed statement on the methodology and the verification outputs.
Star verification Star verification manual results report. All 28 star cases checked by hand in Stellarium Web, case by case, including the five that did not confirm and the reasoning on each.
Astronomical analysis Dr. Salerno’s own analysis, including his reasoning on numerical difference against visual impact, and his general conclusion.
Planet and moon verification The fifteen-case planet and Moon set, with calculated values, differences, tolerances, and pass or fail status for each. The nine cases marked FAIL are the magnitude cases described above.