Plotting for honesty

Mean pregnancy length for first babies is 38.601 weeks, and for other babies it is 38.523 [S:41:47]; the difference is 0.078 weeks, which works out to 13 hours, or about 0.2 per cent of a pregnancy [S:41:47]. The standard deviation σ of the same measurements is 2.7 weeks, so ordinary variation is more than thirty times the difference between the two groups [S:41:47].

Two bar charts can be drawn from those same numbers, one making the difference resemble a canyon and the other making it resemble nothing at all, and neither chart contains a false digit. This section asks which graph tells the truth, and the answer is that a graph is honest when the picture and the caption together let a reader recover the real ratio. !The same two means, 38.601 weeks for first babies and 38.523 for others, drawn twice with the same bar width. On the left the axis runs from 0 to 40 weeks, where the 0.078-week gap fills 0.2 per cent of the height. On the right the axis runs from 38.5 to 38.7, where the same gap fills 39 per cent, a magnification of 200. Neither chart contains a false digit. The right one is honest only when the caption prints where its axis starts and the magnification, and the spread is drawn beside the difference; because its mark is a bar, the full repair is to change the mark to a dot or a line. Status: published.

The idea. A graph is honest when the picture and the caption together let a reader recover the real ratio, and that requirement produces three rules you can check before you publish a figure. !The gap and the variation on one scale. The two means differ by 0.078 weeks, about 13 hours or 0.029 standard deviations, while one standard deviation of the measurements is 2.7 weeks. The variation is about thirty-five times the gap, so two bars drawn without the band would present a difference smaller than the spread that produced it. Status: schematic.

Plot every run. A curve of means conceals the spread that produced it, so display the individual runs or shade the region they occupy, as SAC does when it places the minimum and the maximum over five trials around the mean curve [S:291:7].

The pregnancy numbers at the top of this section are the case to try it on, so draw the two means as dots rather than bars and put the spread beside them. The gap is 0.078 weeks and one standard deviation is 2.7 weeks, so a picture that shows only the two means hides the variation that makes the gap small.

Give the comparison the same axes and the same budget. An ablation at unequal cost is not a comparison, and a baseline tuned more than the method is not a baseline either, so the budget belongs in the caption as well as in the code [S:288:5].

Write a caption that restores the scale. The caption names the quantity measured, the set it was measured on, the number of runs, the seeds and any unusual axis limit. If the axis does not start at zero, it prints where it starts and the magnification, and it shows the spread beside the difference. For a bar, the caption is not enough: the mark itself has to become a dot or a line, because a bar measures its value as length from zero.

The magnification factor. An axis that spans a narrow interval stretches every difference drawn on it, and the stretch factor is the ratio of the full range to the plotted range:

$$\text{magnification}=\frac{40}{0.2}=200$$

A chart whose axis covers 0.2 weeks, when the full scale is 40 weeks, renders every difference 200 times taller than a chart of the whole range would.

Worked example. Take the two charts of the pregnancy means: the first runs from 0 to 40 weeks, so the 0.078-week gap occupies 0.078/40, which is 0.2 per cent of the height. The second runs from 38.5 to 38.7 weeks, so the same gap occupies 0.078/0.2, which is 39 per cent of the height. The second chart is 200 times the first, and it is not a lie provided that the caption says where the axis starts and what the magnification is, and that the spread is drawn beside the difference. A bar measures its value from zero, so the mark also has to become a dot or a line; with the caption and the mark in place the reader can recover the ratio.

Count the comparisons. A figure that displays six ablations contains six comparisons, and the tallest bar is the one most likely to be an accident. The informal principle from Mining of Massive Datasets applies to a wall of plots as much as to a database query. Estimate how many results of that kind random data would produce, and distrust the display when that count swamps the real ones [S:57:26].

Say the size in units a reader can feel. The source reports the pregnancy gap as 13 hours and as 0.029 standard deviations, which it calls small [S:41:48], and a figure can carry both facts at once, with the difference drawn to scale and the variation drawn beside it.

A bar chart of two means with no spread beside them is dishonest when the gap is smaller than the run-to-run variation [S:41:47], and the pregnancy gap of 0.078 beside a standard deviation of 2.7 is that case. It allows the reader to assume that the two bars are the whole story.

A truncated axis is a decision about emphasis, and emphasis is the part that can mislead a reader who trusts the picture.

The limitation is that no plotting rule can rescue a comparison that was never fair, and a beautiful figure of a leaking experiment is a beautiful figure of nothing. The axis arithmetic cannot detect the leak, and plotting is the last step of the chapter because it displays a result that the earlier steps have to earn.

The formal statement. Take a linear vertical axis running from a lower limit to an upper limit. A difference occupies the fraction of the height equal to the difference divided by the span of the axis. Two charts of the same quantity differ in stretch by

$$\text{magnification}=\frac{\text{full range}}{\text{plotted range}}$$

Read it as: the span you would have drawn divided by the span you did draw. Every difference on the narrow chart is that many times taller, provided the axis is linear and both charts use the same units.

The derivation. A linear axis maps a value to a position by an affine function, so the rendered height of a difference is proportional to the difference divided by the span. Divide the narrow fraction by the wide one and the difference cancels. That cancellation is why the magnification belongs to the axis alone and not to the difference on display. It is also why the factor can be quoted once, in the caption, and apply to every pair of bars in the figure.

The edge case. A logarithmic axis breaks the cancellation. On a log axis equal ratios occupy equal heights, so a fixed absolute gap shrinks as the values grow. A caption that quotes a magnification for a log axis reports a number the picture does not use, and the factor that is correct for a linear pair is wrong there. State the axis type beside the limits, and not the factor alone.

The second edge case. The size of a visual difference depends on the encoding. A bar encodes its value as length from zero, so an axis that omits zero breaks the encoding of the value itself rather than only the gap. A dot or a line encodes position, so a labelled truncation leaves the difference readable. The corpus's support is the general rule against intentional scale distortion [S:116:439]; the encoding distinction needs a visualization source, not the effect-size page. The truncated bar chart is the worse offence for this reason. Labelling it is necessary and, for a bar, not sufficient; the full repair is to change the mark to a dot or a line.

What L1 hid, and why it was safe. L1 asks the visual difference to match the real difference. A difference is only one quantity worth matching, and the interval is the other. Llama 3's benchmark tables report 95 per cent confidence intervals beside the scores [S:431:30], which is the demand of 4.1 moved from the figure into the table. L1 was safe because the section was about the axis, and the interval belongs to the first section.

Where the analogy breaks. A map's scale bar states a distortion the map cannot avoid, so the warning is part of the instrument. An axis carries no such obligation. The label is optional, and a picture without it looks complete, which is the difference between a distortion you are told about and one you are not.

Original sources. The pregnancy example and the effect-size arithmetic are from Think Stats. The means are 38.601 and 38.523 weeks, the difference is 13 hours and about 0.2 per cent, and the standard deviation is 2.7 weeks [S:41:47]. The same difference is 0.029 standard deviations [S:41:48]. The book's own instruction is to make reports honest and to acknowledge uncertainty and limitations [S:41:48].

An open question. Whether a figure should show a distribution or a summary is not settled by any rule the field enforces. The corpus shows both conventions in the same literature. SAC shades the region between the minimum and the maximum over five trials [S:291:7], while a table of benchmark scores may carry confidence intervals and no figure at all [S:431:30]. The open question is which summary a reader can act on when the comparison is adaptive, that is, when the figures themselves were selected after seeing the numbers. Evidence that would settle it: a reader study in which one result is presented as a point, as an interval and as a full distribution, with the decisions readers make recorded. The corpus does not contain one, so the choice of summary is settled by convention rather than by evidence.

A frontier result. A minimum-to-maximum band is one convention for showing variation; SAC draws one over five trials [S:291:7]. The corpus shows the convention, not how prevalent it is. The band is honest about the range and silent about the shape. A band drawn from five runs says nothing about the middle of the distribution, and two different procedures can produce the same band.

What to read next. Read [S:41:47] and [S:41:48] for the arithmetic of effect size, because the same three numbers support both the canyon and the flat line. Then read [S:291:7] for the band and [S:431:30] for the interval in a table, and compare what each one hides.