Formulas
Program committee members and administrators can search and display formulas that calculate properties of paper scores—for instance, the standard deviation of papers’ Overall merit scores, or average Overall merit among reviewers with high Reviewer expertise.
To display a formula, use a search term such as “show:var(OveMer)” (show the variance in Overall merit scores, along with statistics for all papers). You can also graph formulas. To search for a formula, use a search term such as “formula:var(OveMer)>0.5” (select papers with variance in Overall merit greater than 0.5). Or save formulas using Search > View options > Edit formulas.
Formulas use a familiar expression language. For example, this computes the sum of the squares of the overall merit scores:
sum(OveMer**2)
This calculates an average of overall merit scores, weighted by expertise (high-expertise reviews are given slightly more weight):
wavg(OveMer, RevExp >= 4 ? 1 : 0.8)
And there are many variations. This version gives more weight to PC reviewers with the “#heavy” tag:
wavg(OveMer, re:#heavy + 1)
(“re:#heavy + 1” equals 2 for #heavy reviews and 1 for others.)
Formulas work better for numeric scores, but you can use them for letter scores too. HotCRP uses alphabetical order for letter scores, so the “min” of scores A, B, and D is A. For instance:
count(confidence=X)
Expressions
Formula expressions are built from the following parts:
Arithmetic | |
| 2 | Numbers |
| true, false | Booleans |
| null | The null value |
| (e) | Parentheses |
| e + e, e - e | Addition, subtraction |
| e * e, e / e, e % e | Multiplication, division, remainder |
| e ** e | Exponentiation |
| e == e, e != e, e < e, e > e, e <= e, e >= e | Comparisons |
| !e | Logical not |
| e1 && e2 | Logical and (returns e1 if e1 is false, otherwise returns e2) |
| e1 || e2 | Logical or (returns e1 if e1 is true, otherwise returns e2) |
| test ? iftrue : iffalse | If-then-else operator |
| let var = val in e | Local variable definition |
| greatest(e, e, ...) | Maximum |
| least(e, e, ...) | Minimum |
| coalesce(e, e, ...) | Null coalescing: return first of es that is not null |
| log(e) | Natural logarithm |
| log(e, b) | Log to the base b |
| round(e[, m]) | Round to the nearest multiple of m |
Submission properties | |
| pid | Paper ID |
| au | Number of authors |
| au:pc | Number of PC authors |
| au:text | Number of authors matching text |
Tags | |
| #tagname | True if this paper has tag tagname |
| tagval:tagname | The value of tag tagname, or null if this paper doesn’t have that tag |
Scores | |
| overall-merit | This review’s Overall merit score Only completed reviews are considered. |
| OveMer | Abbreviations also accepted |
| OveMer:external | Overall merit for external reviews, null for other reviews |
| OveMer:R2 | Overall merit for round R2 reviews, null for other reviews |
Submitted reviews | |
| re:type | Review type |
| re:round | Review round |
| re:auwords | Review word count (author-visible fields only) |
| re:primary | True for primary reviews |
| re:secondary | True for secondary reviews |
| re:external | True for external reviews |
| re:pc | True for PC reviews |
| re:sylvia | True if reviewer matches “sylvia” |
Review preferences | |
| pref | Review preference |
| prefexp | Predicted expertise |
Aggregate functions
Aggregate functions calculate a value based on all of a paper’s submitted reviews and/or review preferences. For instance, “max(OveMer)” would return the maximum Overall merit score assigned to a paper.
An aggregate function’s argument is calculated once per visible review or preference. For instance, “max(OveMer/RevExp)” calculates the maximum value of “OveMer/RevExp” for any review, whereas “max(OveMer)/max(RevExp)” divides the maximum overall merit by the maximum reviewer expertise.
The top-level value of a formula expression cannot be a raw review score or preference. Use an aggregate function to calculate a property over all review scores.
Aggregates | |
| max(e), min(e) | Maximum, minimum |
| count(e) | Number of reviews where e is not null or false |
| sum(e) | Sum |
| avg(e) | Average (mean) |
| wavg(e, weight) | Weighted average; equals “sum(e * weight) / sum(weight)” |
| median(e) | Median |
| quantile(e, p) | Quantile; 0≤p≤1; 0 yields min, 0.5 median, 1 max |
| stddev(e) | Population standard deviation |
| var(e) | Population variance |
| stddev_samp(e), var_samp(e) | Sample standard deviation, sample variance |
| any(e) | True if any of the reviews have e true |
| all(e) | True if all of the reviews have e true |
| argmin(x, e) | Value of x when e is minimized |
| argmax(x, e) | Value of x when e is maximized |
| my(e) | Calculate e for your review |