4th P4 Workshop Home
Help · Sign in ·

 

Formulas

Program committee members and administrators can search and display formulas that calculate properties of paper scores—for instance, the standard deviation of papers’ Overall merit scores, or average Overall merit among reviewers with high Reviewer expertise.

To display a formula, use a search term such as “show:var(OveMer)” (show the variance in Overall merit scores, along with statistics for all papers). You can also graph formulas. To search for a formula, use a search term such as “formula:var(OveMer)>0.5” (select papers with variance in Overall merit greater than 0.5). Or save formulas using Search > View options > Edit formulas.

Formulas use a familiar expression language. For example, this computes the sum of the squares of the overall merit scores:

sum(OveMer**2)

This calculates an average of overall merit scores, weighted by expertise (high-expertise reviews are given slightly more weight):

wavg(OveMer, RevExp >= 4 ? 1 : 0.8)

And there are many variations. This version gives more weight to PC reviewers with the “#heavy” tag:

wavg(OveMer, re:#heavy + 1)

(“re:#heavy + 1” equals 2 for #heavy reviews and 1 for others.)

Formulas work better for numeric scores, but you can use them for letter scores too. HotCRP uses alphabetical order for letter scores, so the “min” of scores A, B, and D is A. For instance:

count(confidence=X)

Expressions

Formula expressions are built from the following parts:

Arithmetic

2Numbers
true, falseBooleans
nullThe null value
(e)Parentheses
e + e, e - eAddition, subtraction
e * e, e / e, e % eMultiplication, division, remainder
e ** eExponentiation
e == e, e != e,
e < e, e > e, e <= e, e >= e
Comparisons
!eLogical not
e1 && e2Logical and (returns e1 if e1 is false, otherwise returns e2)
e1 || e2Logical or (returns e1 if e1 is true, otherwise returns e2)
test ? iftrue : iffalseIf-then-else operator
let var = val in eLocal variable definition
greatest(e, e, ...)Maximum
least(e, e, ...)Minimum
coalesce(e, e, ...)Null coalescing: return first of es that is not null
log(e)Natural logarithm
log(e, b)Log to the base b
round(e[, m])Round to the nearest multiple of m

Submission properties

pidPaper ID
auNumber of authors
au:pcNumber of PC authors
au:textNumber of authors matching text

Tags

#tagnameTrue if this paper has tag tagname
tagval:tagnameThe value of tag tagname, or null if this paper doesn’t have that tag

Scores

overall-meritThis review’s Overall merit score
Only completed reviews are considered.
OveMerAbbreviations also accepted
OveMer:externalOverall merit for external reviews, null for other reviews
OveMer:R2Overall merit for round R2 reviews, null for other reviews

Submitted reviews

re:typeReview type
re:roundReview round
re:auwordsReview word count (author-visible fields only)
re:primaryTrue for primary reviews
re:secondaryTrue for secondary reviews
re:externalTrue for external reviews
re:pcTrue for PC reviews
re:sylviaTrue if reviewer matches “sylvia”

Review preferences

prefReview preference
prefexpPredicted expertise

Aggregate functions

Aggregate functions calculate a value based on all of a paper’s submitted reviews and/or review preferences. For instance, “max(OveMer)” would return the maximum Overall merit score assigned to a paper.

An aggregate function’s argument is calculated once per visible review or preference. For instance, “max(OveMer/RevExp)” calculates the maximum value of “OveMer/RevExp” for any review, whereas “max(OveMer)/max(RevExp)” divides the maximum overall merit by the maximum reviewer expertise.

The top-level value of a formula expression cannot be a raw review score or preference. Use an aggregate function to calculate a property over all review scores.

Aggregates

max(e), min(e)Maximum, minimum
count(e)Number of reviews where e is not null or false
sum(e)Sum
avg(e)Average (mean)
wavg(e, weight)Weighted average; equals “sum(e * weight) / sum(weight)”
median(e)Median
quantile(e, p)Quantile; 0≤p≤1; 0 yields min, 0.5 median, 1 max
stddev(e)Population standard deviation
var(e)Population variance
stddev_samp(e), var_samp(e)Sample standard deviation, sample variance
any(e)True if any of the reviews have e true
all(e)True if all of the reviews have e true
argmin(x, e)Value of x when e is minimized
argmax(x, e)Value of x when e is maximized
my(e)Calculate e for your review