Skip to content

Special Judge

This page introduces the methods for writing Special Judge (spj) for some judging tools/OJs.

Introduction

Special Judge (abbreviated as spj, also known as checker) is a program used to determine the validity of an answer when a problem has multiple possible solutions.

Warning

The spj should also check whether there is extra content at the end of the file and whether the output format is correct (e.g., the problem requires numbers to be separated by a single space, but the contestant uses newlines). However, currently only Testlib can conveniently handle the former, while the latter is rarely specifically checked.

When judging floating-point numbers, be cautious of NaN. Unreasonable judgment methods may lead to situations where outputting NaN results in an AC.

When reading the contestant's file, ensure that the required content is correctly read to prevent runtime errors in the spj. (Some OJs treat spj runtime errors as system errors.)

Note

The following examples use C++ as the programming language, with the requirement: "the difference between the standard answer and the contestant's answer must be less than 1e-3, the file name is num, and the full score for a single test case is 10 points."

Testlib

Refer to: Testlib/Introduction, Testlib/Checker

Testlib is a C++ library designed to assist problem setters in writing algorithmic competition problems using C++.

Judging tools/OJs that must use Testlib for spj: Codeforces, Luogu, UOJ, etc.

Judging tools/OJs that can use Testlib for spj: LibreOJ (Lyrio), Lemon, Nowcoder, etc.

The modified version of Testlib required for SYZOJ 2 is hosted on pastebin1, but this version does not modify the interactive mode. A version of Testlib that supports interactive mode on SYZOJ 2 is hosted at syzoj/testlib.

The modified version of Testlib required for Lemon is hosted on GitHub - GitPinkRabbit/Testlib-for-Lemons. Note that when registering a checker with this version of Testlib, use registerLemonChecker() instead of registerTestlibCmd(). This version is based on matthew99's older version and adds some new features of Testlib. If you use LemonLime, you can use the native Testlib.

The modified version of Testlib required for DOMJudge is hosted on cn-xcpc-tools/testlib-for-domjudge. This version of Testlib can serve as both a checker for Special Judge and an interactor for interactive problems.

The modified version of Testlib required for Arbiter is hosted on testlib-for-arbiter.

For other judging tools/OJs, most require adapting Testlib according to their spj writing format, and uploading testlib.h together with the spj; or placing testlib.h in the include directory.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
#include "testlib.h"
//
#include <cmath>

int main(int argc, char *argv[]) {
  /*
   * inf: input
   * ouf: contestant output
   * ans: standard output
   */
  registerTestlibCmd(argc, argv);

  double pans = ouf.readDouble(), jans = ans.readDouble();

  if (abs(pans - jans) < 1e-3)
    quitf(_ok, "Good job\n");
  else
    quitf(_wa, "Too big or too small, expected %f, found %f\n", jans, pans);
}

Lemon

Note

Lemon has an existing modified version of Testlib, and it is recommended to use Testlib.

The latest version of LemonLime already supports using the original Testlib to write checkers. If you use LemonLime, it is recommended to use Testlib.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#include <cmath>
#include <cstdio>

int main(int argc, char* argv[]) {
  /*
   * argv[1]: input
   * argv[2]: contestant output
   * argv[3]: standard output
   * argv[4]: score for a single test case
   * argv[5]: output final score (0 ~ argv[4])
   * argv[6]: output error report
   */
  FILE* fin = fopen(argv[1], "r");
  FILE* fout = fopen(argv[2], "r");
  FILE* fstd = fopen(argv[3], "r");
  FILE* fscore = fopen(argv[5], "w");
  FILE* freport = fopen(argv[6], "w");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3) {
    fprintf(fscore, "%s", argv[4]);
    fprintf(freport, "Good job\n");
  } else {
    fprintf(fscore, "%d", 0);
    fprintf(freport, "Too big or too small, expected %f, found %f\n", jans,
            pans);
  }
}

Cena

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#include <cmath>
#include <cstdio>

int main(int argc, char* argv[]) {
  /*
   * FILENAME.in: input
   * FILENAME.out: contestant output
   * argv[1]: score for a single test case
   * argv[2]: standard output
   * score.log: output final score (0 ~ argv[1])
   * report.log: output error report
   */
  FILE* fin = fopen("num.in", "r");
  FILE* fout = fopen("num.out", "r");
  FILE* fstd = fopen(argv[2], "r");
  FILE* fscore = fopen("score.log", "w");
  FILE* freport = fopen("report.log", "w");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3) {
    fprintf(fscore, "%s", argv[1]);
    fprintf(freport, "Good job\n");
  } else {
    fprintf(fscore, "%d", 0);
    fprintf(freport, "Too big or too small, expected %f, found %f\n", jans,
            pans);
  }
}

CCR

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#include <cmath>
#include <cstdio>

int main(int argc, char* argv[]) {
  /*
   * stdin: input
   * argv[2]: standard output
   * argv[3]: contestant output
   * stdout:L1: output final score ratio (0 ~ 1)
   * stdout:L2: output error report
   */
  FILE* fout = fopen(argv[3], "r");
  FILE* fstd = fopen(argv[2], "r");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3) {
    printf("%d\n", 1);
    printf("Good job\n");
  } else {
    printf("%d\n", 0);
    printf("Too big or too small, expected %f, found %f\n", jans, pans);
  }
}

Arbiter

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#include <cmath>
#include <cstdio>

int main(int argc, char* argv[]) {
  /*
   * argv[1]: input
   * argv[2]: contestant output
   * argv[3]: standard output
   * /tmp/_eval.score:L1: output error report
   * /tmp/_eval.score:L2: output final score
   */
  FILE* fout = fopen(argv[2], "r");
  FILE* fstd = fopen(argv[3], "r");
  FILE* fscore = fopen("/tmp/_eval.score", "w");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3) {
    fprintf(fscore, "Good job\n");
    fprintf(fscore, "%d", 10);
  } else {
    fprintf(fscore, "Too big or too small, expected %f, found %f\n", jans,
            pans);
    fprintf(fscore, "%d", 0);
  }
}

HUSTOJ

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#include <cmath>
#include <cstdio>

#define AC 0
#define WA 1

int main(int argc, char* argv[]) {
  /*
   * argv[1]: input
   * argv[2]: standard output
   * argv[3]: contestant output
   * exit code: return judgment result
   */
  FILE* fin = fopen(argv[1], "r");
  FILE* fout = fopen(argv[3], "r");
  FILE* fstd = fopen(argv[2], "r");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3)
    return AC;
  else
    return WA;
}

QDUOJ

In comparison, QDUOJ is slightly more troublesome. For problems with spj, there is no standard output file; the standard solution must be written into the spj, and after generating the standard output, the judgment is made.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#include <cmath>
#include <cstdio>

#define AC 0
#define WA 1
#define ERROR -1

double solve(...) {
  // std
}

int main(int argc, char* argv[]) {
  /*
   * argv[1]: input
   * argv[2]: contestant output
   * exit code: return judgment result
   */
  FILE* fin = fopen(argv[1], "r");
  FILE* fout = fopen(argv[2], "r");

  double pans, jans;
  fscanf(fout, "%lf", &pans);

  jans = solve(...);
  if (abs(pans - jans) < 1e-3)
    return AC;
  else
    return WA;
}

HDOJ

HDOJ is essentially the same as QDUOJ; the spj also needs to implement the standard solution and compare it with the contestant's output. However, unlike QDUOJ, HDOJ compares the answer with the content output by the spj to the standard output before giving the final result. Therefore, when uploading the output, only the spj's output when correct needs to be uploaded.

HDOJ requires uploading a compiled binary file for Windows, not the source code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#include <cmath>
#include <cstdio>

double solve(FILE* fin) {
  // std, read input from fin
}

int main(int argc, char* argv[]) {
  /*
   * argv[1]: input
   * stdin: contestant output
   */
  FILE* fin = fopen(argv[1], "r");

  double pans, jans;
  if (scanf("%lf", &pans) != 1) {
    printf("WA\n");
    goto finish;
  }

  jans = solve(fin);
  if (abs(pans - jans) < 1e-3)
    printf("AC\n");
  else
    printf("WA\n");

finish:
  fclose(fin);
  return 0;
}

The corresponding answer file is:

1
AC

SYZOJ 2

Note

SYZOJ 2 has an existing modified version of Testlib, and it is recommended to use Testlib.

The latest version of LibreOJ is no longer based on SYZOJ but on Lyrio. Lyrio supports using the original Testlib to write checkers, which is more universal and recommended.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#include <cmath>
#include <cstdio>

int main(int argc, char* argv[]) {
  /*
   * in: input
   * user_out: contestant output
   * answer: standard output
   * code: contestant code
   * stdout: output final score (0 ~ 100)
   * stderr: output error report
   */
  FILE* fin = fopen("input", "r");
  FILE* fout = fopen("user_out", "r");
  FILE* fstd = fopen("answer", "r");
  FILE* fcode = fopen("code", "r");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3) {
    printf("%d", 100);
    fprintf(stderr, "Good job\n");
  } else {
    printf("%d", 0);
    fprintf(stderr, "Too big or too small, expected %f, found %f\n", jans,
            pans);
  }
}

Nowcoder

Note

Nowcoder has an existing modified version of Testlib, and it is recommended to use Testlib.

Refer to: How to create Special Judge programming problems on Nowcoder

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#include <cmath>
#include <cstdio>

#define AC 0
#define WA 1

int main(int argc, char* argv[]) {
  /*
   * input: input
   * user_output: contestant output
   * output: standard output
   * exit code: return judgment result
   */
  FILE* fin = fopen("input", "r");
  FILE* fout = fopen("user_output", "r");
  FILE* fstd = fopen("output", "r");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3)
    return AC;
  else
    return WA;
}

DOMJudge

Note

DOMJudge supports spj written in any language. Refer to: problemarchive.org output validator format.

DOMJudge has an existing modified version of Testlib, and it is recommended to use Testlib.

DOMJudge's Testlib and documentation for importing Polygon problem packages: https://github.com/cn-xcpc-tools/testlib-for-domjudge

DOMJudge's default comparator includes floating-point comparison with precision. Simply add float_tolerance 1e-3 to the validator_flags in the problem configuration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
#include <cmath>
#include <cstdio>

#define AC 42
#define WA 43
char reportfile[50];

int main(int argc, char* argv[]) {
  /*
   * argv[1]: input
   * argv[2]: standard output
   * argv[3]: folder for judging information output
   * stdin: contestant output
   */
  FILE* fin = fopen(argv[1], "r");
  FILE* fstd = fopen(argv[2], "r");
  sprintf(reportfile, "%s/judgemessage.txt", argv[3]);
  FILE* freport = fopen(reportfile, "w");

  double pans, jans;
  scanf("%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3) {
    fprintf(freport, "Good job\n");
    return AC;
  } else {
    fprintf(freport, "Too big or too small, expected %f, found %f\n", jans,
            pans);
    return WA;
  }
}

You can also use the header file validate.h provided by Kattis Problem Tools to write more complex functionalities.

References

  1. LibreOJ now supports testlib checkers! ↩

本页面最近更新：，更新历史
发现错误？想一起完善？ 在 GitHub 上编辑此页！
本页面贡献者：OI-wiki
本页面的全部内容在 CC BY-SA 4.0 和 SATA 协议之条款下提供，附加条款亦可能应用