Test runners¶

Test runner is part of Arete's functional subsystem

Source code is at gitlab.

Currently there are 5 runners:

uva
hackerrank
kattis
fsharp
python (3.9) without machine-learning packages
python-heavy (3.9) with machine-learning packages
java (java11, java15, java17)
prolog
c

API integration testers make requests against other platforms.

Webhook Updating¶

When tests are updated, arete can be made aware of those updates by sending push events through webhooks from the test git repositories.

Gitlab¶

URL: https://cs.ttu.ee/services/arete/api/v2/exercise
Secret: <Hook User> <Hook User Token>

You can create the hook user at https://cs.ttu.ee/services/arete/admin in the User table.

Github¶

Because Github uses its own method for authenticating webhook events, the events can't be authenticated through a hook user.
Github webhooks need a separate token, which can be created at https://cs.ttu.ee/services/arete/admin

URL: https://cs.ttu.ee/services/arete/api/v2/exercise
Secret: <Git Token> Content Type: application/json

Usage¶

First make sure you have the latest version installed. You can use arete front to do so.

you can place arete.json file in the root of the repository("/") or grouping folder("/EXAM/") or exercise("/EXAM/exam1/") in either tests repository or student repository for customised testing.

Features, that students can use are marked with double asterix**

Example:

{
  "dockerTimeout": 120,
  "dockerExtra": "stylecheck",
  "systemExtra": [
    "noMail",
    "noTesterFiles",
    "noStudentFiles",
    "noStd",
    "minimalFeedback"
  ],
  "groupingFolders": ["KT", "TK", "EXAM"],
  "solutionsRepository": "https://gitlab.cs.ttu.ee/iti0102-2020/ex",
  "testingPlatform": "python"
}

dockerTimeout - Maximum allowed code execution time

dockerContentRoot - Change root where student is

dockerTestRoot - Change where problem is

priority - priority of the job. Jobs are ran in order of higher priority. Min priority is 1 and max is 10. * priority < 8 - job will receive 4 GB on memory and 2 CPUs * priority >= 8 - job will receive 8 GB on memory and 4 CPUs

dockerExtra - Every subtester has different ones (seek subtester repository for more)

stylecheck - Check style in Python tester
-r TESTNG,COMPILER,CHECKSTYLE,FILEWRITER,REPORT - Check style in Java tester

systemExtra - More testing specifics

anonymous - Nothing is sent to arete backend (Use with caution)
allowAppending - All extra parameters what are collections - instead of overriding, append
giveTestRunnerControl - Allow tester setting slug and timestamp. Useful when integrating against external API
overrideContentRoot - Allow student changing content root which is being sent to tester (Use with caution)
overrideTestRoot - Allow student changing test root which is being sent to tester (Use with caution)
overrideExtra - Allow student changing extra which is being sent to tester (Use with caution)
overrideTestingPlatform - Allow student changing which tester is being used (Use with caution)
noOverall - Overall section in HTML is not shown
skipCopyingTests - Skip copying test to tester - also skips
skipCopyingStudent - Skip copying student files to tester
skipCopying - skipCopyingTests and skipCopyingStudent combined
noMail - Student doesn't get a mail with testing results**
noFiles - No files are returned from testing
noTesterFiles - No test files are returned from testing
noStudentFiles - No student files are returned from testing
noStd - Test container logs are not returned from testing
noStyle - Mail won't containt any style related fields and all traces of any stylecheck will be removed from return json
allowExternalMail - Mail will ve sent to external mails like @gmail.com. Without parameters @taltech.ee and @ttu.ee are only valid options
noOverride - Don't let arete.json files override existing parameters
noAurora - Don't send areteResponseDTO to aurora
noFeedback -
minimalFeedback -

Otherwise student sees the following: full

groupingFolders - Folders which are not slugs themselves but contain slugs. Note: students still need to have the same file tree for testing to be successful. For example with grouping folder EX path can be "EX/ex01-things/mycode" and then "EX/ex01-things" can be used as a test folder instead of "EX"

testerFolders - Folders which will be additionally copied to tester for testing. For example testingFolder "course-tester" will be copied to tester as "tests/course-tester" folder.

\ tree in git repository
-\course-tester

\ tree inside tester
-\host
-\tester
--\course-tester
-\student

solutionsRepository - Repository which contains solutions against what tests are ran after being updated.

testingPlatform - Programming language - used to determine tester

studentSlugs - List of slugs where student files are located.

dockerData - Easily expandable json form of sending extra data to testers (currently only used in python tester, can also be given through areteRequestDTO)

testfile_groups - Allows for grouping test files. It will follow 2 rules. Multiple groups can be created.
- Only show a test file if it gets a grade above 0%
- Don't show the next test file in the group, if the previous one didn't get 100%
error_report - Enables configurability of the error message output
- error_line - Show's the line of the error in the code i.e. assert value == expected, "assertion error message""
- error_message - Show's the error message from pytest i.e. assertion error message IF a specific error message is provided assert 1 == 2 IF an error message is not provided
ignore_flake8 - List of flake8 errors to ignore
ignore_pep257 - List of pep257 errors to ignore
colourful_output - If set to true, tester will output colourful pytest output in html

Example:

{
  "dockerData": {
    "testfile_groups": [["file_test1.py", "file_test2.py"]],
    "error_report": "error_line",
    "ignore_flake8": ["W292"],
    "ignore_pep257": ["D209"]
  }
}

Developing test runners¶

It is recommended to set up a CI-CD to docker repository where Arete will fetch the latest image.

ci-cd

.gitlab-ci.yml file should be similar to:

stages:
  - stage

deploy:
  tags:
    - ained
  stage: stage
  script:
    - docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
    - docker build -t automatedtestingservice/uva-tester:latest .
    - docker push automatedtestingservice/uva-tester:latest
    - |
      curl -X POST "https://cs.ttu.ee/services/arete/api/v2/course/uva-tester" -H "accept: */*" -H "X-Docker-Token: uva-tester $ARETE_TOKEN"
  only:
    - master

Replace uva-tester with your test-runner image name. Make sure it ends with -tester!

Input¶

Sub tester will get an input in the following fashion:

input

If you want to send files to students' mail then write the files to /host/ folder but make sure that the name of the file won't be either input.json nor output.json!

Where input.json will have content similar to but can be anything you want it to be:

{
  "contentRoot": "/student",
  "testRoot": "/tester",
  "extra": "stylecheck"
}

contentRoot - student files or whatever you want it to be

testRoot - test files or whatever you want it to be

extra - some extra parameters for tester

It is recommended, that your tester returns the response with Arete response structure and place the result into output.json

Where the type must be: arete

Few examples as follows: (Sub tester response parsed into Arete response)

Output¶

Sub tester should write its output to output.json file

Where output.json should have next fields and next fields only:

{
  "type": "arete",
  "version": "2.0",
  "testSuites": [],
  "errors": []
}

Content of testSuites and errors is visible to student so there should be as much information as possible.

NB! Please don't fill any OTHER fields in output.json as it may break testing (like email, hash, output, ...). Arete does that for you.

Arete response can be constructed by following the AreteResponseDTO schema

Example input for python-tester: example-input.json Example arete-output.json using example-input.json arete-example-output.json