CALL FOR ARTIFACT

The reproducibility of experimental results is crucial to foster an atmosphere of trustworthy, open, and reusable research. To improve and reward reproducibility, FormaliSE 2023 continues its Artifact Evaluation (AE) procedure.

An artifact is any additional material (software, data sets, machine-checkable proofs, etc.) that substantiates the claims made in the paper and ideally makes them fully reproducible. Submission of an artifact is optional but encouraged for all papers where it can support the results presented in the paper.

Artifact review is single-blind (the paper corresponding to an artifact must still follow the double-blind submissions requirements) and will be conducted concurrently with the paper reviewing process.

Artifacts will be handled by a separate AE Committee, and the AE process will be set up such that the anonymization of the corresponding papers will not be compromised. Therefore, the reviewers of the papers will not directly interact with the reviewers of the artifacts. Recommendations about artifact acceptance will be handled by the AE Chair, and passed over to the paper reviewers. The results of the evaluation may be taken into consideration for the corresponding paper acceptance decision. Accepted papers with a successfully evaluated artifact will receive a badge to be shown on the published paper’s title page.

Artifacts will be assessed with respect to their consistency with the results presented in the paper, their completeness, their documentation, and their ease of use. The AE will include an initial check for technical issues. Authors of artifacts will interact with the reviewers from January 25th to February 8th by means of a shared document, to help resolve any technical problems that prevent the evaluation of an artifact, if necessary.

ARTIFACT SUBMISSION GUIDELINES

An artifact consists of an Artifact Submission and an Artifact Repository. The Artifact Repository contains the actual code and documentation. The Artifact Submission includes a link to the Artifact Repository.

The Artifact Submission consists of:

• Abstract: this includes the paper ID of the paper associated to the artifact, and a brief description of the artifact, also indicating the repository where the artifact is located. To ensure the integrity of the submitted artifacts, please compute the SHA checksum of the artifact file and provide it within the artifact submission. A checksum can be obtained by running the command sha1sum (Linux, macOS), or File Checksum Integrity Verifier (Microsoft Windows). The memory requirements of the artifact should also be indicated in the submission, e.g. 4 GB, 8 GB, etc.
  The use of permanent public repositories such as Figshare or Zenodo is highly recommended, although not a mandatory requirement.
• Paper: the submitted paper. This is required to facilitate reviewers, but complete documentation of the artifact should be provided in the repository so that replication and reuse are possible without having an accurate reading of the paper.

The Artifact Submission should be made by through EasyChair.

Artifact Submissions should be made by the Artifact Registration deadline. Updates on Artifact Submissions are allowed until the Artifact Submission deadline.

The Artifact Repository consists of:

• README.md file, describing the content of the artifact (folders and files), and detailed guidelines on how to replicate the original study, and possibly reuse the artifact.
• LICENSE.md file, containing the license(s) used for the artifact. At least the evaluation committee should be allowed to review the Artifact.
• Artifact content , including all files needed to execute the artifact, as described in the Readme.md file.
• Artifact Virtual Machine (VM) Image, that must be kept accessible via a working web link throughout the entire evaluation process.
  As the OS of the VM image, please choose a commonly used Linux distribution that has been tested with the virtual machine software. For the preparation of the VM image please use VirtualBox and save the VM image as an Open Virtual Appliance (OVA) file.
  For non-executable artifacts (e.g., datasets), the Artifact VM Image is not required.

The Artifact Repository should be completed by the Artifact Submission deadline. More guidelines on the content of the repository are reported below in the section “Guidelines for Artifact Repository Preparation”.

GUIDELINES FOR ARTIFACT REPOSITORY PREPARATION

We ask the authors to consider the following guidelines when preparing the artifact:

• Document how to reproduce most, or ideally all, of the (experimental) results of the paper using the artifact.
• Provide also clear documentation on how to use the artifact, e.g. which files to read and commands to execute, assuming minimum expertise of the user.
• The evaluation process can be kept simple (and documentation reduced) by providing easy-to-use scripts, which favour artifact usability.
• The artifact should not require the user to install additional software before running, that is, all required packages, libraries, etc. have to be installed on the provided VM image.
• For the special case of experiments that require large amounts of resources (hardware or time), it is recommended to provide a way to replicate a subset of the results of the paper with reasonably modest resources (RAM, number of cores), so that the results can be reproduced on common laptop hardware in a reasonable amount of time. Do include the full set of experiments as well (for those reviewers with sufficient hardware or time), just make it optional. Please indicate in the EasyChair submission form how much memory a reviewer will need to run your artifact (at least to replicate the chosen subset).

POSSIBILITY FOR EXEMPTION

In case you intend to present an artifact but cannot comply with the guideline above, please do not hesitate to contact the Artifact Evaluation Chair in advance before the artifact submission deadline.

IMPORTANT DATES

• Artifact Registration: (optional) January 16th, 2023 January 27th, 2023
• Artifact Submission: January 23rd, 2023 January 31st, 2023
• Reviewers-Authors Interaction to Fix Artifacts: January 25th-February 8th, 2023 February 1st-February 8th, 2023
• Artifact Notification: February 21st, 2023