CALL FOR ARTIFACT
The reproducibility of experimental results is crucial to
foster an atmosphere of trustworthy, open, and reusable research.
To improve and reward reproducibility, FormaliSE 2023 continues its
Artifact Evaluation (AE) procedure.
An artifact is any additional material
(software, data sets, machine-checkable proofs, etc.)
that substantiates the claims made in the paper and ideally
makes them fully reproducible.
Submission of an artifact is optional but encouraged for
all papers where it can support the results presented in
Artifact review is single-blind
(the paper corresponding to an artifact must still follow
the double-blind submissions requirements) and will be
conducted concurrently with the paper reviewing process.
Artifacts will be handled by a separate AE
Committee, and the AE process will be set up such that the
anonymization of the corresponding papers will not be
compromised. Therefore, the reviewers of the papers will
not directly interact with the reviewers of the artifacts.
Recommendations about artifact acceptance will be handled
by the AE Chair, and passed over to the paper reviewers.
The results of the evaluation may be taken into
consideration for the corresponding paper acceptance
decision. Accepted papers with a successfully evaluated
artifact will receive a badge to be shown on the published
paper’s title page.
Artifacts will be assessed with respect to
their consistency with the results presented in the paper,
their completeness, their documentation, and their ease
of use. The AE will include an initial check for technical
issues. Authors of artifacts will interact with the
reviewers from January 25th to February 8th by means of a
shared document, to help resolve any technical problems
that prevent the evaluation of an artifact, if necessary.
ARTIFACT SUBMISSION GUIDELINES
An artifact consists of an Artifact
Submission and an Artifact Repository.
The Artifact Repository contains the actual code and
documentation. The Artifact Submission includes a link
to the Artifact Repository.
The Artifact Submission consists of:
- Abstract: this includes the paper ID of the paper
associated to the artifact, and a brief description of
the artifact, also indicating the repository where the
artifact is located. To ensure the integrity of the
submitted artifacts, please compute the SHA checksum
of the artifact file and provide it within the artifact
submission. A checksum can be obtained by running the
(Linux, macOS), or
File Checksum Integrity Verifier (Microsoft Windows).
The memory requirements of the artifact should also
be indicated in the submission, e.g. 4 GB, 8 GB, etc.
The use of permanent public repositories such as Figshare
or Zenodo is highly recommended, although not a mandatory
- Paper: the submitted paper.
This is required to facilitate reviewers, but complete
documentation of the artifact should be provided in the repository
so that replication and reuse are possible without having
an accurate reading of the paper.
The Artifact Submission should be made by through
Artifact Submissions should be made by the Artifact
Registration deadline. Updates on Artifact Submissions are
allowed until the Artifact Submission deadline.
The Artifact Repository consists of:
file, describing the content of the artifact (folders and files),
and detailed guidelines on how to replicate the original study,
and possibly reuse the artifact.
file, containing the license(s) used for the artifact.
At least the evaluation committee should be allowed to review the Artifact.
- Artifact content
, including all files needed to execute the artifact,
as described in the Readme.md file.
- Artifact Virtual Machine (VM) Image, that must be kept
accessible via a working web link throughout the entire evaluation process.
As the OS of the VM image, please choose a commonly used Linux distribution that has been tested
with the virtual machine software.
For the preparation of the VM image please use
VirtualBox and save the VM image as an
Open Virtual Appliance (OVA) file.
For non-executable artifacts (e.g., datasets), the Artifact VM Image is not required.
The Artifact Repository should be completed by the Artifact
Submission deadline. More guidelines on the content of the repository
are reported below in the section “Guidelines for Artifact Repository Preparation”.
GUIDELINES FOR ARTIFACT REPOSITORY PREPARATION
We ask the authors to consider the following guidelines
when preparing the artifact:
- Document how to reproduce most, or ideally all, of the (experimental)
results of the paper using the artifact.
- Provide also clear documentation on how to use the artifact, e.g.
which files to read and commands to execute, assuming minimum expertise
of the user.
- The evaluation process can be kept simple (and documentation reduced)
by providing easy-to-use scripts, which favour artifact usability.
- The artifact should not require the user to install additional
software before running, that is, all required packages, libraries,
etc. have to be installed on the provided VM image.
- For the special case of experiments that require large amounts of resources
(hardware or time), it is recommended to provide a way to replicate
a subset of the results of the paper with reasonably modest resources
(RAM, number of cores), so that the results can be reproduced on
common laptop hardware in a reasonable amount of time.
Do include the full set of experiments as well (for those reviewers with
sufficient hardware or time), just make it optional.
Please indicate in the EasyChair submission form how much memory
a reviewer will need to run your artifact
(at least to replicate the chosen subset).
POSSIBILITY FOR EXEMPTION
In case you intend to present an artifact but cannot
comply with the guideline above, please do not hesitate to contact the Artifact
Evaluation Chair in advance before the artifact submission deadline.
- Artifact Registration: January 16th, 2023
- Artifact Submission: January 23rd, 2023
- Reviewers-Authors Interaction to Fix Artifacts: January 25th-February 8th, 2023
- Artifact Notification: February 21st, 2023