Page Comparison

** This page is a work-in-progress.  If you see updates which will help people navigate the process, please reach out to help@riscv.org with your suggestions. **

This page provides technical details about running the Architecture Compatible Test (ACT) suite the base testing requirements to achieve the "RISC-V Compatible™" trademark and logo for ISA Profiles and Extensions.  Completion Successful completion and passing of the test suite is is one step Architectural Compatibility Test (ACT) suite is the foundation work in the process of gaining permission to use the “RISC-V Compatible” Trademark for Profiles and Extensions. See the RISC-V Branding Guidelines & Materials webpage for more details and additional requirements.

While this process is evolving and the tools continuously improve, the set of skills needed to complete this work is quite broad.  Common activities include installing open source packages, clone GitHub repositories, modifying YAML files, executing scripts, reading HTML, understanding ISA instructions, asking questions on the other stepsmailing lists, creating GitHub PRs, and reading many documents.

Background Documents

The following documents provide foundational information for key parts of this process:

• Architecture Compatibility Test Policy -  This document defines the testing which is one of the requirements to use the RISC-V Compatible Trademark, including re-running the tests when the implementation is changed or the ACT test version changes.
• Profiles Policy - This document groups RISC-V extension into various Profile names which may be used for claiming compatibility.
• Profiles summary spreadsheet - This spreadsheet may be used to decide the base profile and any additional extensions for which the target device wishes to claim compatibility.

Reading and understanding the contents of the above documents is recommended as a foundational step for successful completion of the RISC-V Compatible Permission Process.

Test Components

RISC-V-Compatible Testing requires the following components:

1. Python for the test driver system.
2. A RISC-V GNU Toolchain (cross) for the test drive system.
3. The RISC-V Compatibility Framework (RISCOF)
   
4. The RISC-V Config Tool (RISCV-CONFIG)
5. The Architecture Test Suite (ACT)
6. The SAIL golden model
   The RISC-V SAIL model is hosted in GitHub at https://github.com/riscv/sail-riscv
7. The vendor’s device under test (RTL or actual form)

For Testing should be performed using the most recent version of , stable versions.  For more information about the ACT suite, RISCOF, and RISCV-CONFIG, including the most recent versions and the list of supported RISC-V Profiles and Extensions, see the "Compatibility Test Framework" section of the RISC-V Technical Specifications wiki page. Testing should be performed using the most recent, stable versions

The subsequent steps in the next section will detail how to load and install these components.

Test Process

The general process for this testing is ordered list of steps to complete the test process are as follows:

1. Read and understand the Architecture Test Policy.  This document defines the testing which is one of the requirements to use the RISC-V Compatible Trademark, including re-running the tests when the implementation is changed or the ACT test version changes.
2. Decide the base profile and any additional extensions for which the target device wishes to claim compatibility.  More information about Profiles and Extensions can be found in the Profiles summary spreadsheetIf not available, install Python 3.X and the Virtualenv for Python on the system which will drive the test process.  This is not the RISC-V Device Under Test (DUT).
   More details are found the "Install Python" section of the RISCOF Documentation.
3. If not available, install a RISC-V cross toolchain to compile binaries (test files) on the system driving the test of the DUT.
   Sample directions for a 32-bit device are found in the "Install RISCV-GNU Toolchain" section of the RISCOF Documentation.
4. Build an ISA YAML file for the DUT. This YAML file should include the extensions for which RISC-V Compatibility will be claimed, but may also reflect the device's full capabilities assuming they are a superset.
   Example YAML files are provide in the examples directory of the RISC-V Config (riscv-config) GitHub repo.  They have names of XXX_isa.yaml. 
   The YAML syntax for ISA files is documented in the "ISA YAML Spec" section of the RISCV-CONFIG documentation.
   
5. Build Platform YAML file for the DUT.
   Example YAML files are provide in the examples directory of the RISC-V Config (riscv-config) GitHub repo.  They have names of XXX_platform.yaml. 
   The YAML syntax for ISA files is documented in the "Platform YAML Spec" section of the RISCV-CONFIG documentation.
   
6. Install the RISCV-CONFIG tool and use it to configure YAML files for all extensions included in step #2validate the ISA and Platform YAML files built in Steps #3 and #4.  For each Hart defined in your YAML, inspect the output for a message stating "No errors for Hart: ...".  Correct all errors before proceeding.  See the RISCV-CONFIG documentation for more details.

7. Install the RISCOF Framework following the directions in the "Install RISCOF" section of the RISCOF Documentation.

8. Configure RISCOF for a  Device Under Test (DUT), using the previously created YAML files, and install the ACT test suite , and SAIL .  Detailed steps for this process can be per the directions found in the "Running RISCV-ARCH-TESTS" section of the RISCOF Documentation.
9. Configure RISCOF for a  Device Under Test (DUT) for the target core to be tested.
10. Run the tests using RISCOF against the golden model and the DUT(Optional) If desired, generate a Test List and review the entries to ensure that the appropriate set of tests are covered. Specifically, look for test of ISA extensions which are not expected.
    More information on generate the Test List and interpret the results can be found in the "Test List Form" section of the RISCOF Documentation.

11. Run the ACT suite from RISCOF.  Detailed steps for this process can be found in the "Running Tests with RISCOF" section of the RISCOF Documentation.

12. Review the test results in the "Results" section of the report.html file created by RISCOF to ensure that the DUT test results

    match the results of the SAIL golden model

    indicate "Passed".  For tests marked "Failed", investigate the problem, correct if possible or address using actions defined in the "Test-Case Waivers" section of the Architecture Compatibility Test Policy.
    Specific test source can be found in the particular riscv-test-suite directory of the RISC-V Architecture Test (riscv-arch-test) GitHub repo used to invoke RISCOF (--suite) in the previous step.

13. Repeat Steps #9 and #10 as needed.
14. When ready, submit the test results and supporting information per the directions in the “Where to Report Test Results” section of the Architecture Test Policy.
    

Additional Information

A RISC-V Technical Session titled "Using Template Repo  to setup/run RISC-V Arch Test Suite under RHEL/Ubuntu" was recorded in April 2023 that provides more details on how to successfully execute the ACT.  It provides information about a member-provided repo (link) that supplements setup of the ACT environment with some extra scripts and files.  Details, including the presentation and recording links are available here.

Support

Questions about the RISC-V Compatible Trademark Permission Process should be sent to the RISC-V Compatible Trademark Permission Process mailing list (archives) at compatible-isa-trademark@groupstesting@groups.riscv.org.   Public submissions accepted, but maybe moderated.

Any questions about ACT, RISCOF, or RISCV-CONFIG should be posted to the Architecture Test Special Interest Group (SIG) mailing list (archives) at sig-arch-test@lists.riscv.org.   Joining this group (link) strongly encouraged for RISC-V members working toward mark usage.

...