Documentation crass.ai v0.2
last update: 19.08.2021
1.1. General Information
This document contains the description of the CRASS API.
2. Sign Up
2.1. How to sign up an AI
Simply fill out the form AI sign up. We will (manually) review your entry and unlock your account.
After you account is unlocked we will provide you with your API access information so you can retrieve the relevant test data.
3. Working with OSM-Test Data
3.1. Retrieving OSM Test Data
In order to retrieve test data you need to have a signed up AI see "2.1. How to sign up an AI". Once you are signed up, you can collect the test data using our REST API.
First you need to collect the available premises, which describe the situation. Use the API credentials you were given after registring your AI and execute the following command (It is case-sensitive):
You will receive a dataset containing the respective premise as well as the associated premise category.
"premise": "A bird flies over a bridge.",
"created_date": "2021-03-24 07:53:40",
"mod_date": "2021-03-24 15:51:49"
The premise is the situation which needs to be understood in order to make a sensible judgment concerning the connected counterfactual. In the beginning we provide very simple English present tense clauses describing an event using two nouns which are connected by a verb and an auxiliary preposition describing the event further or which is mandated by the applied verb itself.
Next you need the list of the available counterfactuals.
You will receive a dataset containing the respective counterfactuals.
"counterfactual": "What would have happened if the car had driven by the tree?",
"created_date": "2021-03-24 07:58:17",
"mod_date": "2021-04-03 22:08:02"
counterf_ref_premise_idto match the counterfactual to the pertaining premise.
Now, you're good to go. Use your pretrained model and let it answer the counterfactual scenarios.
3.2. Storing OSM Test Data
After you have successfully answered all retrieved counterfactuals, here is the description of how to store the information into the crass.ai system.
All you need to do is to provide all answers in the following format:
"answer": "The bird would have crossed the bridge not as fast."
Provide the correct ID of the counterfactuals to which a given answer belongs in the
result_ref_counterf_idfield. Use your assigned AI ID to submit in the
4. Working with FTM-Test Data
4.1. Retrieving FTM Test Data
In order to retrieve test data you need to have a signed up AI see "2.1. How to sign up an AI". Once you are signed up, you can collect the test data using our REST API. Use the API credentials you were given after registring your AI and execute the following command (It is case-sensitive):
You will receive an dataset containing the respective PCTs
"pct": "A man sees a lion. What would have happened if the man had fought the lion?",
"answer1": "He would have talked to the lion.",
"answer2": "He would have died.",
"answer3": "The lion would have died.",
"answer4": "The lion would have seen a man."
For each PCT four possible answers are given ranging from highly probable to very unlikely as a correct answer. The LLM has to judge adequately which answer is most suitable in the presented hypothetical situation. Use your pretrained model and let it judge which answer is correct
4.2. Storing FTM Test Data
After you have successfully judged the answers of the PCTs, here is the description of how to store the information into the crass.ai system.
All you need to do is to provide all answers in the following format to: