# AI Inference · Chromia Docs (concept)

> The AI Inference extension brings AI inference capabilities directly into dapps, enabling intelligent, real-time decision-making within the on-chain…

Source: https://docs.chromia.com/ecosystem/extensions/ai_inference/

# AI Inference

On this page

* [Enable extension](#enable-extension)
* [Blockchain configuration](#blockchain-configuration)
* [Configuration parameters](#configuration-parameters)
* [Integrating with Rell](#integrating-with-rell)
* [Minimal implementation example](#minimal-implementation-example)
* [Advanced usage examples](#advanced-usage-examples)
* [Chat-based inference](#chat-based-inference)
* [Batch processing](#batch-processing)
* [Conditional AI processing](#conditional-ai-processing)
* [Deployment to Testnet](#deployment-to-testnet)
* [Leasing a container](#leasing-a-container)
* [Deployment](#deployment)
* [Testing and examples](#testing-and-examples)
* [Simple inference test](#simple-inference-test)
* [Chat inference test](#chat-inference-test)
* [Check inference status](#check-inference-status)
* [Batch testing](#batch-testing)

The AI Inference extension brings AI inference capabilities directly into dapps, enabling intelligent, real-time
decision-making within the on-chain environment. By leveraging externally trained models, developers can seamlessly
embed AI-driven logic into their applications while maintaining blockchain's core principles of trustlessness and
transparency. This integration unlocks new possibilities for responsive and adaptive dapps.

This extension uses the Hybrid Compute framework.

[AI inference extension repository.](https://gitlab.com/chromaway/core/ai-inference-extension)

## Enable extension

You can either pick the AI inference extension when leasing your container. Or you add the AI inference JAR extension to
your container, making it possible to combine with other extensions.

## Blockchain configuration

You will need to enable the Hybrid Compute framework, configure it to use the AI inference engine provided by this
extension, set the appropriate timeout for inference computations and configure the model to use for inference.

chromia.ymlYAML

```
blockchains:
  ai_inference_dapp: # Ensure this name is unique. Rename it if you encounter duplicate issues during deployment.
    module: main
    config:
      sync_ext:
        - "net.postchain.hybridcompute.HybridComputeSynchronizationInfrastructureExtension"
      gtx:
        modules:
          - "net.postchain.hybridcompute.HybridComputeGTXModule"
      hybridcompute:
        engine: "net.postchain.gtx.extensions.ai_inference.AiInferenceComputeEngine"
      ai_inference:
        model: "your-model-name" # Specify model name here. Example: Qwen/Qwen2.5-1.5B-Instruct
        inference_timeout_seconds: 60 # Timeout in seconds for each inference
        verification_timeout_seconds: 10 # Timeout in seconds for each verification
        max_completion_tokens:
          100 # An upper bound for the number of tokens that can be generated for a completion,
          # including visible output tokens and reasoning tokens.
```

### Configuration parameters

| Parameter | Description | Default | Required |
| --- | --- | --- | --- |
| `model` | Name of the AI model to use for inference | - | Yes |
| `inference_timeout_seconds` | Timeout in seconds for each inference | 60 | No |
| `verification_timeout_seconds` | Timeout in seconds for each verification | 10 | No |
| `max_completion_tokens` | Upper bound for the number of tokens that can be generated for a completion, including visible output tokens and reasoning tokens | - | Yes |

## Integrating with Rell

Install the Rell libraries:

chromia.ymlYAML

```
libs:
  hybridcompute:
libs:
  com.chromia.hybridcompute:
    version: 3.35.4
  com.chromia.ai_inference:
    version: 0.3.6
compile:
  rellVersion: 0.14.15
```

Import the module:

Rell

```
import ai: lib.ai_inference;
```

Submit a simple inference request with this function:

Rell

```
/**
 * Submits an inference request.
 *
 * @param id A unique identifier for the inference request.
 * @param prompt The prompt to generate text for.
 */
function submit_inference_request(id: text, prompt: text)
```

Or submit a chat inference request with this function:

Rell

```
/**
 * Submits an chat inference request.
 *
 * @param id A unique identifier for the inference request.
 * @param messages A list of messages comprising the conversation so far.
 */
function submit_chat_inference_request(id: text, messages: list<chat_message>)
```

Fetch the result with this function:

Rell

```
struct inference_result {
    /** The result of the inference, or null if an error occurred. */
    result: text?;

    /** An error message if the inference failed, or null if no error occurred. */
    error: text?;

    /** RID of the transaction where the result was reported. */
    tx_rid: byte_array;

    /** Index of the operation where the result was reported. */
    op_index: integer;
}

/**
 * Fetches the result of a previously submitted inference request.
 *
 * @param id The identifier of the inference request for which the result is being fetched.
 * @return The result, or `null` if not ready yet
 */
function fetch_inference_result(id: text): inference_result?
```

Or extend this to be notified:

Rell

```
/**
 * Called when an inference is finished, successfully or failed.
 *
 * @param id The identifier of the inference request
 * @param inference_result The result
 */
@extendable function on_inference_result(id: text, inference_result)
```

## Minimal implementation example

Here's a minimal implementation
[example in Rell](https://gitlab.com/chromaway/core/ai-inference-extension/-/tree/dev/example):

main.rellRell

```
module;

import ai: lib.ai_inference;

// Note: add authentication to this operation before production use
operation submit_inference_request(id: text, prompt: text) {
    ai.submit_inference_request(id, prompt);
}

// Note: add authentication to this operation before production use
operation submit_chat_inference_request(id: text, prompt: text) {
    ai.submit_chat_inference_request(id, [ai.chat_message(role="user", message=prompt)]);
}

query fetch_inference_result(id: text): ai.inference_result? = ai.fetch_inference_result(id);
```

## Advanced usage examples

### Chat-based inference

chat\_inference.rellRell

```
operation submit_conversation(conversation_id: text, user_message: text) {
    // For a simple chat, create a conversation with system prompt and user message
    val messages = [
        ai.chat_message(role="system", message="You are a helpful assistant."),
        ai.chat_message(role="user", message=user_message)
    ];

    ai.submit_chat_inference_request(conversation_id, messages);
}

operation continue_conversation(conversation_id: text, conversation_history: list<ai.chat_message>, user_message: text) {
    // Add the new user message to the conversation history
    val updated_messages = conversation_history + [ai.chat_message(role="user", message=user_message)];

    ai.submit_chat_inference_request(conversation_id, updated_messages);
}
```

### Batch processing

batch\_inference.rellRell

```
operation submit_batch_inference(base_id: text, prompts: list<text>) {
    var index = 0;
    for (prompt in prompts) {
        val request_id = "%s_%d".format(base_id, index);
        ai.submit_inference_request(request_id, prompt);
        index += 1;
    }
}

query get_batch_results(base_id: text, count: integer): list<ai.inference_result?> {
    val results = list<ai.inference_result?>();
    var index = 0;
    while (index < count) {
        val request_id = "%s_%d".format(base_id, index);
        results.add(ai.fetch_inference_result(request_id));
        index += 1;
    }
    return results;
}
```

### Conditional AI processing

conditional\_ai.rellRell

```
operation smart_content_filter(content_id: text, content: text) {
    // Only process content that meets certain criteria
    if (content.size() > 10 and content.size() < 1000) {
        val prompt = "Analyze the following content for appropriateness: %s".format(content);
        ai.submit_inference_request(content_id, prompt);
    }
}

@extend(ai.on_inference_result)
function (id: text, result: ai.inference_result) {
    if (result.result != null) {
        // Parse AI response and take action
        if (result.result.contains("inappropriate")) {
            // Flag content for review
            flag_content_for_review(id, result.tx_rid, result.op_index);
        } else {
            // Approve content
            approve_content(id, result.tx_rid, result.op_index);
        }
    }
}
```

## Deployment to Testnet

### Leasing a container

If you are unfamiliar with leasing a container, follow these [steps](https://docs.chromia.com/build/deployment/testnet-tokens/get-tchr-chromia/).

info

Ensure you select the `AI Inference extension` when leasing the container. The AI Inference Extension is part of
Chromia's extension ecosystem and may require cluster-level approval depending on your deployment target.

### Deployment

Follow the [deployment steps](https://docs.chromia.com/get-started/create-dapp/deploy-to-testnet/).

You can find the BRID for the Testnet Directory Chain in the
[explorer](https://explorer.chromia.com/testnet/cluster/system).

The deployment section in your `chromia.yml` will look like this:

YAML

```
deployments:
testnet: # Deployment Target name
  brid: x"6F1B061C633A992BF195850BF5AA1B6F887AEE01BB3F51251C230930FB792A92" # Blockchain RID for Testnet Directory Chain
  url: https://node0.testnet.chromia.com # Target URL for one of the nodes in Testnet
  container: 4d7890243fe710...08c724700cbd385ecd17d6f # Replace with your container ID (Example: container: 15ddfcb25dcb43577ab311fe78aedab14fda25757c72a787420454728fb80304)
  chains:
    ai_inference_dapp: x"9716FBFF3...663FEC673710" # Replace with the actual BRID after the dapp gets deployed. You can find it in the terminal during deployment.
```

## Testing and examples

### Simple inference test

In the examples provided, `ai_inference_dapp` and `testnet` refer to the
[deployment parameters](#deployment-to-testnet).

To send a simple inference request, use:

Terminal

```
chr tx -bc ai_inference_dapp -d testnet submit_inference_request 'a1' 'What is artificial intelligence?'
```

To get a response, execute:

Terminal

```
chr query -bc ai_inference_dapp -d testnet fetch_inference_result id=a1
```

Output:

Terminal

```
[
  "error": null,
  "result": "Artificial intelligence (AI) refers to the simulation of human intelligence in machines...",
  "tx_rid": "1234567890ABCDEF...",
  "op_index": 0
]
```

### Chat inference test

To send a chat inference request, use:

Terminal

```
chr tx -bc ai_inference_dapp -d testnet submit_chat_inference_request 'chat1' 'Hello, how are you?'
```

### Check inference status

Terminal

```
chr query -bc ai_inference_dapp -d testnet get_inference_status id=a1
```

### Batch testing

Terminal

```
# Submit multiple requests
chr tx -bc ai_inference_dapp -d testnet submit_inference_request 'batch_1' 'What is artificial intelligence?'
chr tx -bc ai_inference_dapp -d testnet submit_inference_request 'batch_2' 'Explain machine learning'
chr tx -bc ai_inference_dapp -d testnet submit_inference_request 'batch_3' 'Define neural networks'

# Check results
chr query -bc ai_inference_dapp -d testnet fetch_inference_result id=batch_1
chr query -bc ai_inference_dapp -d testnet fetch_inference_result id=batch_2
chr query -bc ai_inference_dapp -d testnet fetch_inference_result id=batch_3
```

[View this page on docs.chromia.com](https://docs.chromia.com/ecosystem/extensions/ai_inference/)Content mirrored from the live docs
