AI Inference extension

The AI Inference extension brings AI inference capabilities directly into dapps, enabling intelligent, real-time decision-making within the on-chain environment. By leveraging externally trained models, developers can seamlessly embed AI-driven logic into their applications while maintaining blockchain's core principles of trustlessness and transparency. This integration unlocks new possibilities for responsive and adaptive dapps.

This extension uses the Hybrid Compute framework.

AI inference extension repository.

Custom subnode image

Ensure you pick the AI extension image when leasing your container.

Blockchain configuration

You will need to enable the Hybrid Compute framework, configure it to use the AI inference engine provided by this extension, set the appropriate timeout for inference computations and configure the model to use for inference.

chromia.yml

blockchains:
  ai_inference_dapp: # Ensure this name is unique. Rename it if you encounter duplicate issues during deployment.
    module: main
    config:
      sync_ext:
        - "net.postchain.hybridcompute.HybridComputeSynchronizationInfrastructureExtension"
      gtx:
        modules:
          - "net.postchain.hybridcompute.HybridComputeGTXModule"
      hybridcompute:
        engine: "net.postchain.gtx.extensions.ai_inference.AiInferenceComputeEngine"
      ai_inference:
        model: "your-model-name" # Specify model name here. Example: HuggingFaceTB/SmolLM2-1.7B-Instruct
        timeout_seconds: 600 # Timeout in seconds for each inference and validation
        max_completion_tokens: 100 # An upper bound for the number of tokens that can be generated for a completion

Configuration parameters

Parameter	Description	Default	Required
model	Name of the AI model to use for inference	-	Yes
timeout_seconds	Timeout in seconds for each inference and validation	600	No
max_completion_tokens	Upper bound for the number of tokens that can be generated for a completion, including visible output tokens and reasoning tokens	100	No

Integrating with Rell

Install the Rell libraries:

chromia.yml

libs:
  hybridcompute:
    registry: https://gitlab.com/chromaway/postchain-chromia
    path: chromia-infrastructure/rell/src/lib/hybridcompute
    tagOrBranch: 3.31.1
    rid: x"7F4921AB6D1D7FB4CE1B415A011CB8327C39D9F6BFDA95FCDCF31954BECCA4EB"
    insecure: false
  ai_inference:
    registry: https://gitlab.com/chromaway/core/ai-inference-extension
    path: rell/src/lib/ai_inference
    tagOrBranch: 0.2.0
    rid: x"0AA58FFDD9EB238AD1CC60A16798E0E7D57DA883C50BD4EF35F9F0BE31CF0A51"
    insecure: false
compile:
  rellVersion: 0.14.11

Import the module:

import ai: lib.ai_inference;

Submit a simple inference request with this function:

/**
 * Submits an inference request.
 *
 * @param id A unique identifier for the inference request.
 * @param prompt The prompt to generate text for.
 */
function submit_inference_request(id: text, prompt: text)

Or submit a chat inference request with this function:

/**
 * Submits an chat inference request.
 *
 * @param id A unique identifier for the inference request.
 * @param messages A list of messages comprising the conversation so far.
 */
function submit_chat_inference_request(id: text, messages: list<chat_message>)

Fetch the result with this function:

struct inference_result {
    /** The result of the inference, or null if an error occurred. */
    result: text?;

    /** An error message if the inference failed, or null if no error occurred. */
    error: text?;

    /** RID of the transaction where the result was reported. */
    tx_rid: byte_array;

    /** Index of the operation where the result was reported. */
    op_index: integer;
}

/**
 * Fetches the result of a previously submitted inference request.
 *
 * @param id The identifier of the inference request for which the result is being fetched.
 * @return The result, or `null` if not ready yet
 */
function fetch_inference_result(id: text): inference_result?

Or extend this to be notified:

/**
 * Called when an inference is finished, successfully or failed.
 *
 * @param id The identifier of the inference request
 * @param inference_result The result
 */
@extendable function on_inference_result(id: text, inference_result)

Minimal implementation example

Here's a minimal implementation example in Rell:

main.rell

module;

import ai: lib.ai_inference;

// TODO should have authentication for this operation
operation submit_inference_request(id: text, prompt: text) {
    ai.submit_inference_request(id, prompt);
}

// TODO should have authentication for this operation
operation submit_chat_inference_request(id: text, prompt: text) {
    ai.submit_chat_inference_request(id, [ai.chat_message(role="user", message=prompt)]);
}

query fetch_inference_result(id: text): ai.inference_result? = ai.fetch_inference_result(id);

Advanced usage examples

Chat-based inference

chat_inference.rell

operation submit_conversation(conversation_id: text, user_message: text) {
    // For a simple chat, create a conversation with system prompt and user message
    val messages = [
        ai.chat_message(role="system", message="You are a helpful assistant."),
        ai.chat_message(role="user", message=user_message)
    ];

    ai.submit_chat_inference_request(conversation_id, messages);
}

operation continue_conversation(conversation_id: text, conversation_history: list<ai.chat_message>, user_message: text) {
    // Add the new user message to the conversation history
    val updated_messages = conversation_history + [ai.chat_message(role="user", message=user_message)];

    ai.submit_chat_inference_request(conversation_id, updated_messages);
}

Batch processing

batch_inference.rell

operation submit_batch_inference(base_id: text, prompts: list<text>) {
    var index = 0;
    for (prompt in prompts) {
        val request_id = "%s_%d".format(base_id, index);
        ai.submit_inference_request(request_id, prompt);
        index += 1;
    }
}

query get_batch_results(base_id: text, count: integer): list<ai.inference_result?> {
    val results = list<ai.inference_result?>();
    var index = 0;
    while (index < count) {
        val request_id = "%s_%d".format(base_id, index);
        results.add(ai.fetch_inference_result(request_id));
        index += 1;
    }
    return results;
}

Conditional AI processing

conditional_ai.rell

operation smart_content_filter(content_id: text, content: text) {
    // Only process content that meets certain criteria
    if (content.size() > 10 and content.size() < 1000) {
        val prompt = "Analyze the following content for appropriateness: %s".format(content);
        ai.submit_inference_request(content_id, prompt);
    }
}

@extend(ai.on_inference_result)
function (id: text, result: ai.inference_result) {
    if (result.result != null) {
        // Parse AI response and take action
        if (result.result.contains("inappropriate")) {
            // Flag content for review
            flag_content_for_review(id, result.tx_rid, result.op_index);
        } else {
            // Approve content
            approve_content(id, result.tx_rid, result.op_index);
        }
    }
}

Deployment to Testnet

Leasing a container

If you are unfamiliar with leasing a container, follow these steps.

info

Ensure you select the AI Inference extension when leasing the container. The AI Inference Extension is part of Chromia's extension ecosystem and may require cluster-level approval depending on your deployment target.

Make sure to pick the AI extension image when leasing your container to ensure proper support for AI inference operations.

Deployment

Follow the deployment steps.

You can find the BRID for the Testnet Directory Chain in the explorer.

The deployment section in your chromia.yml will look like this:

deployments:
testnet: # Deployment Target name
  brid: x"6F1B061C633A992BF195850BF5AA1B6F887AEE01BB3F51251C230930FB792A92" # Blockchain RID for Testnet Directory Chain
  url: https://node0.testnet.chromia.com # Target URL for one of the nodes in Testnet
  container: 4d7890243fe710...08c724700cbd385ecd17d6f # Replace with your container ID (Example: container: 15ddfcb25dcb43577ab311fe78aedab14fda25757c72a787420454728fb80304)
  chains:
    ai_inference_dapp: x"9716FBFF3...663FEC673710" # Replace with the actual BRID after the dapp gets deployed. You can find it in the terminal during deployment.

Testing and examples

Simple inference test

In the examples provided, ai_inference_dapp and testnet refer to the deployment parameters.

To send a simple inference request, use:

chr tx -bc ai_inference_dapp -d testnet submit_inference_request 'a1' 'What is artificial intelligence?'

To get a response, execute:

chr query -bc ai_inference_dapp -d testnet fetch_inference_result id=a1

Output:

[
  "error": null,
  "result": "Artificial intelligence (AI) refers to the simulation of human intelligence in machines...",
  "tx_rid": "1234567890ABCDEF...",
  "op_index": 0
]

Chat inference test

To send a chat inference request, use:

chr tx -bc ai_inference_dapp -d testnet submit_chat_inference_request 'chat1' 'Hello, how are you?'

Check inference status

chr query -bc ai_inference_dapp -d testnet get_inference_status id=a1

Batch testing

You can access the source code and additional information about the AI Inference extension in the official repository.

# Submit multiple requests
chr tx -bc ai_inference_dapp -d testnet submit_inference_request 'batch_1' 'What is artificial intelligence?'
chr tx -bc ai_inference_dapp -d testnet submit_inference_request 'batch_2' 'Explain machine learning'
chr tx -bc ai_inference_dapp -d testnet submit_inference_request 'batch_3' 'Define neural networks'

# Check results
chr query -bc ai_inference_dapp -d testnet fetch_inference_result id=batch_1
chr query -bc ai_inference_dapp -d testnet fetch_inference_result id=batch_2
chr query -bc ai_inference_dapp -d testnet fetch_inference_result id=batch_3