Introduction

SciLifeLab Serve allows hosting of web applications with models that require access to a GPU to make inferences. We have a number of GPUs available on our infrastructure for this purpose. This cannot be done from the web interface independently by the users, one of the team members needs to enable this for your project. If your app needs a GPU, send us an email (serve@scilifelab.se) with a motivated request (including how exactly your app benefits from having access to a GPU, see FAQ below). After we add a corresponding option to your project on SciLifeLab Serve, you will be able to launch your app with a GPU. We do not promise that we will be able to accommodate every request but we will do our best.

You can build an application using any framework that will allow you to package it as a Docker image and fulfill other requirements of SciLifeLab Serve. For example, in the past we had users with Gradio, Streamlit, Dash applications that made use of GPUs. Follow the corresponding user guide in this documentation (see the menu to the left) to build your application and prepare your Docker image. On this page, you will find additional information regarding GPU support.

Packaging your application

We have NVIDIA A40 GPUs that are currently running with: Driver Version: 580.178.04, CUDA Version: 13.0. This means that apps with packages requiring CUDA versions between 11.8 (including) and 13.0 (including) should work.

Package your application as a Docker image using the documentation we have for your specific app type or for the general other app types category. You will in addition need to make sure that:

• Your code detects and uses torch.device('cuda') when available.
• Your dependencies (e.g., PyTorch) are installed with GPU support.
• We do not recommend including model weights in the image (unless they are smaller than approximately 200-300 MB) or downloading them at startup; instead we recommend uploading the model weights separately through the File Manager in your SciLifeLab Serve project and attaching that folder to the app (see mount path option in app settings).

Please note that we allow multiple applications to run on a single GPU using time-slicing. From your perspective as an application developer you do not need to manage GPU sharing yourself. Create your application assuming it will run in an environment with a GPU. However, GPU compute time and memory are shared between applications. This means that GPU performance may vary depending on other workloads, and your application should not assume exclusive access to GPU resources. In particular, ensure that your application handles GPU memory limitations appropriately. In most cases, if you are using standard packages, this should be handled for you, and you do not need to do anything special.

Example application

Below are example files for an app with a model that requires access to a GPU. These example files follow the Custom App type (other apps category) requirements (this could have also been deployed as a Gradio app but the Dockerfile would be slightly different there, see the documentation page on Gradio). Feel free to get in touch with us if you run into issues while preparing your app for SciLifeLab Serve.

Example Dockerfile for GPU-enabled apps:

# GPU-ready PyTorch base image (includes torch with CUDA 12.4)
FROM pytorch/pytorch:2.6.0-cuda12.4-cudnn9-runtime

# Create user and home directory
ENV USER=username
ENV HOME=/home/$USER
RUN useradd -m -u 1000 $USER

# Show Python output in the app logs immediately
ENV PYTHONUNBUFFERED=1

# Where the app looks for the model weights. This folder is the
# mount path of the project storage, so the weights are not in the image.
ENV MODEL_DIR=$HOME/app/models
ENV MODEL_FILE=resnet18.pth

# Set working directory
WORKDIR $HOME/app

# Install Python dependencies (torch is already in the base image)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY main.py start-script.sh ./

# Make script executable and fix ownership
RUN chmod +x start-script.sh && chown -R $USER:$USER $HOME

# Switch to non-root user
USER $USER

# Port the Gradio app listens on
EXPOSE 7860

# Start the app
ENTRYPOINT ["./start-script.sh"]

In this example model weights are stored in the the folder /home/app/models and the weights file is resnet18.pth. This should be uploaded separately through the File Manager.

If your app needs system packages, install them with apt-get before the Python dependencies. Only add the packages you actually need, as each one increases the image size.

Example requirements.txt:

# Do not list torch here: it is already included in the base image,
# and reinstalling it from PyPI can replace it with an incompatible build.
torchvision==0.21.0  # matches torch 2.6.0 in the base image
gradio

We recommend pinning all packages to the versions you have tested with (for example gradio==<version>), so that rebuilding the image later does not change your app's behaviour.

Example main.py:

import logging
import os
import sys
from pathlib import Path

import torch
import gradio as gr
from torchvision.models import resnet18, ResNet18_Weights

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger(__name__)

# --- GPU check at startup ---------------------------------------------------
# Without this check, an app can silently fall back to the CPU on Serve.
# Look for these lines in your app's logs after it starts.
cuda_available = torch.cuda.is_available()
log.info("PyTorch version: %s (built with CUDA %s)", torch.__version__, torch.version.cuda)
log.info("torch.cuda.is_available(): %s", cuda_available)
if cuda_available:
    log.info("torch.cuda.get_device_name(): %s", torch.cuda.get_device_name(0))
    device = torch.device("cuda")
else:
    log.warning("No GPU detected, running on CPU. Inference will be much slower.")
    device = torch.device("cpu")

# --- Load the model weights from the project file storage ------------------------
model_dir = Path(os.environ.get("MODEL_DIR", Path.home() / "models"))
model_path = model_dir / os.environ.get("MODEL_FILE", "resnet18.pth")

if not model_path.is_file():
    log.error(
        "Model weights not found at %s. Upload the file with the File Manager "
        "and check that the app's Storage field uses the mount path %s.",
        model_path,
        model_dir,
    )
    sys.exit(1)

log.info("Loading model weights from %s", model_path)
model = resnet18(weights=None)
model.load_state_dict(torch.load(model_path, map_location=device, weights_only=True))
model = model.eval().to(device)

# Preprocessing and labels that match the pretrained ResNet-18 weights
# (these are defined in torchvision and do not download anything)
preprocess = ResNet18_Weights.DEFAULT.transforms()
labels = ResNet18_Weights.DEFAULT.meta["categories"]


def predict(image):
    if image is None:
        raise gr.Error("Please upload an image.")

    batch = preprocess(image.convert("RGB")).unsqueeze(0).to(device)

    try:
        with torch.inference_mode():
            probs = torch.nn.functional.softmax(model(batch)[0], dim=0)
    except torch.cuda.OutOfMemoryError:
        # The GPU is shared with other apps, so memory can run out.
        # Free cached memory and show a friendly message instead of crashing.
        torch.cuda.empty_cache()
        raise gr.Error("The GPU is busy right now. Please try again in a moment.")

    probs = probs.cpu()
    return {labels[i]: float(probs[i]) for i in range(len(labels))}


demo = gr.Interface(
    fn=predict,
    inputs=gr.Image(type="pil"),
    outputs=gr.Label(num_top_classes=3),
)

# Process one request at a time on the GPU to limit memory use
demo.queue(default_concurrency_limit=1)
demo.launch(server_name="0.0.0.0", server_port=7860)

Example start-script.sh:

#!/bin/bash
set -e
exec python main.py

The Dockerfile above makes the script executable. If you also run it outside Docker, make it executable with:

chmod +x start-script.sh

Checking that your app uses the GPU

When the app starts, main.py logs whether PyTorch can see a GPU. Open your app's logs on Serve (click the three dots next to your app and choose "Logs") and look for lines like these:

INFO torch.cuda.is_available(): True
INFO torch.cuda.get_device_name(): NVIDIA A40

If the log says torch.cuda.is_available(): False instead, the app is running on the CPU. Common causes are:

• GPU access not enabled. You launched your app with a hardware allocation that did not include a GPU. Check the Hardware setting for your app in the SciLifeLab Serve user interface.
• CPU-only PyTorch. PyTorch in your image was installed without GPU support, for example because torch was listed in requirements.txt and replaced the GPU build from the base image.
• Unsupported CUDA version. The CUDA version in your image is not supported (see above).

Testing your image locally

First, build the Docker image with your app (see our other documentation pages for more on this if you are unsure):

docker build --platform linux/amd64 -t my-gpu-app .

To test locally, put resnet18.pth in a folder called models next to your Dockerfile and mount that folder at the same path the app uses on Serve. Specifically, if you have access to a computer with an NVIDIA GPU and the NVIDIA Container Toolkit installed, run the app with GPU access:

docker run --rm --gpus all -p 7860:7860 \
  -v "$(pwd)/models:/home/username/models" \
  my-gpu-app

Without a GPU, you can run the same command without --gpus all to check that the app starts and works on the CPU. The log will then show torch.cuda.is_available(): False, which is expected locally but not on Serve.

Then open http://localhost:7860 in your browser and check the startup logs as described above.

Frequently Asked Questions

I am stuck while following the guide. Can I get help?

Yes, feel free to get in touch with us (serve@scilifelab.se). Please provide a link to your code (for example, to your GitHub repository) and model in your email so that we can best help you.

In which case can I ask for a GPU for my application?

We aim to allocate GPUs to the applications and models that genuinely require it to run or to run fast enough. For example, if your application can run and provide a reasonably fast response (for example, within a minute) to most users' requests on CPUs, you are probably best off working with CPUs. We can allocate up to 12 vCPUs to each app. On the other hand, if your application takes many minutes to give a response to a user with CPUs but does so much faster with GPUs, you probably have a good reason to ask for access to a GPU.

Can I test my app with your GPUs before publishing?

Yes, we recommend that you launch your application with visibility set to Private/Project/Link before you make it public. This will allow you to see the logs (click the three dots for your app on the project overview page and click on Logs) and test the app behavior (go to your chosen URL and test it).