Simple vLLM

Note that some models may require authentication through Hugging Face tokens, which can be done using konduktor secret (see complex example here). The model deepseek-ai/DeepSeek-R1-Distill-Llama-8B does not require one.

Prerequisites

Current Working Directory

$ ls
deployment.yaml

Launching

$ konduktor serve launch deployment.yaml

Deployment.yaml

# no autoscaling + default port (8000) + single GPU
name: serving-vllm-simple

resources:
  cpus: 4
  memory: 32
  accelerators: A100:1
  image_id: vllm/vllm-openai:v0.7.1
  labels:
    kueue.x-k8s.io/queue-name: user-queue

serving: 
  min_replicas: 1

run: |
  python3 -m vllm.entrypoints.openai.api_server \
    --model deepseek-ai/DeepSeek-R1-Distill-Llama-8B \
    --max-model-len 4096

Complex General Complex vLLM

Schemas

Example Task Yamls

Example Deployment Yamls

Prerequisites

Current Working Directory

Launching

Deployment.yaml

​Prerequisites

​Current Working Directory

​Launching

​Deployment.yaml

Prerequisites

Current Working Directory

Launching

Deployment.yaml