Create a Managed Inference Job in the web interface

Create a Managed Inference Job in the CosmicAC web interface. The form has six sections, and Continue takes you from one to the next. For a description of every field, see the Job configuration reference.

CosmicAC recommends serving parameters, environment variables, and hardware for each supported model. See Recommended model parameters.

Prerequisites

You need the following before you start:

A running CosmicAC deployment. See Installation.
Access to the CosmicAC web interface.

Steps

Open the new job form

In the left navigation, click Jobs, then click New Job.

The Models page offers a shortcut. Deploy model opens the same form with Managed Inference already selected, so you can skip the next step.

Select the job type

In the What kind of job? section, select Managed Inference, then click Continue.

Enter the basics

In the Basics section, enter a Job name, choose a Location, and add Tags. To add a tag, type it, then press Enter or comma.

A job name needs at least three characters, and can contain only lowercase letters, numbers, and dashes.

Select a model

In the Model to serve section, select a Model. CosmicAC prefills the configuration from the model master stored for that model. The fields depend on the model's runtime.

For a vLLM model, the form shows a Serving configuration:

Runtime image (CUDA): the vLLM serving image.
Data type: the numeric precision the model runs at.
Quantisation: how to compress the model weights.
Tensor parallel: how many GPUs to split the model across.
GPU memory utilisation: the fraction of GPU memory to use, with 85%, 90%, and 95% presets.
Max model length: the maximum context length, with 8k, 16k, 32k, and 128k presets.
Max concurrent sequences: the maximum requests handled at once, with 64, 128, 256, and 512 presets.
Reasoning parser: the parser that separates thinking tokens from the final response.
Video & image input: whether the model accepts multimodal input, for vision-language models only.

Apply the Recommended model parameters for supported models. You can edit every field except the runtime image, which comes from the model master. To serve a different image, update the model master.

For nvidia/parakeet-tdt-0.6b-v3, the form shows a Parakeet configuration instead, which covers audio chunking and upload limits:

Chunk duration: seconds of audio per chunk.
Chunk overlap: seconds of overlap between adjacent chunks.
Max file size: maximum upload size in MB.

Set the root disk and environment variables

Under Instance resources, set the Root disk (GB). Pick a 250, 500, or 1000 GB preset, or type a value.

Under Environment variables, review the prefilled variables, then edit them if needed. To add another, click Add variable, then enter a Name and Value. To drop one, click the cross beside it.

The form shows no Instance resources or Environment variables for a Parakeet job, so this step doesn't apply to one.

Name the endpoint and set replicas

In the same Model to serve section, under Endpoint, enter an Endpoint name. It must be globally unique across active inference jobs. Will be reachable at shows the endpoint URL that name produces, with a button to copy it.

Set Replicas to 1, 2, or 4. You can't change the count after you create the job. See Replicas.

A Parakeet job has no replicas, so set only the Endpoint name. It serves /v1/audio/transcriptions. See Audio transcriptions.

Require an API key

Under API key required, select Require Authorization header. See Create an API key.

Choose the hardware

In the Hardware section, select a GPU and the GPU count, then set the CUDA / driver. The section lists the GPUs available in the location you chose, and shows how many are free.

Choose the notifications

In the Notifications section, turn on each job lifecycle event you want this job to report. CosmicAC turns all four on by default.

job.failed: the job transitions to Failed. Includes the failure reason.
job.degraded: healthy replicas drop below desired. The endpoint stays live.
job.recovered: the job returns to Active from Degraded or Failed.
job.restart_storm: any replica restarts three times within 10 minutes.

These preferences cover this job alone. An event you turn on here reaches your webhook only if it's also turned on in Settings → Notifications, which is also where you manage the model health events. See Set up webhook notifications.

Review and create the job

In the Review & launch section, confirm the section shows Ready to create, then click Create job. If it reports issues instead, click Edit on the row that names the problem, fix it, then come back to this section.

A configuration that departs from the model's recommended specs raises a Job creation warning, which names the values that differ. Create job stays disabled until you select Create anyway with this configuration. A job created that way can still fail to start.

Open the endpoint

Wait for the job to start running, then click Open endpoint. To send it a request, see Connect to a vLLM Managed Inference endpoint, or Transcribe audio with a Parakeet endpoint for a Parakeet job.