vLLM Export#

vLLM is a popular high-throughput and memory-efficient inference and serving engine for Large Language Models (LLMs).

In Brevitas, we provide an export flow that produces vLLM-compatible quantized models directly from the LLM entrypoint. The exported model can then be loaded and served through vLLM with quantization applied transparently.

Note

vLLM is not a dependency of Brevitas and must be installed separately. The vLLM import only occurs when --export-target vllm is specified.

QDQ-based Export#

The current vLLM export flow is based on a Quantize-DeQuantize (QDQ) approach:

  • Weights are stored in their dequantized (full precision) form via model.save_pretrained().

  • All quantization metadata (scales, zero points, bit widths, handler class types, rotation configurations) is serialized into a brevitas_config.json file alongside the model.

  • At load time, vLLM discovers the brevitas_config.json, reconstructs the quantization handlers for each layer, and re-quantizes the weights.

This means the model directory produced by the export is a standard HuggingFace-compatible checkpoint, with the addition of brevitas_config.json that vLLM uses to apply quantization.

A Quantized Operations (QOp) approach, where operations such as matrix multiplication run directly on quantized data using accelerated kernels, is currently a work in progress and is not yet available.

Supported Configurations#

Weight Quantization#

The following weight quantization formats are supported:

  • Integer: int8 (symmetric), uint8 (asymmetric)

  • FP8: e4m3 (both float and float_ocp variants)

  • MX formats: MXInt8, MXFloat8e4m3

Weight quantization granularity options:

  • Per tensor

  • Per channel

  • Per group (groupwise), with configurable group size

Activation Quantization#

The following activation quantization formats are supported:

  • Integer: int8 (symmetric), uint8 (asymmetric)

  • FP8: e4m3 (both float and float_ocp variants)

  • MX formats: MXInt8, MXFloat8e4m3

Activation quantization granularity options:

  • Per tensor (static)

  • Per row (dynamic) – FP8 only

  • Per group (dynamic)

LLM Entrypoint#

Brevitas’ LLM entrypoint allows the user to load, quantize, test, and export many of the LLM available on HuggingFace, by simply passing a series of command line arguments that can control, among other things:

  • Weights and activations bit width

  • Weights and activation quantization format (int vs float, asym vs sym, etc.)

  • PTQ algorithms to apply and their options

  • and much more…

Below are some example configurations for the vLLM export target.

Weight-only INT8 symmetric per-group quantization:

brevitas_ptq_llm --model org/model --weight-bit-width 8 --weight-quant-format int \
  --weight-quant-type sym --weight-quant-granularity per_group --weight-group-size 128 \
  --export-target vllm --export-prefix ./exported_model

Weight-only INT4 symmetric per-group quantization:

brevitas_ptq_llm --model org/model --weight-bit-width 4 --weight-quant-type sym \
  --weight-quant-granularity per_group --weight-group-size 128 \
  --export-target vllm --export-prefix ./exported_model

FP8 weight and dynamic per-row activation quantization:

brevitas_ptq_llm --model org/model --weight-bit-width 8 \
  --weight-quant-format float_ocp_e4m3 --weight-quant-granularity per_channel \
  --input-bit-width 8 --input-quant-format float_ocp_e4m3 \
  --input-quant-granularity per_row --input-scale-type dynamic \
  --act-calibration --export-target vllm --export-prefix ./exported_model

These commands will produce quantized models without any extra pre-processing, but several PTQ algorithms are compatible with this export flow (see below).

The --export-prefix argument specifies the output directory where the model, tokenizer, and brevitas_config.json will be saved.

Compatible PTQ Algorithms#

The following PTQ algorithms are compatible with the vLLM export flow:

  • GPTQ / GPFQ

  • MagR

  • SmoothQuant (fused version only, not layerwise)

  • Weight Equalization

  • QuaRot / SpinQuant (all rotation modes: fx, fused_no_fx, layerwise)

  • Bias Correction

  • Qronos

The following are not currently supported:

  • AWQ

  • Learned Round (support is a work in progress)

  • SVDQuant

Export Output#

When the export completes, the output directory (specified by --export-prefix) will contain:

  • Model weights: saved via model.save_pretrained() in standard HuggingFace format.

  • Tokenizer files: saved via tokenizer.save_pretrained().

  • brevitas_config.json: a JSON file containing per-layer quantization metadata, including:

    • Quantization handler class type per proxy (weight, input, output, bias)

    • Scale and zero point values

    • Bit width, float-to-int implementation type, and scaling restriction settings

    • Rotation configurations (Hadamard matrix shape and k value) for layers wrapped in RotatedModule

At load time in vLLM, the registered quant_brevitas quantization method discovers brevitas_config.json and assigns a QuantLinear method to each linear layer. This method reconstructs the quantization handlers from the saved configuration and applies quantization during inference.

Using the Exported Model in vLLM#

Once the model has been exported, it can be loaded in vLLM like any other quantized model by pointing to the export directory:

from vllm import LLM

llm = LLM(model="./exported_model", quantization="quant_brevitas")

Or via the vLLM CLI:

vllm serve ./exported_model --quantization quant_brevitas

FAQ#

  • What is the difference between QDQ and QOp?

QDQ (Quantize-DeQuantize) exports dequantized weights alongside quantization metadata. At load time, vLLM reconstructs the quantization and re-quantizes the weights. QOp (Quantized Operations) operates directly on quantized tensors using accelerated kernels (e.g., MXFP4 via aiter), avoiding the dequantize-requantize overhead. Currently only the QDQ approach is available; QOp is a work in progress.

  • Can I use custom quantizers with the vLLM export?

Yes, the --custom-quantizer flag is compatible with the vLLM export path, as long as the underlying quantization format maps to one of the supported inference handlers.

  • Why do I get an import error for vLLM?

vLLM is not bundled with Brevitas and must be installed separately in your environment. The vLLM-specific code is only imported when --export-target vllm is specified, so vLLM is not required for other Brevitas workflows.

On this page
Show Source