为什么Mac用户选Ollama不选vLLM?
P
digoal约 30 分钟
digoal约 30 分钟为什么Mac用户选Ollama不选vLLM?
用macOS apple arm chip使用vLLM, 不如Ollama方便, 性能有BUG? 实测下来真的无法使用, 不知道是不是哪里不对? 按vLLM自己官网说的, 现在mac chip是实验性质支持.
制作Mac apple arm chip vLLM容器
https://docs.vllm.ai/en/stable/getting_started/installation/cpu/index.html
1、下载ubuntu:22.04镜像
docker pull ubuntu:22.04
可能需要科学上网
或者你也可以从其他地方下载ubuntu:22.04镜像, 在下面的Dockerfile中把FROM改成你下载的即可.
2、克隆vllm开源项目
cd ~/Downloads
git clone --depth 1 https://github.com/vllm-project/vllm
3、由于vllm官方docker只提供了x86版本, 所以需要自己build for apple chip的镜像.
修改Dockerfile.arm
cd ~/Downloads/vllm
vi Dockerfile.arm
# 修改如下, 增加几行即可, 主要是加速和解决访问huggingface.co问题.
...
ENV CMAKE_CXX_COMPILER_LAUNCHER=ccache
# 加1行
+ RUN sed -i 's|http://ports.ubuntu.com|http://mirrors.aliyun.com|g' /etc/apt/sources.list
RUN --mount=type=cache,target=/var/cache/apt \
...
# tcmalloc provides better memory allocation efficiency, e.g., holding memory in caches to speed up access of commonly-used objects.
# 加2行
+ RUN pip config set global.index-url https://mirrors.aliyun.com/pypi/simple
+ RUN pip config set install.trusted-host mirrors.aliyun.com
RUN --mount=type=cache,target=/root/.cache/pip \
pip install py-cpuinfo # Use this to gather CPU info and optimize based on ARM Neoverse cores
...
# 加2行
# 假如宿主机地址192.168.64.1 , 并已按../202407/20240704_01.md配置代理, 解决启动容器后 huggingface.co 无法访问的报错.
# 如果你在宿主机提前下载好huggingface.co里的模型, 并使用volume map到容器内, 应该可以不需要配置proxy.
+ ENV http_proxy="http://192.168.64.1:22222"
+ ENV https_proxy="http://192.168.64.1:22222"
WORKDIR /workspace/
RUN ln -s /workspace/vllm/tests && ln -s /workspace/vllm/examples && ln -s /workspace/vllm/benchmarks
ENTRYPOINT ["python3", "-m", "vllm.entrypoints.openai.api_server"]
build
docker build -f Dockerfile.arm --no-cache -t vllm-apple-arm-env --shm-size=4g .
从huggingface.co下载模型
注意, 目前vLLM 支持 macOS apple silicon芯片为实验特性, 并且赞助仅支持FP32 and FP16 datatypes (Currently the CPU implementation for macOS supports FP32 and FP16 datatypes.). 如果模型不是fp32/fp16的, 需要转换, 否则使用时会报错.
ERROR 02-22 03:00:34 engine.py:140] RuntimeError: "rms_norm_impl" not implemented for'BFloat16'
https://docs.vllm.ai/en/stable/getting_started/installation/cpu/index.html
1、可以使用cli下载模型, 也可以直接从huggingface.co网站下载模型
https://huggingface.co/docs/huggingface_hub/main/en/guides/cli
https://huggingface.co/models
首先升级python到3.12.x , 可参考 《mac + mlx-examples 大模型微调(fine-tuning)实践 - 让它学完所有PolarDB文章》
然后安装huggingface cli
python -m pip install --upgrade pip
pip config set global.index-url https://mirrors.aliyun.com/pypi/simple
pip config set install.trusted-host mirrors.aliyun.com
pip install -U "huggingface_hub[cli]"
配置宿主机目录, 将模型下载到该目录中
M_DIR="$HOME/Downloads/model_from_huggingface"
mkdir $M_DIR
使用镜像加速, 下载Qwen2.5-1.5B
M_DIR="$HOME/Downloads/model_from_huggingface"
MODEL="Qwen/Qwen2.5-1.5B"
HF_ENDPOINT=https://hf-mirror.com nohup huggingface-cli download $MODEL --local-dir $M_DIR/$MODEL --cache-dir $M_DIR/.cache --resume-download --max-workers 4 >/dev/null 2>&1 &
2、可以使用llama.cpp进行转换, 参考:
《手把手教你炼丹 | 用Mac本地微调大模型 , 扩展: 微调数据JSONL的内容格式和什么有关?》
下载 llama
cd ~
git clone --depth 1 [email protected]:ggerganov/llama.cpp.git
# 或 git clone --depth 1 https://github.com/ggerganov/llama.cpp
编译llama
cd llama.cpp
mkdir build
cd build
cmake ..
make -j 8
转换帮助
cd ~/llama.cpp
$ python3 convert_hf_to_gguf.py -h
usage: convert_hf_to_gguf.py [-h] [--vocab-only] [--outfile OUTFILE] [--outtype {f32,f16,bf16,q8_0,tq1_0,tq2_0,auto}] [--bigendian] [--use-temp-file] [--no-lazy]
[--model-name MODEL_NAME] [--verbose] [--split-max-tensors SPLIT_MAX_TENSORS] [--split-max-size SPLIT_MAX_SIZE] [--dry-run] [--no-tensor-first-split]
[--metadata METADATA] [--print-supported-models]
[model]
Convert a huggingface model to a GGML compatible file
positional arguments:
model directory containing model file
options:
-h, --help show this help message and exit
--vocab-only extract only the vocab
--outfile OUTFILE path to write to; default: based on input. {ftype} will be replaced by the outtype.
--outtype {f32,f16,bf16,q8_0,tq1_0,tq2_0,auto}
output format - use f32 for float32, f16 for float16, bf16 for bfloat16, q8_0 for Q8_0, tq1_0 or tq2_0 for ternary, and auto for the highest-fidelity
16-bit floattype depending on the first loaded tensor type
--bigendian model is executed on big endian machine
--use-temp-file use the tempfile library while processing (helpful when running out of memory, process killed)
--no-lazy use more RAM by computing all outputs before writing (use incase lazy evaluation is broken)
--model-name MODEL_NAME
name of the model
--verbose increase output verbosity
--split-max-tensors SPLIT_MAX_TENSORS
max tensors in each split
--split-max-size SPLIT_MAX_SIZE
max size per split N(M|G)
--dry-run only print out a split plan and exit, without writing any new files
--no-tensor-first-split
do not add tensors to the first split (disabled by default)
--metadata METADATA Specify the path for an authorship metadata override file
--print-supported-models
Print the supported models
转换
cd ~/llama.cpp
M_DIR="$HOME/Downloads/model_from_huggingface"
MODEL="Qwen/Qwen2.5-1.5B"
python3 convert_hf_to_gguf.py --outfile $M_DIR/${MODEL}-fp16 --outtype f16 --model-name qwen2.5-1.5b-fp16 $M_DIR/$MODEL
INFO:hf-to-gguf:Set model quantization version
INFO:gguf.gguf_writer:Writing the following files:
INFO:gguf.gguf_writer:/Users/digoal/Downloads/model_from_huggingface/Qwen/Qwen2.5-1.5B-fp16: n_tensors = 338, total_size = 3.1G
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
Writing: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.09G/3.09G [00:10<00:00, 300Mbyte/s]
INFO:hf-to-gguf:Model successfully exported to /Users/digoal/Downloads/model_from_huggingface/Qwen/Qwen2.5-1.5B-fp16
启动vLLM容器, 加载宿主机下载好的模型
1、使用build好的镜像vllm-apple-arm-env启动容器, 并加载宿主机下载好的Qwen/Qwen2.5-1.5B模型
https://docs.vllm.ai/en/stable/deployment/docker.html
M_DIR="$HOME/Downloads/model_from_huggingface"
MODEL="Qwen/Qwen2.5-1.5B"
# 注意, --model /data/$MODEL 必须放到命令最后, 是ENTRYPOINT接收的参数, 不是docker run的参数.
docker run -d -it --name vllm -v $M_DIR:/data -p 8001:8000 vllm-apple-arm-env --model /data/$MODEL
2、查看vLLM容器日志:
docker logs vllm
3、使用curl查看当前vLLM容器支持的模型
curl http://localhost:8001/v1/models
{"object":"list","data":[{"id":"/data/Qwen/Qwen2.5-1.5B","object":"model","created":1740193123,"owned_by":"vllm","root":"/data/Qwen/Qwen2.5-1.5B","parent":null,"max_model_len":131072,"permission":[{"id":"modelperm-a1ebaaadb8ab45588c62f23ff944bd9d","object":"model_permission","created":1740193123,"allow_create_engine":false,"allow_sampling":true,"allow_logprobs":true,"allow_search_indices":false,"allow_view":true,"allow_fine_tuning":false,"organization":"*","group":null,"is_blocking":false}]}]}
chat API例子
curl http://localhost:8001/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "/data/Qwen/Qwen2.5-1.5B",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "你好"}
]
}'
容器logs 查看到报错, 原因之前说了
ERROR 02-22 03:00:34 engine.py:140] RuntimeError: "rms_norm_impl" not implemented for'BFloat16'
使用转换为fp16后的模型
启动容器
M_DIR="$HOME/Downloads/model_from_huggingface"
MODEL="Qwen/Qwen2.5-1.5B-fp16"
# 注意, --model /data/$MODEL 必须放到命令最后, 是ENTRYPOINT接收的参数, 不是docker run的参数.
docker run -d -it --name vllm -v $M_DIR:/data -p 8001:8000 vllm-apple-arm-env --model /data/$MODEL
curl http://localhost:8001/v1/models
{"object":"list","data":[{"id":"/data/Qwen/Qwen2.5-1.5B-fp16","object":"model","created":1740195680,"owned_by":"vllm","root":"/data/Qwen/Qwen2.5-1.5B-fp16","parent":null,"max_model_len":131072,"permission":[{"id":"modelperm-05c0e9e610844e3db9e96054b5ef62d2","object":"model_permission","created":1740195680,"allow_create_engine":false,"allow_sampling":true,"allow_logprobs":true,"allow_search_indices":false,"allow_view":true,"allow_fine_tuning":false,"organization":"*","group":null,"is_blocking":false}]}]}
curl http://localhost:8001/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "/data/Qwen/Qwen2.5-1.5B-fp16",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "你好"}
]
}'
这个卡住了, 不知道为什么, CPU耗费巨大, 没有使用gpu, 是不是在Mac上用docker没有把gpu分配给docker容器? 跑Janus时也遇到类似问题, 《DeepSeek多模态大模型Janus的使用 (文本生成 , 图像生成 , 图像分析), 卡住, 不如DiffusionBee》
日志如下
INFO 02-22 03:38:04 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
INFO 02-22 03:38:09 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.4%, CPU KV cache usage: 0.0%.
更多API例子参考:
https://docs.vllm.ai/en/latest/getting_started/quickstart.html
直接在macOS上编译使用vllm, 也是只见cpu转, 不见结果
直接在macOS编译vllm
cd ~/Downloads
git clone --depth 1 https://github.com/vllm-project/vllm.git
cd vllm/
pip install -r requirements-cpu.txt
pip install -e .
启动vllm 兼容openAI API服务
vllm serve $HOME/Downloads/model_from_huggingface/Qwen/Qwen2.5-1.5B
日志
...
INFO 02-22 15:38:52 api_server.py:208] Started engine process with PID 57803
INFO 02-22 15:38:52 config.py:2423] For macOS with Apple Silicon, currently bfloat16 is not supported. Setting dtype to float16.
WARNING 02-22 15:38:52 config.py:2454] Casting torch.bfloat16 to torch.float16.
INFO 02-22 15:38:54 __init__.py:207] Automatically detected platform cpu.
INFO 02-22 15:38:54 config.py:2423] For macOS with Apple Silicon, currently bfloat16 is not supported. Setting dtype to float16.
WARNING 02-22 15:38:54 config.py:2454] Casting torch.bfloat16 to torch.float16.
...
INFO: Started server process [57786]
INFO: Waiting for application startup.
INFO: Application startup complete.
调用api, 依旧只见cpu狂转, 未输出结果
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "/Users/digoal/Downloads/model_from_huggingface/Qwen/Qwen2.5-1.5B",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "你好"}
]
}'
vllm 端日志
INFO 02-22 15:29:09 launcher.py:31] Route: /invocations, Methods: POST
INFO: Started server process [55443]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: 127.0.0.1:57447 - "GET /v1/models HTTP/1.1" 200 OK
INFO: 127.0.0.1:57515 - "GET /v1/chat/completions HTTP/1.1" 405 Method Not Allowed
INFO 02-22 15:30:24 chat_utils.py:332] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
INFO 02-22 15:30:24 logger.py:39] Received request chatcmpl-8db6a49a391248679083f60a216af4ed: prompt: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n你好<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=131052, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO 02-22 15:30:24 engine.py:280] Added request chatcmpl-8db6a49a391248679083f60a216af4ed.
WARNING 02-22 15:30:25 cpu.py:143] Pin memory is not supported on CPU.
INFO 02-22 15:30:25 metrics.py:455] Avg prompt throughput: 3.7 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 02-22 15:30:30 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
附 vllm help
vllm -h
INFO 02-22 15:18:51 __init__.py:207] Automatically detected platform cpu.
usage: vllm [-h] [-v] {chat,complete,serve} ...
vLLM CLI
positional arguments:
{chat,complete,serve}
chat Generate chat completions via the running API server
complete Generate text completions based on the given prompt via the running API server
serve Start the vLLM OpenAI Compatible API server
options:
-h, --help show this help message and exit
-v, --version show program's version number and exit
$ vllm serve -h
INFO 02-22 15:37:44 __init__.py:207] Automatically detected platform cpu.
usage: vllm serve <model_tag> [options]
positional arguments:
model_tag The model tag to serve
options:
--additional-config ADDITIONAL_CONFIG
Additional config for specified platform in JSON format. Different platforms may support different configs. Make sure the configs are valid for the platform
you are using. The input format is like '{"config_key":"config_value"}'
--allow-credentials Allow credentials.
--allowed-headers ALLOWED_HEADERS
Allowed headers.
--allowed-local-media-path ALLOWED_LOCAL_MEDIA_PATH
Allowing API requests to read local images or videos from directories specified by the server file system. This is a security risk. Should only be enabled
in trusted environments.
--allowed-methods ALLOWED_METHODS
Allowed methods.
--allowed-origins ALLOWED_ORIGINS
Allowed origins.
--api-key API_KEY If provided, the server will require this key to be presented in the header.
--block-size {8,16,32,64,128}
Token block size for contiguous chunks of tokens. This is ignored on neuron devices and set to ``--max-model-len``. On CUDA devices, only block sizes up to
32 are supported. On HPU devices, block size defaults to 128.
--calculate-kv-scales
This enables dynamic calculation of k_scale and v_scale when kv-cache-dtype is fp8. If calculate-kv-scales is false, the scales will be loaded from the
model checkpoint if available. Otherwise, the scales will default to 1.0.
--chat-template CHAT_TEMPLATE
The file path to the chat template, or the template in single-line form for the specified model.
--chat-template-content-format {auto,string,openai}
The format to render message content within a chat template. * "string" will render the content as a string. Example: ``"Hello World"`` * "openai" will
render the content as a list of dictionaries, similar to OpenAI schema. Example: ``[{"type": "text", "text": "Hello world!"}]``
--code-revision CODE_REVISION
The specific revision to use for the model code on Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the
default version.
--collect-detailed-traces COLLECT_DETAILED_TRACES
Valid choices are model,worker,all. It makes sense to set this only if ``--otlp-traces-endpoint`` is set. If set, it will collect detailed traces for the
specified modules. This involves use of possibly costly and or blocking operations and hence might have a performance impact.
--compilation-config COMPILATION_CONFIG, -O COMPILATION_CONFIG
torch.compile configuration for the model.When it is a number (0, 1, 2, 3), it will be interpreted as the optimization level. NOTE: level 0 is the default
level without any optimization. level 1 and 2 are for internal testing only. level 3 is the recommended level for production. To specify the full
compilation config, use a JSON string. Following the convention of traditional compilers, using -O without space is also supported. -O3 is equivalent to -O
3.
--config CONFIG Read CLI options from a config file.Must be a YAML with the following options:https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#cli-
reference
--config-format {auto,hf,mistral}
The format of the model config to load. * "auto" will try to load the config in hf format if available else it will try to load in mistral format
--cpu-offload-gb CPU_OFFLOAD_GB
The space in GiB to offload to CPU, per GPU. Default is 0, which means no offloading. Intuitively, this argument can be seen as a virtual way to increase
the GPU memory size. For example, if you have one 24 GB GPU and set this to 10, virtually you can think of it as a 34 GB GPU. Then you can load a 13B model
with BF16 weight, which requires at least 26GB GPU memory. Note that this requires fast CPU-GPU interconnect, as part of the model is loaded from CPU memory
to GPU memory on the fly in each model forward pass.
--device {auto,cuda,neuron,cpu,openvino,tpu,xpu,hpu}
Device type for vLLM execution.
--disable-async-output-proc
Disable async output processing. This may result in lower performance.
--disable-custom-all-reduce
See ParallelConfig.
--disable-fastapi-docs
Disable FastAPI's OpenAPI schema, Swagger UI, and ReDoc endpoint.
--disable-frontend-multiprocessing
If specified, will run the OpenAI frontend server in the same process as the model serving engine.
--disable-log-requests
Disable logging requests.
--disable-log-stats Disable logging statistics.
--disable-logprobs-during-spec-decoding [DISABLE_LOGPROBS_DURING_SPEC_DECODING]
If set to True, token log probabilities are not returned during speculative decoding. If set to False, log probabilities are returned according to the
settings in SamplingParams. If not specified, it defaults to True. Disabling log probabilities during speculative decoding reduces latency by skipping
logprob calculation in proposal sampling, target sampling, and after accepted tokens are determined.
--disable-mm-preprocessor-cache
If true, then disables caching of the multi-modal preprocessor/mapper. (not recommended)
--disable-sliding-window
Disables sliding window, capping to sliding window size.
--distributed-executor-backend {ray,mp,uni,external_launcher}
Backend to use for distributed model workers, either "ray" or "mp" (multiprocessing). If the product of pipeline_parallel_size and tensor_parallel_size is
less than or equal to the number of GPUs available, "mp" will be used to keep processing on a single host. Otherwise, this will default to "ray"if Ray is
installed and fail otherwise. Note that tpu only supports Ray for distributed inference.
--download-dir DOWNLOAD_DIR
Directory to download and load the weights, default to the default cache dir of huggingface.
--dtype {auto,half,float16,bfloat16,float,float32}
Data typefor model weights and activations. * "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. * "half"for
FP16. Recommended for AWQ quantization. * "float16" is the same as "half". * "bfloat16"for a balance between precision and range. * "float" is shorthand
for FP32 precision. * "float32"for FP32 precision.
--enable-auto-tool-choice
Enable auto tool choice for supported models. Use ``--tool-call-parser`` to specify which parser to use.
--enable-chunked-prefill [ENABLE_CHUNKED_PREFILL]
If set, the prefill requests can be chunked based on the max_num_batched_tokens.
--enable-lora If True, enable handling of LoRA adapters.
--enable-lora-bias If True, enable bias for LoRA adapters.
--enable-prefix-caching, --no-enable-prefix-caching
Enables automatic prefix caching. Use ``--no-enable-prefix-caching`` to disable explicitly.
--enable-prompt-adapter
If True, enable handling of PromptAdapters.
--enable-prompt-tokens-details
If set to True, enable prompt_tokens_details in usage.
--enable-reasoning Whether to enable reasoning_content for the model. If enabled, the model will be able to generate reasoning content.
--enable-request-id-headers
If specified, API server will add X-Request-Id header to responses. Caution: this hurts performance at high QPS.
--enable-sleep-mode Enable sleep mode for the engine. (only cuda platform is supported)
--enforce-eager Always use eager-mode PyTorch. If False, will use eager mode and CUDA graph in hybrid for maximal performance and flexibility.
--fully-sharded-loras
By default, only half of the LoRA computation is sharded with tensor parallelism. Enabling this will use the fully sharded layers. At high sequence length,
max rank or tensor parallel size, this is likely faster.
--generation-config GENERATION_CONFIG
The folder path to the generation config. Defaults to None, no generation config is loaded, vLLM defaults will be used. If set to 'auto', the generation
config will be loaded from model path. If set to a folder path, the generation config will be loaded from the specified folder path. If `max_new_tokens` is
specified in generation config, then it sets a server-wide limit on the number of output tokens for all requests.
--gpu-memory-utilization GPU_MEMORY_UTILIZATION
The fraction of GPU memory to be used for the model executor, which can range from 0 to 1. For example, a value of 0.5 would imply 50% GPU memory
utilization. If unspecified, will use the default value of 0.9. This is a per-instance limit, and only applies to the current vLLM instance.It does not
matter if you have another vLLM instance running on the same GPU. For example, if you have two vLLM instances running on the same GPU, you can set the GPU
memory utilization to 0.5 for each instance.
--guided-decoding-backend GUIDED_DECODING_BACKEND
Which engine will be used for guided decoding (JSON schema / regex etc) by default. Currently support https://github.com/outlines-dev/outlines,
https://github.com/mlc-ai/xgrammar, and https://github.com/noamgat/lm-format-enforcer. Can be overridden per request via guided_decoding_backend parameter.
Backend-sepcific options can be supplied in a comma-separated list following a colon after the backend name. Valid backends and all available options are:
[xgrammar:no-fallback, outlines:no-fallback, lm-format-enforcer:no-fallback]
--hf-overrides HF_OVERRIDES
Extra arguments for the HuggingFace config. This should be a JSON string that will be parsed into a dictionary.
--host HOST Host name.
--ignore-patterns IGNORE_PATTERNS
The pattern(s) to ignore when loading the model.Default to `original/**/*` to avoid repeated loading of llama's checkpoints.
--kv-cache-dtype {auto,fp8,fp8_e5m2,fp8_e4m3}
Data type for kv cache storage. If "auto", will use model data type. CUDA 11.8+ supports fp8 (=fp8_e4m3) and fp8_e5m2. ROCm (AMD GPU) supports fp8
(=fp8_e4m3)
--kv-transfer-config KV_TRANSFER_CONFIG
The configurations for distributed KV cache transfer. Should be a JSON string.
--limit-mm-per-prompt LIMIT_MM_PER_PROMPT
For each multimodal plugin, limit how many input instances to allow for each prompt. Expects a comma-separated list of items, e.g.: `image=16,video=2`
allows a maximum of 16 images and 2 videos per prompt. Defaults to 1 for each modality.
--load-format {auto,pt,safetensors,npcache,dummy,tensorizer,sharded_state,gguf,bitsandbytes,mistral,runai_streamer}
The format of the model weights to load. * "auto" will try to load the weights in the safetensors format and fall back to the pytorch bin format if
safetensors format is not available. * "pt" will load the weights in the pytorch bin format. * "safetensors" will load the weights in the safetensors
format. * "npcache" will load the weights in pytorch format and store a numpy cache to speed up the loading. * "dummy" will initialize the weights with
random values, which is mainly for profiling. * "tensorizer" will load the weights using tensorizer from CoreWeave. See the Tensorize vLLM Model script in
the Examples section for more information. * "runai_streamer" will load the Safetensors weights using Run:aiModel Streamer * "bitsandbytes" will load the
weights using bitsandbytes quantization.
--logits-processor-pattern LOGITS_PROCESSOR_PATTERN
Optional regex pattern specifying valid logits processor qualified names that can be passed with the `logits_processors` extra completion argument. Defaults
to None, which allows no processors.
--long-lora-scaling-factors LONG_LORA_SCALING_FACTORS
Specify multiple scaling factors (which can be different from base model scaling factor - see eg. Long LoRA) to allow for multiple LoRA adapters trained
with those scaling factors to be used at the same time. If not specified, only adapters trained with the base model scaling factor are allowed.
--long-prefill-token-threshold LONG_PREFILL_TOKEN_THRESHOLD
For chunked prefill, a request is considered long if the prompt is longer than this number of tokens. Defaults to 4% of the model's context length.
--lora-dtype {auto,float16,bfloat16}
Data typefor LoRA. If auto, will default to base model dtype.
--lora-extra-vocab-size LORA_EXTRA_VOCAB_SIZE
Maximum size of extra vocabulary that can be present in a LoRA adapter (added to the base model vocabulary).
--lora-modules LORA_MODULES [LORA_MODULES ...]
LoRA module configurations in either 'name=path' formator JSON format. Example (old format): ``'name=path'`` Example (new format): ``{"name": "name",
"path": "lora_path", "base_model_name": "id"}``
--max-cpu-loras MAX_CPU_LORAS
Maximum number of LoRAs to store in CPU memory. Must be >= than max_loras. Defaults to max_loras.
--max-log-len MAX_LOG_LEN
Max number of prompt characters or prompt ID numbers being printed inlog. Default: Unlimited
--max-logprobs MAX_LOGPROBS
Max number of log probs to return logprobs is specified in SamplingParams.
--max-long-partial-prefills MAX_LONG_PARTIAL_PREFILLS
For chunked prefill, the maximum number of prompts longer than --long-prefill-token-threshold that will be prefilled concurrently. Setting this less than
--max-num-partial-prefills will allow shorter prompts to jump the queue in front of longer prompts in some cases, improving latency. Defaults to 1.
--max-lora-rank MAX_LORA_RANK
Max LoRA rank.
--max-loras MAX_LORAS
Max number of LoRAs in a single batch.
--max-model-len MAX_MODEL_LEN
Model context length. If unspecified, will be automatically derived from the model config.
--max-num-batched-tokens MAX_NUM_BATCHED_TOKENS
Maximum number of batched tokens per iteration.
--max-num-partial-prefills MAX_NUM_PARTIAL_PREFILLS
For chunked prefill, the max number of concurrent partial prefills.Defaults to 1
--max-num-seqs MAX_NUM_SEQS
Maximum number of sequences per iteration.
--max-parallel-loading-workers MAX_PARALLEL_LOADING_WORKERS
Load model sequentially in multiple batches, to avoid RAM OOM when using tensor parallel and large models.
--max-prompt-adapter-token MAX_PROMPT_ADAPTER_TOKEN
Max number of PromptAdapters tokens
--max-prompt-adapters MAX_PROMPT_ADAPTERS
Max number of PromptAdapters in a batch.
--max-seq-len-to-capture MAX_SEQ_LEN_TO_CAPTURE
Maximum sequence length covered by CUDA graphs. When a sequence has context length larger than this, we fall back to eager mode. Additionally for encoder-
decoder models, if the sequence length of the encoder input is larger than this, we fall back to the eager mode.
--middleware MIDDLEWARE
Additional ASGI middleware to apply to the app. We accept multiple --middleware arguments. The value should be an import path. If a function is provided,
vLLM will add it to the server using ``@app.middleware('http')``. If a class is provided, vLLM will add it to the server using ``app.add_middleware()``.
--mm-processor-kwargs MM_PROCESSOR_KWARGS
Overrides for the multimodal input mapping/processing, e.g., image processor. For example: ``{"num_crops": 4}``.
--model MODEL Name or path of the huggingface model to use.
--model-impl {auto,vllm,transformers}
Which implementation of the model to use. * "auto" will try to use the vLLM implementation if it exists and fall back to the Transformers implementation if
no vLLM implementation is available. * "vllm" will use the vLLM model implementation. * "transformers" will use the Transformers model implementation.
--model-loader-extra-config MODEL_LOADER_EXTRA_CONFIG
Extra config for model loader. This will be passed to the model loader corresponding to the chosen load_format. This should be a JSON string that will be
parsed into a dictionary.
--multi-step-stream-outputs [MULTI_STEP_STREAM_OUTPUTS]
If False, then multi-step will stream outputs at the end of all steps
--ngram-prompt-lookup-max NGRAM_PROMPT_LOOKUP_MAX
Max size of window for ngram prompt lookup in speculative decoding.
--ngram-prompt-lookup-min NGRAM_PROMPT_LOOKUP_MIN
Min size of window for ngram prompt lookup in speculative decoding.
--num-gpu-blocks-override NUM_GPU_BLOCKS_OVERRIDE
If specified, ignore GPU profiling result and use this number of GPU blocks. Used for testing preemption.
--num-lookahead-slots NUM_LOOKAHEAD_SLOTS
Experimental scheduling config necessary for speculative decoding. This will be replaced by speculative config in the future; it is present to enable
correctness tests until then.
--num-scheduler-steps NUM_SCHEDULER_STEPS
Maximum number of forward steps per scheduler call.
--num-speculative-tokens NUM_SPECULATIVE_TOKENS
The number of speculative tokens to sample from the draft model in speculative decoding.
--otlp-traces-endpoint OTLP_TRACES_ENDPOINT
Target URL to which OpenTelemetry traces will be sent.
--override-generation-config OVERRIDE_GENERATION_CONFIG
Overrides or sets generation config in JSON format. e.g. ``{"temperature": 0.5}``. If used with --generation-config=auto, the override parameters will be
merged with the default config from the model. If generation-config is None, only the override parameters are used.
--override-neuron-config OVERRIDE_NEURON_CONFIG
Override or set neuron device configuration. e.g. ``{"cast_logits_dtype": "bloat16"}``.
--override-pooler-config OVERRIDE_POOLER_CONFIG
Override or set the pooling method for pooling models. e.g. ``{"pooling_type": "mean", "normalize": false}``.
--pipeline-parallel-size PIPELINE_PARALLEL_SIZE, -pp PIPELINE_PARALLEL_SIZE
Number of pipeline stages.
--port PORT Port number.
--preemption-mode PREEMPTION_MODE
If 'recompute', the engine performs preemption by recomputing; If 'swap', the engine performs preemption by block swapping.
--prompt-adapters PROMPT_ADAPTERS [PROMPT_ADAPTERS ...]
Prompt adapter configurations in the format name=path. Multiple adapters can be specified.
--qlora-adapter-name-or-path QLORA_ADAPTER_NAME_OR_PATH
Name or path of the QLoRA adapter.
--quantization {aqlm,awq,deepspeedfp,tpu_int8,fp8,ptpc_fp8,fbgemm_fp8,modelopt,marlin,gguf,gptq_marlin_24,gptq_marlin,awq_marlin,gptq,compressed-tensors,bitsandbytes,qqq,hqq,experts_int8,neuron_quant,ipex,quark,moe_wna16,None}, -q {aqlm,awq,deepspeedfp,tpu_int8,fp8,ptpc_fp8,fbgemm_fp8,modelopt,marlin,gguf,gptq_marlin_24,gptq_marlin,awq_marlin,gptq,compressed-tensors,bitsandbytes,qqq,hqq,experts_int8,neuron_quant,ipex,quark,moe_wna16,None}
Method used to quantize the weights. If None, we first check the `quantization_config` attribute in the model config file. If that is None, we assume the
model weights are not quantized and use `dtype` to determine the data type of the weights.
--ray-workers-use-nsight
If specified, use nsight to profile Ray workers.
--reasoning-parser {deepseek_r1}
Select the reasoning parser depending on the model that you're using. This is used to parse the reasoning content into OpenAI API format. Required for
``--enable-reasoning``.
--response-role RESPONSE_ROLE
The role name to return if ``request.add_generation_prompt=true``.
--return-tokens-as-token-ids
When ``--max-logprobs`` is specified, represents single tokens as strings of the form 'token_id:{token_id}' so that tokens that are not JSON-encodable can
be identified.
--revision REVISION The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.
--root-path ROOT_PATH
FastAPI root_path when app is behind a path based routing proxy.
--rope-scaling ROPE_SCALING
RoPE scaling configuration in JSON format. For example, ``{"rope_type":"dynamic","factor":2.0}``
--rope-theta ROPE_THETA
RoPE theta. Use with `rope_scaling`. In some cases, changing the RoPE theta improves the performance of the scaled model.
--scheduler-cls SCHEDULER_CLS
The scheduler class to use. "vllm.core.scheduler.Scheduler" is the default scheduler. Can be a class directly or the path to a class of form
"mod.custom_class".
--scheduler-delay-factor SCHEDULER_DELAY_FACTOR
Apply a delay (of delay factor multiplied by previous prompt latency) before scheduling next prompt.
--scheduling-policy {fcfs,priority}
The scheduling policy to use. "fcfs" (first come first served, i.e. requests are handled in order of arrival; default) or "priority" (requests are handled
based on given priority (lower value means earlier handling) and time of arrival deciding any ties).
--seed SEED Random seed for operations.
--served-model-name SERVED_MODEL_NAME [SERVED_MODEL_NAME ...]
The model name(s) used in the API. If multiple names are provided, the server will respond to any of the provided names. The model name in the model field
of a response will be the first name in this list. If not specified, the model name will be the same as the ``--model`` argument. Noted that this name(s)
will also be used in `model_name` tag content of prometheus metrics, if multiple names provided, metrics tag will take the first one.
--skip-tokenizer-init
Skip initialization of tokenizer and detokenizer.
--spec-decoding-acceptance-method {rejection_sampler,typical_acceptance_sampler}
Specify the acceptance method to use during draft token verification in speculative decoding. Two types of acceptance routines are supported: 1)
RejectionSampler which does not allow changing the acceptance rate of draft tokens, 2) TypicalAcceptanceSampler which is configurable, allowing for a higher
acceptance rate at the cost of lower quality, and vice versa.
--speculative-disable-by-batch-size SPECULATIVE_DISABLE_BY_BATCH_SIZE
Disable speculative decoding for new incoming requests if the number of enqueue requests is larger than this value.
--speculative-disable-mqa-scorer
If set to True, the MQA scorer will be disabled in speculative and fall back to batch expansion
--speculative-draft-tensor-parallel-size SPECULATIVE_DRAFT_TENSOR_PARALLEL_SIZE, -spec-draft-tp SPECULATIVE_DRAFT_TENSOR_PARALLEL_SIZE
Number of tensor parallel replicas for the draft model in speculative decoding.
--speculative-max-model-len SPECULATIVE_MAX_MODEL_LEN
The maximum sequence length supported by the draft model. Sequences over this length will skip speculation.
--speculative-model SPECULATIVE_MODEL
The name of the draft model to be used in speculative decoding.
--speculative-model-quantization {aqlm,awq,deepspeedfp,tpu_int8,fp8,ptpc_fp8,fbgemm_fp8,modelopt,marlin,gguf,gptq_marlin_24,gptq_marlin,awq_marlin,gptq,compressed-tensors,bitsandbytes,qqq,hqq,experts_int8,neuron_quant,ipex,quark,moe_wna16,None}
Method used to quantize the weights of speculative model. If None, we first check the `quantization_config` attribute in the model config file. If that is
None, we assume the model weights are not quantized and use `dtype` to determine the data type of the weights.
--ssl-ca-certs SSL_CA_CERTS
The CA certificates file.
--ssl-cert-reqs SSL_CERT_REQS
Whether client certificate is required (see stdlib ssl module's).
--ssl-certfile SSL_CERTFILE
The file path to the SSL cert file.
--ssl-keyfile SSL_KEYFILE
The file path to the SSL key file.
--swap-space SWAP_SPACE
CPU swap space size (GiB) per GPU.
--task {auto,generate,embedding,embed,classify,score,reward,transcription}
The task to use the model for. Each vLLM instance only supports one task, even if the same model can be used for multiple tasks. When the model only
supports one task, ``"auto"`` can be used to select it; otherwise, you must specify explicitly which task to use.
--tensor-parallel-size TENSOR_PARALLEL_SIZE, -tp TENSOR_PARALLEL_SIZE
Number of tensor parallel replicas.
--tokenizer TOKENIZER
Name or path of the huggingface tokenizer to use. If unspecified, model name or path will be used.
--tokenizer-mode {auto,slow,mistral,custom}
The tokenizer mode. * "auto" will use the fast tokenizer if available. * "slow" will always use the slow tokenizer. * "mistral" will always use the
`mistral_common` tokenizer. * "custom" will use --tokenizer to select the preregistered tokenizer.
--tokenizer-pool-extra-config TOKENIZER_POOL_EXTRA_CONFIG
Extra config for tokenizer pool. This should be a JSON string that will be parsed into a dictionary. Ignored if tokenizer_pool_size is 0.
--tokenizer-pool-size TOKENIZER_POOL_SIZE
Size of tokenizer pool to use for asynchronous tokenization. If 0, will use synchronous tokenization.
--tokenizer-pool-type TOKENIZER_POOL_TYPE
Type of tokenizer pool to use for asynchronous tokenization. Ignored if tokenizer_pool_size is 0.
--tokenizer-revision TOKENIZER_REVISION
Revision of the huggingface tokenizer to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.
--tool-call-parser {granite-20b-fc,granite,hermes,internlm,jamba,llama3_json,mistral,pythonic} or name registered in --tool-parser-plugin
Select the tool call parser depending on the model that you're using. This is used to parse the model-generated tool call into OpenAI API format. Required
for ``--enable-auto-tool-choice``.
--tool-parser-plugin TOOL_PARSER_PLUGIN
Special the tool parser plugin write to parse the model-generated tool into OpenAI API format, the name register in this plugin can be used in ``--tool-
call-parser``.
--trust-remote-code Trust remote code from huggingface.
--typical-acceptance-sampler-posterior-alpha TYPICAL_ACCEPTANCE_SAMPLER_POSTERIOR_ALPHA
A scaling factor for the entropy-based threshold for token acceptance in the TypicalAcceptanceSampler. Typically defaults to sqrt of --typical-acceptance-
sampler-posterior-threshold i.e. 0.3
--typical-acceptance-sampler-posterior-threshold TYPICAL_ACCEPTANCE_SAMPLER_POSTERIOR_THRESHOLD
Set the lower bound threshold for the posterior probability of a token to be accepted. This threshold is used by the TypicalAcceptanceSampler to make
sampling decisions during speculative decoding. Defaults to 0.09
--use-v2-block-manager
[DEPRECATED] block manager v1 has been removed and SelfAttnBlockSpaceManager (i.e. block manager v2) is now the default. Setting this flag to True or False
has no effect on vLLM behavior.
--uvicorn-log-level {debug,info,warning,error,critical,trace}
Log level for uvicorn.
--worker-cls WORKER_CLS
The worker class to use for distributed execution.
-h, --help show this help message and exit