
How to run gpt-oss-120b
Bedrock, Vertex AI, or your own GPUs · Checked November 21, 2026
gpt-oss-120b has 117B parameters, so it needs about 281GB of GPU memory at full precision and about 141GB at FP8. A practical starting point is 4-bit quantized on 1x 80GB GPU.
You have three ways to run it: a managed host that already serves it, your own GPUs in your cloud, or both behind one endpoint. Then you point your agents at it. Here is what each path takes.
gpt-oss-120b hardware requirements
| Precision | Memory needed | Smallest setup that fits |
|---|---|---|
| BF16 (full precision) | ~281GB | 4x 80GB GPUs |
| FP8 | ~141GB | 2x 80GB GPUs |
| 4-bit quantized | ~71GB | 1x 80GB GPU (H100 or A100 80GB) |
Our estimates: 117B parameters times bytes per parameter, plus 20% for the KV cache and the serving runtime. As a mixture-of-experts model, gpt-oss-120b activates only 5.1B per token, which makes it fast, but every expert still has to sit in memory. The official gpt-oss-120b weights ship 4-bit quantized, so that row is the default. Using the full 128K-token context needs more. Check the model card for the precision the weights ship in and any recommended settings.
Where to run gpt-oss-120b
01Amazon Bedrock
Availablegpt-oss-120b is available on Amazon Bedrock as a managed, serverless model. You call it from your AWS account with IAM, and AWS runs the GPUs. Source
02Google Cloud Vertex AI
Availablegpt-oss-120b is available on Vertex AI as a managed API in Model Garden. You call it from your Google Cloud project, and Google runs the GPUs. Source
03Self-host on your own GPUs
vLLM and SGLang support confirmedDownload openai/gpt-oss-120b from Hugging Face and serve it with an inference server that exposes an OpenAI-compatible endpoint. vLLM and SGLang support it. Start with 4-bit quantized on 1x 80GB GPU (H100 or A100 80GB), then load-test with your real prompts before you size for production.
- Keep the endpoint private to your VPC, so prompts and payment data never leave your cloud.
- Read the Apache 2.0 license, which allows commercial use.
- Run two replicas or keep a hosted model as a fallback, so agents keep working during an upgrade.
Run gpt-oss-120b with Runtime
Serving the model is half the job. The other half is giving agents a safe place to use it. Runtime is the AI agent harness for payment and fintech teams, and it is model-neutral: connect gpt-oss-120b from Bedrock, Vertex AI, or your own endpoint, and your agents use it next to Claude, GPT, or Gemini.
- In your cloud. Runtime runs in your AWS, GCP, or Azure account or fully self-hosted, next to the model.
- Routed by evals. Evals show which steps gpt-oss-120b handles as well as a frontier model, so routine work moves to the cheaper model.
- Fallbacks. If your endpoint is down or slow, agents fall back to another model instead of stopping.
- Guardrails. Agents start read-only, wait for approval before money moves, and every run is recorded.
Harnesses that can drive gpt-oss-120b on Runtime:
Running gpt-oss-120b: FAQ
How much GPU memory does gpt-oss-120b need?
+
About 281GB at BF16, 141GB at FP8, and 71GB at 4 bits. These are our estimates: the weights plus 20% headroom. Long contexts need more memory for the KV cache.
What is the cheapest way to self-host gpt-oss-120b?
+
Start with 4-bit quantized on 1x 80GB GPU (H100 or A100 80GB). Quantizing further saves memory but can cost accuracy, so test it on your own tasks first.
Is gpt-oss-120b available on Amazon Bedrock?
+
Yes. gpt-oss-120b is available on Amazon Bedrock as a managed, serverless model.
Is gpt-oss-120b available on Google Vertex AI?
+
Yes. gpt-oss-120b is available on Vertex AI as a managed API in Model Garden.
Can I use gpt-oss-120b with Runtime?
+
Yes. Runtime is model-neutral. Serve gpt-oss-120b from Bedrock, Vertex AI, or your own endpoint, connect it to Runtime, and agents use it through a harness like Codex. Your data stays in your cloud, and evals tell you which steps it can take over from a more expensive model.