NOS
A fast, flexible Inference Server
Run LLMs and multi-modal models cost-efficiently and scalably on any cloud or AI hardware with NOS, a fast and flexible multi-modal inference server built from the ground-up.
Fast & Flexible
Designed to optimize, serve and auto-scale PyTorch models in prod. without compromise.
.svg){kind=small}
Multi-Model
Serve multiple foundation models simultaneously in a single instance.
HW-aware Runtime
Deploy PyTorch models on any AI HW (NVIDIA, AMD, AWS Inf2, GCP TPUs).
Run Anywhere
Run on any cloud (AWS, GCP, Azure, On-Prem) with our ready-to-use inference containers.
NOS is fully open-source with a commercially-usable license (Apache 2.0).
Star us on GithubCheck out our nos-playground for more examples.
