Vertex AI gives you access to Google's foundation models — including Gemini — through a managed API. Pairing it with FastAPI makes it straightforward to expose an AI-powered endpoint your frontend or other services can call.

Prerequisites

  • Google Cloud project with Vertex AI API enabled
  • Service account with roles/aiplatform.user
  • Python 3.11+

Project setup

pip install fastapi uvicorn google-cloud-aiplatform

Set your credentials:

export GOOGLE_APPLICATION_CREDENTIALS="path/to/service-account.json"
export GCP_PROJECT_ID="your-project-id"
export GCP_LOCATION="us-central1"

Initialize Vertex AI

import vertexai
from vertexai.generative_models import GenerativeModel
import os

vertexai.init(
    project=os.environ["GCP_PROJECT_ID"],
    location=os.environ["GCP_LOCATION"],
)

model = GenerativeModel("gemini-1.5-flash")

Basic FastAPI endpoint

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class PromptRequest(BaseModel):
    prompt: str
    max_tokens: int = 1024

@app.post("/generate")
async def generate(req: PromptRequest):
    response = model.generate_content(
        req.prompt,
        generation_config={"max_output_tokens": req.max_tokens},
    )
    return {"text": response.text}

Streaming responses

For long outputs, streaming avoids timeout issues and improves perceived latency.

from fastapi.responses import StreamingResponse

@app.post("/generate/stream")
async def generate_stream(req: PromptRequest):
    def stream():
        for chunk in model.generate_content(req.prompt, stream=True):
            yield chunk.text

    return StreamingResponse(stream(), media_type="text/plain")

Adding a system prompt

Wrapping user input with a system context keeps responses focused:

from vertexai.generative_models import Content, Part

@app.post("/chat")
async def chat(req: PromptRequest):
    system = "You are a concise technical assistant. Answer in plain text."
    response = model.generate_content([
        Content(role="user", parts=[Part.from_text(system)]),
        Content(role="model", parts=[Part.from_text("Understood.")]),
        Content(role="user", parts=[Part.from_text(req.prompt)]),
    ])
    return {"text": response.text}

Running locally

uvicorn main:app --reload

Test with curl:

curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Explain vector embeddings in two sentences."}'

Deploying to Cloud Run

gcloud run deploy vertex-api \
  --source . \
  --region us-central1 \
  --allow-unauthenticated \
  --set-env-vars GCP_PROJECT_ID=your-project-id,GCP_LOCATION=us-central1

On Cloud Run, Application Default Credentials are picked up automatically — no GOOGLE_APPLICATION_CREDENTIALS needed.

Key takeaways

  • GenerativeModel handles authentication and retries via the google-cloud-aiplatform SDK
  • Streaming with stream=True is the right default for any prompt that might return more than a few sentences
  • Keep the model instance module-level so it is initialised once at startup, not per request