Formatho Foundry
Internal LLM Infrastructure for Enterprise
Your own LLM platform, inside your network. We design and install private large-language-model infrastructure: high-throughput open-source inference on GPUs you own, behind single sign-on, with per-team metering, quotas and audit logs - exposed to every application through one OpenAI-compatible API. Zero data egress by design.
1. Assess
Model selection, GPU sizing, quantization strategy, and a TCO model: internal platform vs cloud APIs at your real token volumes - including SSO and metering requirements gathering.
2. Pilot
Single-cluster deployment with an OpenAI-compatible internal endpoint: SSO authentication (OIDC/SAML), per-team API keys, token metering, and latency/throughput benchmarks against your SLOs.
3. Production
Kubernetes autoscaling, multi-model routing, per-team quotas and chargeback reporting, observability dashboards, load tests, runbooks and handover training.
Why enterprises run LLMs in-house
- Cost inflection. Past roughly 50-100M tokens/month, owned GPUs typically undercut per-token API pricing - the assessment quantifies your exact crossover.
- Data stays home. Prompts, documents and completions never leave your network. Same zero-egress principle Formatho is built on.
- Governance from day one. SSO for who can call models, quotas for how much, metering for who pays, audit logs for what happened - controls cloud APIs cannot give your security team.
- No rate limits, no model deprecations. Your endpoint, your model versions, your upgrade schedule.
What the engagement covers
SSO: OIDC, SAML and LDAP integration
Per-team token metering & chargeback
Quotas, rate limits and priority lanes
OpenAI-compatible API gateway & routing
Multi-model serving on one cluster
Fine-tuned and adapter-based models
Quantization for cost-efficient serving
Kubernetes autoscaling (scale-to-zero)
Prometheus / Grafana dashboards & SLOs
Load testing (tokens/s, p95 latency)
Audit logging & security hardening
Model CI/CD, runbooks + handover
Talk to us about your workload
Send us your token volumes, latency needs, SSO provider and current API spend - you get back a GPU sizing, TCO estimate and platform blueprint in the first call, no commitment.
Replies within one business day. All infrastructure work happens inside your accounts and network.
FAQ
▶How much GPU do we need?
Rule of thumb: VRAM must hold model weights plus 10-20% headroom for serving. A 70B model at FP16 needs roughly 140GB (two 80GB accelerators); quantized to 4-bit it fits on a single 48GB card. The assessment sizes this precisely against your concurrency and context-length needs.
▶When does an internal platform beat cloud APIs on cost?
Typically past 50-100M tokens/month with steady utilization, though it depends on model size, quantization tolerance and whether you can scale to zero. We build the TCO model with your real numbers - including GPU rental vs purchase - before anyone commits.
▶Can teams keep using the OpenAI SDK?
Yes. The internal gateway is OpenAI-compatible, so existing applications usually work after changing the base URL and API key - no rewrites.
▶How does SSO work with model access?
The gateway authenticates users through your existing identity provider (OIDC, SAML or LDAP) and maps identity to per-team API keys, quotas and audit trails - so access reviews and offboarding work exactly like every other internal system.
▶Can we track and charge back usage per team?
Yes - per-request metering records tokens, model, latency and calling team, exported to your billing or FinOps pipeline. Quotas and rate limits enforce the budgets you set.
▶How long does an engagement take?
Assessment is 1-2 weeks. A benchmarked pilot with SSO and metering is typically 2-3 weeks. Production hardening (autoscaling, observability, runbooks, handover) runs 3-6 weeks depending on your environment.
▶Is our data exposed during the engagement?
No. All work happens inside your cloud accounts and network. We never need your prompts or documents - sizing uses token volumes and concurrency profiles, not content.
Formatho builds private infrastructure for AI agents - self-hosted runtimes, MCP tooling and free client-side developer tools used by thousands of engineers monthly.