Every organization is making AI decisions right now — often without a framework for making them well.
Some organizations have banned ChatGPT outright, only to watch employees use it on their phones in the office. Others have adopted Microsoft 365 Copilot or Google Gemini for Workspace without examining what data those tools can access or what their employees are actually sending to them. A growing number are standing up local language models on internal infrastructure and discovering that “private AI” is not quite as simple as running ollama pull llama3.
The question of local AI vs frontier models is not primarily a technical question. It’s a governance question — about what data your organization is willing to share with external AI providers, what level of control you need over AI behavior, and what compliance obligations shape your choices.
This guide works through the full decision: the technical landscape of local and cloud AI, the privacy and security dimensions, and — crucially — the AI usage policies that need to exist regardless of which technical path you choose, scaled to organizations from 5 to 5,000 employees.
Understanding the Landscape: What “Local AI” Actually Means
The term “local AI” means different things at different scales, and the distinction matters for planning.
Local AI: The Spectrum
Fully local, single device — A language model running entirely on a developer’s laptop or workstation, with no network connectivity required. Tools like Ollama (the dominant local LLM runner as of 2026), LM Studio, and Jan make this accessible without deep technical expertise. Pull a model, run it, everything stays on the device.
Practical capability in 2026: Models in the 7B–13B parameter range run adequately on a modern MacBook Pro (M-series chip) or a Windows workstation with a discrete GPU. Models in the 70B range require significant hardware. Frontier-level capability on local hardware requires either expensive enterprise GPU hardware or accepting a capability tradeoff.
Self-hosted, internal infrastructure — A language model running on a server within your organization’s network, accessible to all users but never leaving your perimeter. Tools like vLLM (high-performance inference server), Ollama (server mode), and LM Deploy run models on GPU-equipped servers that serve multiple users simultaneously.
This is the configuration most relevant to organizations of 20+ people who want a shared internal AI capability without cloud dependency. A single A100 GPU server can handle dozens of concurrent users with appropriately sized models.
Private cloud deployment — Self-managed deployment on dedicated cloud infrastructure (Hetzner, CoreWeave, RunPod, or similar), isolated from other tenants, with no data sharing with the infrastructure provider beyond standard cloud compute. Combines private data handling with cloud-scale compute.
Vendor-managed private deployment — Some frontier AI providers offer private deployment options: Azure OpenAI (Microsoft-hosted, isolated from OpenAI’s training data), AWS Bedrock (managed access to foundation models in your AWS account), Google Vertex AI (similar model). These use frontier model capability with improved data isolation compared to consumer APIs.
Frontier Models: The Spectrum
Consumer/developer APIs — Direct API access to OpenAI (GPT-4o, o1), Anthropic (Claude Sonnet/Opus), Google (Gemini Pro/Ultra). Standard terms: your data may be used to improve the model unless you opt out or use enterprise tiers. Frontier capability, maximum convenience, variable data handling.
Enterprise API tiers — Most frontier providers offer enterprise agreements with data processing agreements (DPAs) that explicitly exclude your data from training, with retention controls and compliance documentation. The capability is the same as consumer tiers; the data handling is meaningfully different.
Microsoft 365 Copilot — Microsoft’s integrated AI across Office, Teams, SharePoint, and the Microsoft 365 ecosystem. Uses Azure OpenAI models with Microsoft’s enterprise data handling commitments. Data stays within your Microsoft 365 tenant. By far the most widely deployed “enterprise AI” in 2026, but the data access scope — Copilot can read your email, Teams messages, SharePoint files — is broad and worth understanding before deployment.
Google Workspace AI (Gemini) — Similar integration across Google’s productivity suite, with Google’s enterprise data handling commitments for paid tiers.
The Decision Framework: Local vs Frontier
The choice between local and frontier AI is not binary — most organizations end up with a mix. The framework for deciding which workloads go where:
Data Sensitivity Classification
Low sensitivity — Public information, marketing content, general research, writing assistance on non-confidential topics. Frontier APIs with enterprise tier are generally appropriate.
Medium sensitivity — Internal communications, standard business documents, non-proprietary code, financial analysis without material non-public information. Enterprise tier frontier APIs are appropriate; evaluate your specific vendor’s data handling commitments.
High sensitivity — Intellectual property, product designs, source code for proprietary products, M&A information, regulated personal data (HIPAA, GDPR), attorney-client privileged materials, material non-public information. This category warrants serious evaluation of local AI or vendor-managed private deployment.
Regulatory sensitivity — Data subject to specific regulatory requirements may have legal constraints on which AI providers can be used. Local AI or Azure Government/compliant cloud services may be required.
The Privacy vs Capability Tradeoff
Local AI gives you:
- Complete data control — nothing leaves your infrastructure
- No dependency on vendor uptime or pricing changes
- No training data concerns
- Straightforward compliance documentation
Local AI costs you:
- Significant hardware investment ($10,000–$150,000+ depending on capability requirements)
- IT operational overhead for model management, updates, and infrastructure
- Lower capability ceiling than current frontier models for complex reasoning tasks
- Slower access to capability improvements
Frontier AI gives you:
- Best-in-class capability with continuous improvement
- Low upfront investment (pay per use)
- Multimodal, long-context, and specialized capabilities not easily replicated locally
- Vendor-managed infrastructure and security
Frontier AI costs you:
- Data leaving your infrastructure (even with enterprise DPAs, you are trusting a third party)
- Ongoing subscription and usage costs that scale with usage
- Vendor dependency on pricing, model availability, and uptime
Recommended Architectures by Organization Size
Very Small Organization (5–20 employees)
Recommendation: Frontier API enterprise tier + selective local for sensitive workloads
At this scale, standing up GPU infrastructure is rarely economical.
- Default tool: Microsoft 365 Copilot (if already on M365) or Claude/ChatGPT at enterprise tier for general productivity
- Sensitive workload tool: Ollama running locally on individual machines for work involving proprietary code, client data, or sensitive documents
- Policy: A simple one-page AI usage policy covering what can and can’t go into AI tools
Budget: ~$30–50/user/month for Copilot or equivalent. No new hardware needed.
Small-Medium Organization (20–100 employees)
Recommendation: Hybrid — shared internal AI server + frontier API for specific use cases
At this scale, a shared internal AI deployment starts to make economic sense.
Internal AI stack:
- Server: A used server with 2× NVIDIA A10G or A100 GPUs ($15,000–$40,000), or a cloud GPU instance ($2,000–$5,000/month)
- Software: vLLM or Ollama (server mode) serving a 70B parameter model (Llama 3.1 70B, Mistral Large, or similar)
- Interface: Open WebUI — a clean browser-based chat interface connecting to your internal model server, with user accounts, conversation history, and model switching
- Document Q&A: AnythingLLM or Open WebUI’s RAG features for uploading internal documents and querying them in natural language
Frontier API for: Tasks requiring frontier reasoning capability, multimodal analysis, long-context processing, or real-time web awareness.
Cost: $15,000–$40,000 hardware (amortized over 3–4 years) vs $50/user/month × 60 users = $3,000/month for Copilot. The hybrid approach often wins economically at this scale.
Medium Organization (100–500 employees)
Recommendation: Private AI platform + enterprise frontier APIs + formal AI governance
Internal AI infrastructure:
- Dedicated GPU cluster: 4–8× A100 or H100 GPUs, on-premises or private cloud
- LiteLLM as a unified API gateway — routes requests to either internal models or frontier APIs based on data sensitivity policy rules. Users and applications use a single endpoint; routing happens automatically.
- Qdrant, Weaviate, or pgvector as vector database for enterprise-scale RAG against your document corpus
- Langfuse for observability — logging what queries are being made, which models are being used, what the outputs are
Frontier APIs: Azure OpenAI or Anthropic Enterprise for tasks requiring frontier capability, with full enterprise data processing agreements.
Large Organization (500–5,000 employees)
Recommendation: Enterprise AI platform with security-as-code governance
- Private AI infrastructure for sensitive workloads
- Enterprise agreements with frontier providers for appropriate workloads
- Unified AI gateway enforcing data handling policies automatically
- SSO integration so AI tool access is managed through your identity provider
- SIEM integration for AI usage logging and anomaly detection
- AI system registry: every AI tool in use, who approved it, what data it accesses
- Automated data classification that flags sensitive content before it reaches AI tools
- AI incident response procedure separate from standard IT incidents
Building Your AI Usage Policy: Templates by Size
This is the part most organizations skip — they implement the tools without the governance framework. An AI usage policy should address at minimum:
- What AI tools are approved — name the specific tools, not “any AI tool you want”
- Data classification for AI use — what categories of information go to which AI tier
- Output validation requirements — AI outputs should not go to clients or be used in regulated decisions without human review
- Attribution and disclosure — when AI-assisted content is shared externally, what disclosure is required
- Prohibited uses — creating misleading content, submitting NDA-covered data to unapproved tools
- Reporting — who employees contact if they accidentally sent sensitive data to the wrong tool
Policy Template: Small Organization (5–25 people)
[Organization Name] AI Tool Usage Policy
Approved Tools: [List specific tools — e.g., “Claude Enterprise (company account), internal AI at ai.company.com”]
What You Can Use AI For: Writing assistance, summarization, research, coding help, brainstorming, drafting communications.
What Stays Off AI Tools: Client personal data, proprietary product information, anything under NDA, legal documents, non-public financials. When in doubt, use our internal AI assistant rather than external tools.
You Are Responsible for Outputs: AI can be wrong. Review everything before using it — especially for client-facing work, technical decisions, and anything published or submitted.
Don’t Use Personal AI Accounts for Work: Company accounts only. Personal ChatGPT/Claude accounts don’t have our data processing agreements.
Questions: Contact [IT/security contact].
Policy Template: Medium Organization (25–200 people)
Data classification table:
| Content Category | Internal AI | Enterprise Frontier API | Personal AI Accounts |
|---|---|---|---|
| Generic, non-sensitive | ✅ | ✅ | ❌ |
| Internal documents, non-proprietary code | ✅ preferred | ✅ acceptable | ❌ |
| Client PII, regulated data, IP, legal, M&A | ✅ only | ❌ | ❌ |
Output standards: AI-generated content submitted to clients must be reviewed by the responsible employee. AI-generated code must pass standard code review before merging.
Incident reporting: Report suspected data misrouting to [email protected] within 24 hours. Honest mistakes reported promptly are treated as coaching opportunities, not disciplinary matters.
Policy Template: Large Organization (200+ people)
- Define AI tools as a category of third-party SaaS requiring procurement security review
- Integrate AI data classification with your existing data classification framework
- Establish an AI governance committee with CISO, Legal, HR, and business representation
- Require security assessment for any new AI tool before organizational deployment
- Require annual review and recertification of AI usage policy
- Integrate AI usage logging into SIEM with AI-specific threat detections
- Define escalation path for AI-related incidents separate from standard IT incidents
Building Localized Internal Corporate AI: The RAG Layer
When security leaders talk about “localizing corporate AI,” they typically mean something more specific than just running a model locally: they mean building an AI-powered knowledge layer over the organization’s internal data.
This is what RAG (Retrieval-Augmented Generation) + an internal model enables:
- Ask questions against your internal wiki, documentation, and policies in natural language
- Search and summarize Confluence, SharePoint, Notion, or Google Drive content
- Get answers grounded in your organization’s actual information — not a generic training corpus
The technical stack:
- Document ingestion pipeline — connects to SharePoint, Confluence, Google Drive, Notion and ingests content
- Vector database — stores document chunks as embeddings (Qdrant, Weaviate, or pgvector)
- Retrieval layer — when a user asks a question, retrieves relevant document chunks
- Language model — generates an answer grounded in retrieved documents
- Access control integration — ensures users only retrieve documents they have permission to access
Recommended open-source tools:
- AnythingLLM — turnkey RAG platform with clean UI, local model support, document ingestion from common sources. Best for quick deployment.
- Open WebUI + Pipelines — more flexible, more configuration required
- Dify — workflow-oriented AI platform with RAG capabilities, good for non-technical teams
- LangChain / LlamaIndex — frameworks for building custom RAG pipelines with full control
Critical security requirement: A RAG system that ignores your existing access controls is a data governance failure. If an employee can ask “what is our CEO’s compensation?” and receive an answer from a document they shouldn’t have access to, you have a serious problem. Access control at the retrieval layer is non-negotiable — the RAG system must respect the same permissions as the underlying document store.
The Shadow AI Problem: Why Policy Alone Isn’t Enough
A written AI policy that employees don’t follow is not a policy. Shadow AI — employees using unauthorized tools on personal devices or unauthorized accounts — is reported by 76% of organizations as a definite or probable problem.
Controls that actually reduce shadow AI:
Make the approved tools actually good. Employees use unauthorized tools because approved alternatives are slower, more restricted, or harder to access. If your internal AI requires a ticket to access, employees will use ChatGPT on their phones. The quality and accessibility of approved tools directly affects compliance.
DLP integration for AI traffic. Modern data loss prevention tools can identify traffic to common AI service endpoints — not necessarily to block (which creates friction), but to log and alert, enabling enforcement when violations are significant.
Training with realistic scenarios. “Don’t use AI tools for sensitive data” is abstract. “You receive a client contract for review — which AI tool do you use to summarize it? Answer: internal AI only” is concrete and memorable.
Clear escalation without punishment for honest mistakes. If employees fear consequences for reporting they accidentally submitted sensitive data to the wrong tool, they won’t report. Distinguish between deliberate/negligent violations and honest mistakes where the employee self-reported. The latter warrants coaching, not discipline.
The Bottom Line
Most organizations end up with a nuanced answer that evolves as technology, pricing, and internal capabilities change.
Start with the data sensitivity question — it determines more than anything else. For genuinely sensitive data, local AI or vendor-managed private deployment is the defensible choice. For everything else, enterprise-tier frontier APIs with proper data processing agreements are appropriate and highly capable.
Build the policy before — or at minimum alongside — the tools. The biggest AI risk in most organizations today is not adversarial attacks on AI systems. It’s employees making unguided decisions about what to submit to which AI tool, without a framework for those decisions.
Whatever technical stack you choose, instrument it. Know what queries are being made, which tools are being used, and what data is being processed. AI governance you cannot observe is governance that exists only on paper.
Technical specifications for AI model capabilities and hardware requirements reflect the market as of April 2026. The local AI tooling ecosystem evolves rapidly — verify current capabilities against current documentation.
