Setting Up an AWS EC2 Instance for Local AI

Who this is actually for

Be clear-eyed about what this guide is for: trying local AI out, not running it long-term. The mini PC guide is the actual destination for most people - own the hardware once, run it forever, nothing recurring. This guide exists for the step before that: no spare machine yet, not ready to spend money on hardware you haven't confirmed you'll use, or a student with credits burning a hole in their AWS account. Treat it as a test drive, not a place to park a real setup.

Think of this the way you'd think of a rental car versus a car you own: fine for figuring out if you like the thing, not where you want to live for years.

There's also a real, different tradeoff here, separate from cost. Running Ollama on a rented AWS instance still means no AI company ever sees your prompts, and nothing trains a model on your conversations - that part of the local-AI pitch holds completely. What doesn't hold is "nothing leaves your house." Your data leaves your home network and lives on a machine you're renting, inside an account Amazon ultimately controls the physical hardware for. Meaningfully better than a cloud AI subscription, but not identical to owning your own box.

Getting AWS credits (especially if you're a student)

AWS Educate is open to anyone 13 or older with an email address, not just students at partner schools - it just gives less credit if your school isn't a member. Register there, not through a regular AWS sign-up, and you'll be offered a choice:

• AWS Educate Starter Account - no credit card required at all. The tradeoff: restricted to CPU-only instance types (which is exactly what this guide recommends anyway) and the us-east-1 (N. Virginia) region only.
• Standard AWS account - needs a card on file, but isn't restricted the same way. Necessary if you want the optional GPU path later.

Either way, expect up to $100 in credits, valid for 12 months. That's a genuinely useful amount if you follow the cost-management advice in this guide - not if you leave the instance running around the clock.

Launching your instance

From the EC2 console, launch a new instance with these choices:

• AMI: Ubuntu Server 24.04 LTS - matches the Linux install path the rest of this site already uses
• Instance type: m5.xlarge (4 vCPU, 16GB RAM) - roughly matches the mini PC guide's specs, and it's on the Starter Account's allowed list, so it works regardless of which account type you picked above
• Key pair: create a new one and actually download the .pem file - you cannot re-download it later if you lose it
• Storage: 50GB is comfortable for the OS plus a handful of models - see the storage guide for the real math if you plan to keep more around
• Security group: allow inbound SSH (port 22) only, from your own IP if the console offers that option. Do not open port 3000 or any other port here - reaching Open WebUI privately is covered separately, and it doesn't need this.

Once it's running, connect over SSH using the key pair you downloaded:

chmod 400 your-key.pem
ssh -i your-key.pem ubuntu@your-instance-public-ip

Installing Ollama and Open WebUI

From here, it's the same Linux install this site already teaches - nothing EC2-specific about Ollama itself:

curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3.2

Follow Getting a Proper Chat Interface Running for Open WebUI next, using the same Docker Engine install path that guide already covers for Linux. The install command binds Open WebUI to 127.0.0.1 by default, exactly like it would on a mini PC - that's still the right default here, and it's what makes the next section actually necessary rather than optional.

Reaching it privately

There's no home Wi-Fi network here, so LAN access doesn't apply the way it does for a mini PC - there's no "local network" for other devices to already be on. Widening the port binding to the whole internet instead would mean an unauthenticated chat interface sitting on a public IP address, found and probed within hours. Don't do that.

Stopping vs. terminating

This is the single most important distinction for keeping this affordable. EC2 bills for compute by the hour while an instance is running - a stopped instance costs nothing in compute charges, and picks up right where you left off when you start it again.

Stop the instance from the EC2 console (or sudo shutdown -h now from inside it) every time you're done with a session. Only terminate if you're genuinely finished with this setup for good.

When to move to owned hardware instead

If you're stopping the instance faithfully between sessions and still checking on it every week, that's your answer: you like this enough to stop renting it. A mini PC's one-time cost passes a continuously-running EC2 instance's cost in under four months - and even with disciplined stop/start habits keeping the bill closer to a few dollars a week, you're still paying every single week, forever, for something a one-time purchase would have already paid off.

Optional: a GPU instance

Want to run something bigger than a 7-8B model, matching what the full PC guide covers for owned hardware? A g4dn.xlarge (T4, 16GB VRAM) is the equivalent here - but be aware of two real friction points before committing to this path:

• It's not available on the Starter Account. GPU instances need a standard AWS account with a card on file - your credits still apply, but the no-card path doesn't support this instance family at all.
• New accounts start with a GPU quota of zero. You have to explicitly request a quota increase (Service Quotas console → Amazon EC2 → search "G instance type" → request an increase) before you can even launch one, and approval can take a day or two - plan ahead rather than expecting to launch one the same day you decide you want it.

What actually goes wrong

"Connection refused" or "Connection timed out" over SSH

Almost always the security group, not the instance itself. Confirm it actually allows inbound SSH from your current IP - home IP addresses can change, and a rule scoped to your old one silently blocks you without any obvious error on the instance's side.

"Permission denied (publickey)"

Check the key file's permissions - SSH refuses to use a key that's readable by anyone else:

chmod 400 your-key.pem

Also confirm you're connecting as the right user for the AMI - ubuntu for Ubuntu Server, not ec2-user or root.

The instance is stopped but you're still being billed

That's the storage cost mentioned above, not a billing error - it's real and expected, just small. If the amount looks bigger than a few dollars, check for other resources left running (an Elastic IP not attached to anything, a second instance you forgot about) in the billing dashboard.

Everything from the regular Ollama/Open WebUI guides applies here too

Once Ollama and Open WebUI are installed, this is the same software as everything else on this site - see the Troubleshooting Hub for anything not specific to EC2 itself.

Common questions