8. Remote / SSH / HPC Deployment Recipes

This chapter solves the "data and compute live on a remote server or HPC, but you want to work from your local laptop" problem. The main thread is the most common case — "the server computes, the local browser looks" — with copy-paste-ready step-by-step commands; a few other scenarios are covered at the end.

Two rules run through the whole chapter:

  1. Which machine the kernel (compute) runs on and which machine you drive the UI from are two independent things.
  2. omicos only listens on 127.0.0.1 by default and exposes nothing publicly. Every remote-access scheme is, at bottom, "get traffic to the loopback address on the machine where the kernel lives."

The Standard HPC Flow: Server Computes, Local Browser Looks

Follow the four steps below (the fourth is optional). All kernel calls stay on the remote side, for the lowest bandwidth and latency.

Step 1 · Install omicos on the server

SSH into the server (for HPC, this is the login node), install globally with npm, and confirm the version:

$ npm install -g @omicverse/omicos
$ omicos --version

Expected output:

omicos 0.3.29+a1b2c3d

The omicos hpc wizard in step 2 requires 0.3.29 or newer. If your version is older, upgrade first: npm install -g @omicverse/omicos@latest.

Step 2 · Choose an environment and start on the server (omicos hpc)

omicos hpc is an interactive wizard built for servers / HPC: it lists every Python environment found on the machine and marks whether it's ready → lets you choose → optionally saves it as the default → starts.

$ cd ~/my-analysis
$ omicos hpc

Expected output (first run, interactive):


Select a Python environment for the OmicOS kernel
  1) /home/users/you/miniforge3/bin/python3                      [✗ missing: omicverse]
  2) /home/users/you/miniforge3/envs/omicverse/bin/python3       [✓ ready]
  …
  9) /scratch/users/you/env/navigo/bin/python3                   [✗ missing: omicverse]
  12) /scratch/users/you/envs/dynast/bin/python3                 [✗ missing: omicverse, ipykernel]
  …
  17) [default] OmicOS bundled env (~1.5 GB) → /home/users/you/.omicos/env    [will download]
  18) Enter a Python path manually
Enter number [1-18]: 2

Make this the default startup env? [Y/n] Y
  ✓ saved — .kernel_choice + `export OMICOS_KERNEL_PYTHON` in /home/users/you/.bashrc

✓ kernel env ready: /home/users/you/miniforge3/envs/omicverse/bin/python3

The terminal then switches to the full-screen dashboard, with the Local line showing the actual listen address http://127.0.0.1:5055.

  • Each environment is tagged [✓ ready] (has omicverse + ipykernel installed) or [✗ missing: …] (lists what's missing) — picking a [✓ ready] one is the easiest path. If you pick one that's missing packages, the wizard will ask whether to install them now.
  • The second-to-last entry is the built-in environment (about 1.5 GB, downloaded on first use); the last entry lets you type an interpreter path manually.
  • After choosing, it asks Make this the default startup env? — answering Y writes .kernel_choice and appends export OMICOS_KERNEL_PYTHON to ~/.bashrc, so future launches reuse this environment automatically without asking again (use omicos hpc --reselect to change it).
  • After the choice, the wizard drops straight into the dashboard (listening on 127.0.0.1:5055). To change the bind address, quit and rerun with flags (the environment choice is remembered); add --no-browser to skip the harmless "can't open a browser" notice on a server.

The second time you run it, it won't ask again:

$ omicos hpc

Expected output:

✓ using saved default env: /home/users/you/miniforge3/envs/omicverse/bin/python3
  (run `omicos hpc --reselect` to choose a different env)

It then goes straight into the full-screen dashboard, listening on 127.0.0.1:5055.

Strongly recommended: run omicos hpc once to settle the environment before you connect — otherwise the first startup will bootstrap the environment in the background, and on a large cluster with slow disks, running analysis in the web UI can show "environment is still being prepared" for a long time.

omicos hpc is interactive. Running it in a batch script with no terminal errors out and tells you to use OMICOS_KERNEL_PYTHON=/path/to/python + omicos serve instead.

Step 3 · Forward a port locally and open the browser

The kernel is running on the server (listening on 127.0.0.1:5055). Now connect a port on your laptop to it. How you do that depends on whether you can SSH directly to the machine running the kernel.

Case 1: You can SSH directly to the machine running the kernel (a plain server / login node)

$ ssh -N -L 15055:127.0.0.1:5055 user@your-server

Expected output: no output — the command hangs and doesn't return; this is normal, just leave the window open.

  • -N only builds the tunnel, without opening a shell.
  • -L 15055:127.0.0.1:5055: the left-hand 15055 is a port on your laptop (swap it for any free port if taken), and the right-hand 127.0.0.1:5055 is the remote machine's own loopback address (where the kernel lives).
  • To find which process on your machine is holding a port: lsof -nP -iTCP:15055 -sTCP:LISTEN.

Case 2: Real HPC — login node + compute node

On real HPC systems (e.g. Stanford Sherlock), you can only SSH from your laptop to the login node, while the omicos hpc in step 2 usually runs on a compute node (requested via salloc / srun / sbatch, not exposed to the public internet). First confirm the node name in the terminal where omicos is running:

$ hostname

Expected output:

sh04-05n13

You might reach for ProxyJump (-J), but it typically fails against a compute node:

$ ssh -N -L 15055:127.0.0.1:5055 -J user@login.sherlock.stanford.edu user@sh04-05n13

Expected output:

user@sh04-05n13: Permission denied (hostbased)

Why: with -J, the SSH hop to the compute node is initiated from your laptop, but the compute node only accepts hostbased connections "initiated from the login node." Jupyter doesn't hit this snag because the login node makes a plain TCP connection to an already-open port on the compute node (no second SSH hop), provided the service is bound to the node's network interface. That leaves two paths.

Method 1: do what Jupyter does (bind the node's interface + single-hop forwarding). Start on the compute node with --host 0.0.0.0 — step 2 already saved a default environment, so this run just reuses it:

$ omicos hpc --host 0.0.0.0

Expected output:

✓ using saved default env: /home/users/you/miniforge3/envs/omicverse/bin/python3
  (run `omicos hpc --reselect` to choose a different env)

It then drops into the dashboard, with the Local line showing http://0.0.0.0:5055.

Then on your laptop (note the right-hand side is the node name, not 127.0.0.1):

$ ssh -N -L 15055:sh04-05n13:5055 user@login.sherlock.stanford.edu

Expected output: no output, the command hangs.

⚠️ --host 0.0.0.0 exposes the kernel on the compute node's network interface. On an exclusive node the risk is limited; on a shared node, use Method 2 instead (the kernel never leaves loopback), or use --host $(hostname) to bind only the node's internal IP and try to request an exclusive node.

Method 2 (safer on shared nodes): a two-hop SSH tunnel. The kernel stays on the default 127.0.0.1; the key is that the second hop is initiated from the login node. Run omicos hpc normally on the compute node, then on your laptop connect to the login node:

$ ssh -L 15055:localhost:15055 user@login.sherlock.stanford.edu

Expected output: drops you into a normal login-node shell prompt.

From that login-node prompt, hop to the compute node:

$ ssh -N -L 15055:localhost:5055 sh04-05n13

Expected output: no output, the command hangs.

Chain: laptop:15055 → login node:15055 → compute node:5055 (kernel). Once both hops are up, open the browser. The kernel never leaves loopback, and the login→compute hop is an encrypted SSH tunnel.

Opening the browser

Once the tunnel is up, open in your local browser (swap in your local forwarded port):

https://app.omicos.cn/#/?ws=localhost%3A15055&auto=true
  • ws=localhost:15055 (%3A is an escaped colon) points at your local forwarded port; auto=true means auto-connect.
  • The web app automatically probes local ports 5051–5070; for a port outside that range (like 15055) you must write ?ws= explicitly.
  • ⚠️ The opening: line omicos prints in the server terminal is the address for the direct-connection case (?ws=127.0.0.1%3A5055). Don't copy it as-is once you've set up port forwarding — swap in your local forwarded port (here, 15055).

Step 4 (optional) · Run on a specific port

The default port is 5055. If 5055 is taken on the server, or you want to run multiple instances on the same machine, use --port:

$ omicos hpc --port 5060

Expected output:

✓ using saved default env: /home/users/you/miniforge3/envs/omicverse/bin/python3
  (run `omicos hpc --reselect` to choose a different env)

It then drops into the dashboard, with the Local line showing http://127.0.0.1:5060.

The port may not be the one you specified. If the port you specify is taken, omicos searches forward from the next one (up to 100 ports) and prints a line like [omicos] port 5060 was busy; using 5061 instead. Always go by the listening: line when setting up forwarding. In scripts, add --report-port, which prints an extra OMICOS_LISTENING_PORT=5061 line for programs to read.

Then forward locally to the actual port (right side changes to the actual port, left side stays any free local port):

$ ssh -N -L 15055:127.0.0.1:5060 user@your-server

Expected output: no output, the command hangs.

The browser's ?ws= still points at your local port (15055, not the server's 5060):

https://app.omicos.cn/#/?ws=localhost%3A15055&auto=true

The same applies for a compute node + custom port: for Method 2, swap 5055 for the actual port in both hops; for Method 1, swap sh04-05n13:5055 for sh04-05n13:<actual port>, and start with --host 0.0.0.0.

Other Scenarios

Pure terminal (no browser)

SSH'd into a machine with no graphical interface, done entirely in the terminal — neither login nor chat needs a browser:

$ omicos login
$ cd ~/my-analysis
$ omicos cli

Expected output: omicos login prints the three [omicos] logged in as … lines (see Chapter 5), and omicos cli switches the terminal to a full-screen chat interface.

Inconvenient to type a password on this machine? Use omicos cli login device-code pairing instead, and confirm on another device already logged in to omicOS.

Local laptop + browser

Both data and analysis live on your own machine — the simplest case:

$ cd ~/my-analysis
$ omicos login
$ omicos serve

Expected output: omicos login prints the three login-info lines, and omicos serve enters the full-screen dashboard and auto-opens the browser to https://app.omicos.cn/#/?ws=127.0.0.1%3A5055&auto=true.

Keeping the Process Alive in the Background

On a regular server, you usually want omicos to keep running after you disconnect from SSH (pair all of these with --no-browser):

$ nohup omicos serve --no-browser > omicos.log 2>&1 &
$ disown

Expected output: the shell prints a job number (like [1] 41287) and returns immediately; the startup banner is written to omicos.log:

[omicos] online: my-analysis (local-9f2c4d7a1b8e4f5c8d3a6b0e2f7c1a94)
[omicos] listening: http://127.0.0.1:5055

Or use tmux / screen:

$ tmux new-session -d -s omicos 'cd ~/my-analysis && omicos serve --no-browser'
$ screen -dm -S omicos bash -c 'cd ~/my-analysis && omicos serve --no-browser'

Expected output: both commands produce no output; the process runs in a background session (tmux attach -t omicos lets you check on it).

For a systemd service, ExecStart:

ExecStart=/usr/local/bin/omicos serve --no-browser

(When omicos detects it's not running in an interactive terminal, it automatically skips the dashboard and sends logs to stderr, which fits the systemd journal well.)

⚠️ Scheduler-based HPC (SLURM, etc.) is different: the kernel runs on a compute node obtained via salloc / srun / sbatch, and the kernel disappears the moment the job ends — tmux / nohup on the login node can't save it. For long-running jobs, put omicos serve --host 0.0.0.0 --no-browser (or omicos hpc) into an sbatch script and request enough wall-clock time, or keep a salloc session from exiting.

Health Checks & Smoke Tests

Confirm the service is healthy on the machine running the kernel:

$ curl -sS http://127.0.0.1:5055/health

Expected output:

{"service":"omicos-core","status":"ok","version":{"build_profile":"release","display":"0.3.29+a1b2c3d","git_rev":"a1b2c3d","semver":"0.3.29","target_triple":"x86_64-unknown-linux-musl"}}

$ curl -sS http://127.0.0.1:5055/api/process/info

Expected output (excerpt):

{"core_workspace_root":"/home/users/you/my-analysis","hostname":"sh04-05n13","id":"local-9f2c4d7a1b8e4f5c8d3a6b0e2f7c1a94","kernel":"native-python-worker","launched_by":"terminal","name":"my-analysis","pid":41287,"port":5055,"process_id":"local-9f2c4d7a1b8e4f5c8d3a6b0e2f7c1a94","runtime":"rust","version":"0.3.29", …}

/api/process/info's response has no status field (that's on /health). The hostname line is useful — it tells you exactly which machine the kernel is running on, which makes a misconfigured tunnel obvious at a glance.

Once the tunnel is up, verify from your laptop by swapping in your local forwarded port:

$ curl -sS http://127.0.0.1:15055/health

Expected output: identical to running it on the server above. If you get nothing back, 99% of the time it's a wrong node name, the kernel isn't on that machine, the port isn't what you thought, or the SSH window dropped.

One Thing to Watch: Switching Accounts Re-Registers Process Identity

If you switch the logged-in account within a workspace, the cloud refuses to let the new account take over the old account's process identifier. omicos handles this by rotating the workspace_id and re-registering under the new identity, entirely within the running process — no restart, no exit. What you might observe is that the session list starts out empty under the new account — the old local session files are still there, and the old account's records still belong to the old account.

Troubleshooting: The Web UI Keeps Saying the Environment Isn't Ready

Port forwarding connects, the UI works, but running analysis keeps saying the environment isn't ready — usually one of these two causes:

  1. The environment hasn't finished installing: the first launch prepares the Python environment in the background, which can take a long time on a large cluster with slow disks. Best practice: run omicos hpc (or omicos env setup) to install the environment first, then serve.
  2. Wrong port / kernel not on the expected machine: confirm the tunnel's local port matches the browser's ?ws=, and that the kernel really is on the machine at the other end of the tunnel (verify with curl /health and the hostname field from /api/process/info above).

For more troubleshooting, see Chapter 15: Troubleshooting.

results matching ""

    No results matching ""