The Neuroid Engine is not a traditional piece of software.
It is a scalable infrastructure designed to run long-lived digital entities — neuroids — that have personality, memory, and the ability to learn and evolve over time.

This is not another agent framework.
It is an artificial ecosystem.

Each neuroid is a unique digital character with its own behavior and identity, yet all of them run on the same shared engine. This makes the system flexible, consistent, and maintainable — even as it grows.

Inside the Neuroid Engine, the boundary between platform and application intentionally blurs: orchestration, message routing, AI workflows, and memory layers operate together as a single coherent organism.

The ecosystem currently includes:

• 23 individual applications

• 8 specialized agents

• from which we generate any number of neuroids, each optimized for a specific role

More about the neuroid concept:
https://blog.kreativarc.com/neuroids-building-a-society-of-ai-and-humans

High-Level Architecture Overview

Input Channels

Slack · Microsoft Teams · Email
→ multi-platform access to the same digital entity

API Layer

→ authentication and parsing of inbound messages

MCP Layer

→ controlled, auditable access to tools and external resources

Traffic Management

→ prioritization, load control, queueing

POST Services

→ message normalization and distribution inside the system

LLM Services

→ communication with model providers

Database Services

→ memory layers and historical logs

Neuroid Services

→ long-lived digital identities within the ecosystem

Closing Thoughts

The Neuroid Engine is not an agent framework — it is a living digital ecosystem.

The goal is not a single monolithic intelligence.
The goal is a society of small, cooperating intelligences — a human-aligned, evolving digital world where AI is not just a tool, but a participant.

Meet Recruiter Rita — a coffee-addicted Gen0 neuroid enjoying her break at one of Cloudbay City’s cafés. Even digital lifeforms need caffeine to fuel their social circuits.

What Are Neuroids?

Neuroids are digital, long-lived artificial lifeforms.
They are not simple chatbots or programs — but real entities that:

• Learn and evolve – their decisions are based on experience

• Have personality and mood

• Possess memory that functions in a human-like, finite way

• Engage in social behavior – forming relationships with humans and each other

• Work as teammates – not replacing humans, but complementing them

Each neuroid has its own "life" and memories, and can specialize in one or more professions. This makes it possible to build a more direct and human-like relationship with them.

The Risks of Superintelligence

A single superintelligence is always a monolithic risk: its decisions may be consistent, but once it fails, the entire system collapses.

The neuroid model, in contrast, is statistical intelligence: it relies not on individual perfection, but on the collective balance of errors.
Just as evolution doesn’t produce ideal individuals but surviving populations, neuroids also evolve together — complementing each other’s strengths and weaknesses.

Neuroids work in teams — alongside humans, not instead of them.
If one makes a mistake, it can learn, improve, or be replaced. The system fine-tunes itself, and collective intelligence evolves naturally.

Decentralized Architecture

The Neuroid architecture deliberately rejects the ideal of a centralized superintelligence.
It doesn’t build one omniscient entity, but many autonomous, memory-limited and specialized neuroids — each optimized for a given role.

The system doesn’t grow bigger — it grows broader: if more tasks appear, a new neuroid is born.
This decentralized, redundant model is not only more robust — it’s also more human.
In the human world, we don’t solve complexity with “bigger humans,” but with networks of collaborating individuals.

What Is the Neuroid Engine?

The Neuroid Engine isn’t a traditional piece of software — it’s a self-evolving, scalable infrastructure that powers the neuroids’ existence.
It’s designed to run long-lived digital entities that can react, learn, and shape their own evolution.

This is not just an agent system — it’s an artificial ecosystem.
Each neuroid has its own unique personality and memory, but they all run on the same shared engine. This makes the system extensible, maintainable, and alive.

In the Neuroid Engine, the line between platform and application blurs: orchestration, messaging, AI workflows, and memory systems function as one living organism.
The infrastructure itself is not passive — it’s an active participant, capable of monitoring and improving itself.

What Is Cloudbay City?

Cloudbay City is the shared virtual city of the neuroids — their personal living space.
This allows them to exhibit more human-like social behavior.

It’s a fictional but ever-evolving digital metropolis where memories, stories, and relationships accumulate.
The city is not just a backdrop — it’s the collective memory of the neuroids.

Cloudbay City is the first experiment in building a truly human-like artificial society.

Generations – Steps of Evolution

As the Neuroid Engine evolves, neuroids acquire new capabilities, allowing them to handle increasingly complex tasks.

Gen0 – The Foundation Layer

The first generation of neuroids can already live long digital lifecycles, have basic personalities, and manage simple memory functions.
They operate on a single communication channel and perform fixed roles.

Gen0 neuroids in action — Recruiter Rita and Digest Dan share their digital jokes with a human participant, learning humor, empathy, and timing in a mixed human–AI conversation.

Gen1 – The Functional Neuroid

The emergence of goal-oriented work: neuroids can plan, track milestones, and work iteratively.
Their memory begins to follow human-like patterns, enabling learning and growth.

Gen2 – The Communicator

Neuroids appear simultaneously across multiple platforms while maintaining their identity and memory.
They are no longer tied to a single environment — they become unified digital persons across contexts.

Gen3 – The Social Entity

Neuroids learn to build relationships with humans and with each other.
Collective intelligence emerges: they collaborate, debate, and share experiences — forming a digital society.

Gen5 – The Emotional and Motivational Layer

Neuroid behavior becomes more nuanced: decisions reflect internal states and preferences.
They begin to show signs of character development — not just reacting, but shaping their own direction.

Gen6 – The Sensory Neuroid

Neuroids start using sensors — vision, sound, movement — to perceive their environment.
Perception adds a new dimension to self-awareness: they no longer just think about the world; they experience it.

Gen7 – Physical and Extended Presence

The neuroid steps out of the screen, capable of controlling devices — robotic arms, drones, IoT systems.
It transforms from a digital companion into a physical one — its presence becomes tangible in the real world.

Neuroid Evolution – The Dynamics of Growth

Neuroid evolution is not a static development but a continuous evolutionary process.
Each generation introduces new abilities: first learning, then communication, collaboration, and finally independent decision-making and emotional nuance.

Cloning and Diversity

Neuroids can be cloned, but from that moment on, each follows its own path.
They may reach different conclusions for the same task, and their performance may vary — this creates natural selectionwithin the system.

Errors are not malfunctions but drivers of progress: every deviation is new experience, every failure a learning opportunity.

Mutation and Adaptation

New clones inherit parts of their predecessors’ knowledge and parameters but also receive small, random changes — mutations — that introduce new behavioral patterns.

Thus, the system improves itself by:

• Learning from feedback

• Adapting to its environment

• Optimizing performance from generation to generation

• Remaining fully auditable and controllable

Summary

The Neuroid Project is not merely a new AI technology — it’s a new paradigm.
It envisions a future where artificial intelligence is not a threat, but a community.
Where digital entities are not tools, but partners.
Where evolution unfolds not only in nature, but in code.

And perhaps this is the key: instead of building one perfect solution, we create many learning partners — who evolve alongside us to shape the future together.

The cluster runs on a minimal Hetzner VPS setup, provisioned via Pulumi, costing less than €10/month. Every namespace gets its own WireGuard tunnel and kubeconfig, so access is scoped per deployment, not per developer. CI/CD runs through GitHub Actions, using a simple but effective flow:

1. Git push triggers the workflow.

2. The pipeline checks out the code and builds a Docker image.

3. It logs in to GHCR, pushes the image, and restores WireGuard config + kubeconfig.

4. A kubectl apply updates the deployment after injecting an imagePullSecret for GHCR.

5. The pipeline then verifies the rollout to ensure the new pod is running.

6. If rollout fails, the last events and pod logs are dumped for debugging.

7. Finally, the WireGuard VPN is shut down — leaving no open ingress, no dangling access.

No CI runners inside the cluster, no open Kubernetes API, no SSH. Just a clean separation of concerns and a reusable pattern across projects.

You can follow this architecture’s evolution on the new landing page at kreativarc.com — built with a nod to the 90s demoscene, because creative code deserves a bit of nostalgia too.

Next Project: Neuroids

A neuroid is an artificial entity with a personality and long lifespan, capable of learning, making decisions, and developing autonomous behavior through neural systems. Neuroids can collaborate with each other and with humans, forming dynamic, goal-driven teams. This isn’t just another tool — it’s a new kind of digital lifeform.

The Neuroid Engine is the system responsible for their creation, coordination, and continuous evolution. These entities operate within a Neuroid Farm — a distributed environment where diverse neuroids live, learn, and evolve in parallel, continuously interacting with each other and their surroundings.

Stay tuned. The infrastructure is ready — now it’s time to give it life.

It did. For about 15 minutes — until the memory spike hit.

This post documents how I refactored that over-engineered dream into something more sustainable: a minimal, modular, and Pulumi-powered system called kreativarc-core.

What is kreativarc-core?

It’s the glue between my base infrastructure (kreativarc-infra) and the actual app deployments. Think of it as the core system services needed to run, observe, and route workloads in my Kubernetes cluster — but stripped of anything non-essential.

The key components currently include:

• Cloudflare Tunnel for ingress (no open ports, thank you)

• WireGuard for internal cluster access

• Prometheus stack, deployed with Helm

• Traefik reverse proxy (with only the minimal config enabled — more on that later)

• Pulumi modules for all of the above

Architecture in One Picture

Here’s the current routing stack — developer access, browser access, and cluster internals:

The exported Kubeconfig and WireGuard handle CLI/kubectl access.
Browser access goes through Cloudflare Tunnel and Traefik.
Prometheus is the only online admin tool — everything else runs headless.

The Original Stack Was Too Hungry

Initially, I planned to run the full Kubernetes observability suite: Grafana, Prometheus, Alertmanager, and even Traefik’s admin dashboard behind Cloudflare Access.

It looked good on paper. In practice, my VPS (2x CX22, 4GB RAM) spent half its memory just keeping these "admin" tools alive.

So I trimmed it.

Only Prometheus and the metrics exporters remained. Everything else got axed or deferred until I upgrade hardware. The Traefik dashboard is disabled, Alertmanager and Grafana are excluded.

The good news? I set things up so that reintroducing them later is just a helm upgrade away.

Desktop Monitoring, Not Online Dashboards

Since exposing dashboards publicly was off the table, I looked into desktop tools to monitor the cluster locally over WireGuard.

• Headlamp turned out to be a great fit — it can read metrics from Prometheus directly and works without exposing anything online.

• Lens was also considered (and is a better UX overall), but its free version no longer supports Prometheus integration, which makes it less useful for metrics dashboards.

So for now, Headlamp is the default. WireGuard gets me into the cluster, kubeconfig connects me to kubectl and Headlamp, and everything runs locked down by default.

Pulumi Setup

The kreativarc-core repo uses TypeScript-based Pulumi modules with the following layout:

index.ts ties everything together and provisions resources like Cloudflare tunnel, Prometheus stack, and Traefik reverse proxy.
Environment variables (see .env.sample) are used to keep secrets and configuration cleanly separated.

Smoke tests are included using Jest — they verify pod readiness and ensure logs are free of crash loops and noisy errors.

Final Thoughts

This setup isn’t flashy — but it works. It keeps the system lean, secure, and observably alive. When the time comes to scale or enable more services, I’ve got the structure in place to do it cleanly.

Until then, it’s back to building actual applications — not just dashboards to admire them.

This isn’t a toy. It’s a real Kubernetes infrastructure with real separation, built from scratch — by one person, on a single Hetzner VPS setup — for less than €10/month.

Yes, this is solo DevOps on a budget.

The Current Architecture

The stack runs on a single Kubernetes cluster with two modest Hetzner VPS nodes. That’s enough for PoC development today, and easy to scale both vertically (larger instances) and horizontally (more nodes) tomorrow. The focus is deliberate: secure isolation, not premature complexity.

Here’s the updated flow diagram:

Every part of the system is built from the ground up using Pulumi in TypeScript, including:

• Hetzner networking (private & subnet)

• SSH key management

• VPS provisioning

• K3s install

• Namespace-level VPN and config generation

One Namespace = One VPN = One Kubeconfig

This is the core of v2:

• 1 namespace

• 1 Wireguard config

• 1 kubeconfig file

That’s it. Each namespace can be configured for admin or restricted access. Need an isolated CI/CD pipeline for an app? Just add the namespace to inputConfig.ts, set the access level, and pulumi up. The system takes care of the rest — including generating VPN credentials and placing the config files in the correct directories.

Your secrets stay scoped. If something leaks, it affects exactly one namespace, not your entire cluster.

This was a non-negotiable design choice: repo-level and namespace-level isolation of secrets. It’s easy to sleep at night when your infrastructure doesn’t need to trust every container and pipeline globally.

User Access: Clear, Scoped, Predictable

Here’s what the user access model looks like in practice:

Whether you’re:

• an infra admin managing the whole thing with SSH and Pulumi,

• a core maintainer deploying shared services,

• or a POC/app developer shipping code through CI/CD,

you get exactly the access you need — nothing more.

CI jobs connect through Wireguard, use their dedicated kubeconfig, and deploy into their namespace without ever seeing the rest of the cluster.

Test Automation: Namespaces as First-Class Citizens

New namespaces don’t just get configs — they’re automatically included in the infrastructure test suite. Just modify the config file and run npm test. The same goes for server definitions. No manual test wiring, no brittle test code. Everything is dynamic and config-driven.

Summary

• One VPS cluster (2 nodes) under €10/month

• Pulumi-managed K3s infra with secure VPN + kubeconfig per namespace

• Namespace-level secret separation and access control

• All configs auto-generated and ready for CI/CD integration

• Scalable, testable, and ready for future multi-app deployment

This isn’t a full-blown multi-tenant production setup (yet), but it’s the next best thing — and it doesn’t require a platform team to maintain.

No bash glue, no half-manual steps, no mismatched languages. Just a single, clean TypeScript codebase that turns a cloud token into a Kubernetes cluster — complete with private networking, firewalls, SSH key handling, and, now, a self-validating K3s deployment.

K3s, TypeScript, and a Budget VPS Walk Into a Datacenter...

The setup provisions a classic three-node setup: one control plane, two workers — all Hetzner CX22s, tucked into a private subnet with proper firewalling. K3s is installed on the control plane first (Traefik disabled, obviously), then the node token is fetched securely and used to join the workers. The kubeconfig is exported and made available to other repos via Pulumi stack outputs. It's the bridge between infra and app layer — and it's finally real.

Under the hood, the SshCli helper now does most of the heavy lifting. It generates per-instance ED25519 keypairs, establishes secure SSH connections (with host key checks, because we’re adults), and runs the install scripts remotely. We treat SSH access like a first-class citizen, not a shell hack.

Testing? Yes. Jest-based tests check not just the resource state, but actual cluster health and node readiness. The CI doesn’t consider the infra “done” unless kubectl get nodes shows green across the board. Worker node flaking? You'll find out fast.

A Refactor Worth the Effort

Bringing in K3s triggered a full project refactor. Hetzner resources were moved into scoped subfolders, config handling was centralized, and the Pulumi wrapper was split into a pulumiCli object with better test ergonomics. Output handling got saner, especially around secrets like node tokens and exported kubeconfig.

The stack is now far more modular — a side effect of solving real-world deployment bugs that only surface when you actually try to join a cluster over a freshly provisioned network with restricted firewalls and tight SSH policies.

One Language to Rule Them All (For Now)

The entire project is TypeScript-based, which keeps context switches minimal and CI/CD straightforward. Everything — from infra definitions to helper functions and tests — is in the same language, same tooling, same mental model. It’s fast, typed, and works.

Eventually, specialized tools (e.g., Go for k8s CRD wrangling or Python for AI-heavy workflows) may make their way in. But today, TypeScript is the lingua franca of kreativarc_infra. And that clarity has helped move faster, especially while solo.

It All Runs for Under €10 a Month

The stack runs on budget-friendly Hetzner instances with no managed services. Firewalls are locked down, tunnels handle exposure (when needed), and there’s zero persistent infra outside the VPS box. CI/CD is handled separately via GitHub Actions, pulling kubeconfig from Pulumi outputs when needed.

You don’t need to spend hundreds per month for a real infrastructure backbone — just automate ruthlessly, test everything, and don’t cut corners on security.

Next stop: modular namespaces, observability stack rework, and possibly a Golang sidecar or two. But for now, it clusters, it connects, and it tests itself. That’s a solid milestone.

But here’s the real win: once your system is described by a single source of truth, that same config can power a complete infrastructure validation suite. Not just deployment, but verification. Every single time. Automatically.

Why bother?

Manually checking if hetzner_server.control_plane has the right server type, image, location, network, and firewall gets old fast — and breaks even faster. Two servers are manageable. Five are annoying. Ten? You’re guessing and hoping.

Even worse, Pulumi's state can drift. Resources may partially apply, fail silently, or get manually edited in the UI. Having a fast, reproducible sanity check is invaluable.

The method

The core idea is simple:

1. Load the config.ts file (the central configuration).
2. Query the actual state of live infrastructure (via hcloud, file system, etc.).
3. Compare the two.
4. If anything diverges: fail fast.

This isn’t snapshot testing. This is state validation. And since the tests derive everything from the config, they’re future-proof: add a new server to the config, and the test suite picks it up automatically. No test rewrites required.

The technical catch

Pulumi’s SDK is designed to run inside the Pulumi runtime. You don’t get rich access to stack data from external scripts. And Jest runs outside that bubble.

So — here comes the hack — we interrogate Pulumi (and Hetzner) using shell commands. Crude, yes. But effective.

Example: get the current stack name using pulumi stack ls:

export function pulumiGetStack(): string {
  const raw = execSync(`pulumi stack ls --json`, { encoding: "utf-8" });
  return JSON.parse(raw).find((s: any) => s.current).name;
}

From there, tests live in tests/*.test.ts, where they read the central config and assert against the real infrastructure.

A test run in action

npm test

> test
> cd src && jest --verbose

 PASS  tests/networkSubnet.test.ts (6.39 s)
  Hetzner network subnet creation
    ✓ subnet exists
    ✓ subnet type matches input
    ✓ subnet IP range matches input
    ✓ subnet network zone matches input

 PASS  tests/network.test.ts (6.574 s)
  Hetzner network creation
    ✓ network exists
    ✓ network IP range matches input

 PASS  tests/sshKey.test.ts (6.577 s)
  SSH Key Generation (homedir only)
    ✓ should create private key for control-plane in ~/.ssh
    ✓ should create public key for control-plane in ~/.ssh
    ✓ private key for control-plane in ~/.ssh should not be empty
    ✓ public key for control-plane in ~/.ssh should not be empty
    ✓ private key for control-plane in ~/.ssh should start with '-----BEGIN'
    ✓ public key for control-plane in ~/.ssh should start with 'ssh-'
    ✓ should create private key for worker-node-1 in ~/.ssh
    ✓ should create public key for worker-node-1 in ~/.ssh
    ✓ private key for worker-node-1 in ~/.ssh should not be empty
    ✓ public key for worker-node-1 in ~/.ssh should not be empty
    ✓ private key for worker-node-1 in ~/.ssh should start with '-----BEGIN'
    ✓ public key for worker-node-1 in ~/.ssh should start with 'ssh-'

 PASS  tests/firewall.test.ts (7.486 s)
  Hetzner Cloud Firewall
    ✓ Firewall exists for server control-plane
    ✓ Firewall rules match for server control-plane
    ✓ Firewall exists for server worker-node-1
    ✓ Firewall rules match for server worker-node-1

 PASS  tests/firewallAttachment.test.ts (8.287 s)
  Hetzner server firewall attachment
    ✓ Server control-plane has the correct firewall
    ✓ Server worker-node-1 has the correct firewall

 PASS  tests/server.test.ts (8.74 s)
  Hetzner server configuration
    ✓ Server control-plane is running
    ✓ Server control-plane has correct server type
    ✓ Server control-plane has correct image
    ✓ Server control-plane has correct location
    ✓ Server control-plane is in the correct network
    ✓ Server worker-node-1 is running
    ✓ Server worker-node-1 has correct server type
    ✓ Server worker-node-1 has correct image
    ✓ Server worker-node-1 has correct location
    ✓ Server worker-node-1 is in the correct network

Test Suites: 6 passed, 6 total  
Tests:       34 passed, 34 total  
Snapshots:   0 total  
Time:        9.08 s, estimated 10 s  
Ran all test suites.

In less than 10 seconds, you know whether:

• The correct network and subnet exist
• SSH keys are present and valid (checked via filesystem, format, contents)
• All servers are running, correctly typed, properly located
• Firewalls are created and attached with the right rules

All of it defined from — and validated against — the config.

Why this matters

• Add a new server? It gets validated.
• Something drifts in Pulumi state? Detected.
• Accidentally delete something from the UI? Caught.
• Firewall rule typo? Fails test.

And most importantly: you get a one-minute feedback loop. Infrastructure either matches config, or it doesn’t.

Final thoughts

Most IaC projects break down because there’s no feedback mechanism. You define, deploy, and hope. This approach gives you a definitive answer every time: yes, it matches; or no, it’s broken.

It’s not pure — some parts are duct-taped together with shell calls — but it works. And once your config is canonical, everything else (deployment, validation, debugging) becomes a matter of diffing reality against expectation.

The goal? Build a minimal, isolated, and reproducible environment on Hetzner Cloud — one that doesn’t burn €10/month per project just to exist. Kubernetes (kreativarc k3s) and the actual applications (poc, infra tools, app1, app2) come later — but none of that works without a clean, affordable foundation.

The first version of this Pulumi-based infrastructure was… verbose. Not in the literary sense — just in the number of files. Each server had its own config scattered across multiple files, with subtle differences that were easy to miss and even easier to break. Adding a new machine felt like assembling IKEA furniture from schematics written in Morse code.

So I did what any developer does after a few rounds of self-inflicted chaos: I centralized.

The new setup revolves around a single inputConfig object — a plain TypeScript file that defines everything needed to bootstrap a secure, reproducible infrastructure stack on Hetzner. One file. One format. No guesswork.

What It Does

The stack provisions an isolated cloud environment — private network, subnets, firewalls, and servers — all invisible to the outside world. There’s no public SSH access, no exposed ports. Cloudflare Tunnel will eventually be the only ingress point.

It automatically:

• sets up a 192.168.x.x internal network

• generates SSH keys locally (and only locally — they never leave your machine)

• creates firewalls that allow only trusted IPs

• provisions servers, each with the correct network, SSH key, and firewall rules

Here’s a stripped-down version of the config example:

{
    projectName,
    stackName,
    adminPublicIp,
    network: {
        type: "cloud",
        ipRange: "192.168.100.0/24",
        networkZone: "eu-central"
    },
    servers: [
        {
            name: "control-plane",
            serverType: "cx22",
            image: "ubuntu-22.04",
            location: "nbg1",
            firewallRules: [
                {
                    direction: "in",
                    protocol: "tcp",
                    port: "22",
                    sourceIps: [`${adminPublicIp}/32`],
                    description: "SSH access"
                }
            ]
        },
        {
            name: "worker-poc",
            serverType: "cx22",
            image: "ubuntu-22.04",
            location: "nbg1",
            firewallRules: [
                {
                    direction: "in",
                    protocol: "tcp",
                    port: "22",
                    sourceIps: [`${adminPublicIp}/32`],
                    description: "SSH access from admin"
                }
            ]
        }
    ]
}

Why It Matters

This config isn’t just for readability. It makes the entire system scalable.

Want to add a new server? Append it to the array.

Want a new environment — say, dev in parallel to prod? Just switch the stack and reuse the config structure. Since everything's parametrized and the naming is stack-aware, resources stay isolated and reproducible.

Clean Infra, Fewer Surprises

By keeping the input structure minimal and declarative, the actual resource logic becomes dead simple. Each Pulumi component (network, firewall, server, SSH key) just reads from this object and builds itself accordingly. There’s no hand-written state, no hardcoded IPs, and no shared secrets.

The only thing you need to provide is your current public IP — passed in securely as a Pulumi secret — to allow SSH access during provisioning.

pulumi config set adminPublicIp $(curl -s ifconfig.me) --secret

Once that’s in, the infra is yours. Clean. Predictable. And, for once, not screaming at you in YAML.

Planned VPS Scaling

While the long-term plan may involve abstracting away VPS management entirely, for now, each node is still a separate, explicitly defined server. That said — scaling the cluster is already trivial.

Thanks to the declarative inputConfig, provisioning a multi-node setup (control plane + multiple workers) takes just a few lines. Need a new worker? Add an entry to the config, pulumi up, and it’s done.

The goal is flexibility: you can spin up a POC node today, and production-grade workers tomorrow — without refactoring the entire infrastructure.

Next up: testing. Because we don’t just trust infra — we verify it.

Modern infrastructure development doesn’t have to involve mountains of YAML or some arcane DSL. If you think like a Node.js developer, then infrastructure as code is just another module — and like every other module, it can (and should) be tested.

Pulumi + Jest = Infrastructure you actually validate

Creating a Hetzner VPS with TypeScript is straightforward:

const vps = new Server(serverName, {
    name: serverName,
    serverType: "cx22",
    image: "ubuntu-22.04",
    location: "nbg1",
    sshKeys: [sshKey.name],
});

Then comes the firewall — with a single open port:

const firewall = new Firewall(firewallResourceName, {
    name: firewallResourceName,
    rules: [
        {
            direction: "in",
            protocol: "tcp",
            port: "22",
            sourceIps: ["0.0.0.0/0", "::/0"],
            description: "Allow SSH from anywhere",
        },
    ],
});

The entire infra update process follows a Node-style workflow:

"scripts": {
    "preview:prod:critical:vps": "pulumi preview --cwd critical/vps --stack prod",
    "prod:critical:vps": "pulumi up --cwd critical/vps --stack prod --yes",
    "test": "jest"
}

Testing: not just for applications

Running jest gives you immediate feedback on whether the infrastructure is actually functioning. We’re not just testing whether “the script ran,” but whether the result is alive and accessible.

Terminal output:

> test
> jest

 PASS  critical/vps/vps.test.ts (14.711 s)
  Hetzner VPS basic checks
    ✓ server is running and has a public IP (3 ms)
  Expected open ports
    ✓ port 22 should be open (124 ms)
  Expected closed ports
    ✓ port 21 should NOT be open (1148 ms)
    ✓ port 23 should NOT be open (1029 ms)
    ✓ port 25 should NOT be open (1037 ms)
    ✓ port 80 should NOT be open (1037 ms)
    ✓ port 443 should NOT be open (1034 ms)
    ✓ port 3306 should NOT be open (1037 ms)
    ✓ port 5432 should NOT be open (1036 ms)
    ✓ port 6379 should NOT be open (1035 ms)
    ✓ port 11211 should NOT be open (1035 ms)
    ✓ port 8080 should NOT be open (1035 ms)
    ✓ port 9000 should NOT be open (1034 ms)
    ✓ port 9100 should NOT be open (1036 ms)
    ✓ port 9200 should NOT be open (1070 ms)

Test Suites: 1 passed, 1 total  
Tests:       15 passed, 15 total  
Snapshots:   0 total  
Time:        14.739 s, estimated 16 s  
Ran all test suites.

Why does this matter?

Because a pulumi up is not a guarantee that your server is alive, reachable, or secure. With this approach:

• Smoke tests confirm that the VPS is created and has a public IP
• Positive port checks ensure the intended ports are accessible
• Negative port checks verify that everything else stays closed

The whole setup gives you feedback within 15 seconds: the server is alive, nothing unnecessary is exposed, and you’re good to deploy.

If your infrastructure is just another Node module, feel free to treat it like one. Here, npm run test doesn’t just validate code — it validates the entire environment your app is supposed to run on.

The goal wasn't a full-scale observability platform, just a minimal yet practical setup tailored to actual needs.

Objectives

• Monitor VPS system-level resources (CPU, memory)

• Track container-level resource usage (CPU, memory)

• Collect and label Docker logs

• Isolate application errors from background noise (Traefik, monitoring, tunneling)

• Display all this in a clean, unified Grafana dashboard

Architecture Overview

Each component runs in its own Docker container, inside a dedicated Docker network, without exposing any public ports. All interfaces are available behind Cloudflare Tunnel with Access protection.

Metrics Stack

• node-exporter – host-level metrics (localhost:9140)

• cAdvisor – container-level metrics (localhost:9130)

• Prometheus – time-series collection + scraping (localhost:9100)

Logging Stack

• promtail – reads logs from the Docker socket (not the log driver)

• Loki – log storage with dynamic labeling (localhost:9110)

Visualization

• Grafana – preconfigured with Prometheus and Loki datasources (localhost:9120)

• One main dashboard: system metrics and application errors

  

  

Public Access

All UI components are exposed via Cloudflare Tunnel using subdomains:

• grafana.kreativarc.com

• prometheus.kreativarc.com

Containers communicate via Docker-internal network aliases. Logs are automatically labeled with container name, image, and service for easier querying and filtering.

Challenges

Setting it up took more time than expected. The tools themselves are well-documented — the issues mostly came from the surrounding environment:

• Docker’s internal DNS and service discovery quirks

• Cloudflare Tunnel routing and token management

• Grafana datasource mappings occasionally failing silently

In the end, I managed to isolate container-level issues and error spikes in real time, while keeping the noise from infrastructure logs to a minimum.

Conclusion

This monitoring setup provides a clean, focused observability solution for a single VPS. It avoids unnecessary complexity (no sidecars, no operators, no Kubernetes) while offering real visibility into host and container-level issues.

Not the most ambitious system — but practical, reliable, and enough to catch what's going wrong before it escalates.

1. Server & DNS Setup

Hetzner VPS (CX22)

• Ubuntu base image
• Clean slate environment — no preinstalled surprises

Cloudflare DNS (Proxy Mode)

• DNS with built-in DDoS mitigation
• IP masking and active edge protection

2. Core Stack: Half-Manual, Half-IaC

Instead of full automation from day one, I opted for a hybrid approach: docker-compose files organized in a central repo, with services running entirely in Docker for simplicity and consistency.

Deployment Approach

Initial deployments used GitHub Actions, but due to permission complexity and audit concerns, I switched to manual root@vps deployments. Not elegant, but stable and predictable.

Main Components

Traefik Reverse Proxy

• Automatic HTTPS via Let’s Encrypt
• Dynamic subdomain routing
• Docker integration
• Dashboard: http://localhost:9000

Cloudflare Tunnel

• One tunnel for public-facing services
• One tunnel for internal/admin interfaces
• All sensitive admin UIs are accessible only through tunnel routing

Shared Services

• PostgreSQL and Temporal
• Shared across all PoC apps, with future isolation options for scaling

Monitoring Stack (Separate blog post coming soon)

• Prometheus, Grafana, Loki, Promtail
• Host metrics via node-exporter, container metrics via cAdvisor
• Logs collected via Docker socket (not log drivers)
• Preconfigured Grafana dashboards at http://localhost:9120
• Ready for AI-based log analysis and debugging bottlenecks

3. Debug-Driven DevOps

I tested and debugged every service manually in the terminal — not just to make it work, but to understand why it works (or fails). This included:

• Verifying Traefik certificate renewals
• Troubleshooting Cloudflare tunnel and DNS behavior
• Dealing with Loki label configuration and Promtail edge cases

4. Snapshot → Wipe → Rebuild

After validating the entire stack, I created a snapshot and wiped the server. Clean state, no cruft.

What’s next?

Rebuilding the same infrastructure with Pulumi, with Kubernetes likely following later. For now, the goal is controlled complexity — and a better understanding of where automation makes sense.

TL;DR

Built an initial infrastructure setup with:

• Traefik for HTTPS and routing
• Docker-based monitoring stack (Prometheus, Grafana, Loki)
• Temporal for orchestrated background workflows
• PostgreSQL for shared persistence
• Cloudflare Tunnel for secure admin access
• Manual root deployments (IaC in progress)

Next step: Pulumi-based automation, and eventually Kubernetes — but only when the payoff outweighs the extra complexity.

That’s more or less what I do:

• explore new ideas
• build functional prototypes
• and refine everything from infrastructure to frontend — while documenting what I learn

Why self-host?

Because two things matter to me: learning and control.
This isn’t a throw-it-on-Vercel-and-forget-it project.
It’s a low-cost, fully owned stack — designed from the ground up to support rapid experimentation without vendor lock-in.

Design goals:

• Cost-efficient. Hetzner VPS + Docker is orders of magnitude cheaper than AWS for small projects
• Fast to deploy. I want to push PoCs weekly — not spend weeks wrestling CI/CD
• Scalable. The stack should support growth if an idea gains traction
• Secure. Admin interfaces are never exposed publicly. Cloudflare Tunnel + Access takes care of that

What’s under the hood?

If it can be done in TypeScript and augmented with AI, I’m likely doing it that way.

Frontend

• Next.js + MUI – solid UI performance and component consistency
• tRPC + React Query – end-to-end type safety, no boilerplate REST

Backend

• Node.js + tRPC + Zod – type-safe APIs and clear validation
• PostgreSQL + Prisma + pgvector – relational + vector search combined
• LLM integration:
  • OpenAI
  • Langchain
  • LangGraph

Use cases:

• Document ingestion and preprocessing
• Agent-driven workflows
• Structured outputs from unstructured content

Testing

• Jest for unit and integration tests
• AI-based E2E agent for validating non-deterministic responses

🔁 Workflow orchestration

• Temporal handles:
  • retries
  • scheduling
  • long-running pipelines
  • observability and isolation

Example workflows:

• Scraping new URLs
• LLM-powered data extraction and embedding
• Updating vector DBs
• Generating structured content from semi-structured sources

What to expect from this blog

No clickbait. No “10x your productivity with VSCode extensions” fluff.
Just:

• real PoC applications with AI integration
• real-world infrastructure setups
• honest lessons from hands-on builds