I have built GPU-accelerated LLM inference infrastructure, with a particular interest in making modern AI runtimes work reliably on hardware and environments with real constraints.

My work spans:

• vLLM and PyTorch inference
• CUDA runtime compatibility
• NCCL and multi-GPU execution
• llama.cpp / GGUF inference
• GPU and LLM observability
• Kubernetes / k3s GPU infrastructure
• C++/CUDA distributed systems
• OpenTelemetry, Prometheus and Grafana
• reproducible benchmarking and runtime diagnostics

A recurring theme across my projects is simple:

Can we make sophisticated LLM infrastructure work predictably on the GPU hardware developers actually have?

Run upstream vLLM reproducibly on Kaggle's dual NVIDIA Tesla T4 GPUs.

kaggle-vllm is a lightweight compatibility and runtime-delivery toolkit for running upstream vLLM inside Kaggle notebooks without replacing Kaggle's preinstalled CUDA/PyTorch stack.

It tackles a practical problem: vLLM is built primarily for modern production GPU environments, while Kaggle provides a tightly controlled notebook environment with its own Python, PyTorch, CUDA, NCCL and storage constraints.

The project validates and organizes that environment instead of pretending those constraints do not exist.

Current capabilities include:

• validated upstream vLLM runtime delivery;
• 2 × NVIDIA Tesla T4 / SM75 execution;
• CPython/PyTorch/CUDA compatibility checking;
• checksum-verified native runtime artifacts;
• safe pip --target runtime staging;
• NCCL two-rank execution;
• vLLM tensor parallelism with TP=2;
• single- and dual-GPU inference benchmarks;
• Qwen2.5-3B FP16 sharded_state save/reload;
• OpenAI-compatible vLLM serving;
• GPU topology and runtime diagnostics;
• reproducible evidence and benchmark artifacts.

Kaggle Notebook
     │
     ├── Python / PyTorch / CUDA compatibility
     │
     ▼
kaggle-vllm SDK
     │
     ├── validated native vLLM runtime
     ├── environment diagnostics
     ├── runtime staging
     ├── benchmark tooling
     │
     ▼
Upstream vLLM
     │
     ├── GPU 0 ──┐
     │            ├── NCCL / TP=2
     └── GPU 1 ──┘
          │
          ▼
     LLM inference
          │
          ├── sharded checkpoints
          └── OpenAI-compatible API

Why it matters: inexpensive notebook GPUs can become useful environments for learning about real inference-runtime behavior—including CUDA compatibility, distributed execution, checkpoint loading and multi-GPU communication—rather than only high-level model APIs.

➡️ Explore kaggle-vllm

Private local LLM inference + Kubernetes-native GPU operations + observability.

Edge Computing LLM explores how the infrastructure patterns normally associated with cloud LLM deployments can be brought to constrained Linux systems and small NVIDIA GPUs.

The project separates the platform into explicit layers instead of building one large application.

flowchart LR
    U[Operator] --> C[edge-cli]

    C --> K[k3s-nvidia-edge]
    C --> O[llm-observability-stack]

    K --> G[NVIDIA GPU Runtime]
    O --> L[Local LLM Runtime]

    G --> T[Telemetry]
    L --> T

    T --> OT[OpenTelemetry]
    OT --> P[Prometheus]
    P --> GR[Grafana]

The reference environment deliberately includes low-VRAM NVIDIA hardware, CPU fallback, local models and single-node k3s.

This is not an attempt to imitate an unlimited cloud cluster.

It is an exploration of how much of the modern LLMOps stack can remain useful when compute, VRAM and infrastructure are limited.

➡️ Explore Edge Computing LLM

CUDA-first Python SDK for GGUF inference and LLM observability.

llamatelemetry grew out of a practical problem I encountered while using llama.cpp tooling for CUDA LLM experiments:

inference worked, but organizing models, runtime state, GPU metrics, server processes and benchmark evidence quickly became messy.

The project turns that workflow into a Python SDK.

Python application
       │
       ▼
llamatelemetry
       │
       ├── InferenceEngine
       ├── ServerManager
       ├── Model Registry
       ├── GGUF metadata
       └── Telemetry
               │
       ┌───────┴────────┐
       ▼                ▼
   llama.cpp         NVML / OTEL
       │                │
       ▼                ▼
 CUDA inference      GPU metrics
       │                │
       └───────┬────────┘
               ▼
        Observable LLM run

It includes:

• high-level GGUF inference;
• llama-server lifecycle management;
• OpenAI-compatible client access;
• model registry and metadata parsing;
• quantization helpers;
• GPU metrics collection;
• OpenTelemetry instrumentation;
• Kaggle dual-T4 presets;
• CUDA/C++ components;
• notebook-based reproducible workflows.

➡️ Repository · Documentation

Beyond the three main projects, I use smaller repositories to explore individual pieces of the LLM infrastructure stack.

                         LLM INFERENCE
                              │
          ┌───────────────────┼───────────────────┐
          │                   │                   │
          ▼                   ▼                   ▼
       PyTorch              GGUF              Serving
        vLLM              llama.cpp          OpenAI APIs
          │                   │                   │
          └───────────────┬───┴───────────────────┘
                          │
                          ▼
                    GPU RUNTIME
                          │
             CUDA · NCCL · NVML · MPI
                          │
                          ▼
                SYSTEMS INFRASTRUCTURE
                          │
             Linux · k3s · Kubernetes
                          │
                          ▼
                     OBSERVABILITY
                          │
        OpenTelemetry · Prometheus · Grafana
                          │
                          ▼
             REPRODUCIBLE EXPERIMENTS

I am particularly interested in the boundary between the AI model and the system underneath it:

• Why does an inference runtime fail on one CUDA environment and work on another?
• When does tensor parallelism actually help?
• What does NCCL communication cost on limited GPUs?
• How should model binaries and checkpoints be distributed reproducibly?
• What telemetry is necessary to understand an inference failure?
• How much production-style infrastructure can run on small or inexpensive hardware?
• How can experiments preserve enough evidence that somebody else can reproduce the result?

CUDA · NCCL · NVML · MPI · CMake · Ninja · TCP/epoll

k3s · containerd · Helm · OpenTelemetry · Prometheus · Grafana

I am currently concentrating my open-source work around three layers:

1. PyTorch LLM inference on constrained GPUs → kaggle-vllm

2. Local / edge LLM infrastructure → Edge-Computing-LLM

3. CUDA-first inference observability → llamatelemetry

Together, these projects explore the same larger problem from different levels:

making GPU inference reproducible, understandable and useful outside large managed GPU clusters.

I am interested in collaborating on:

• LLM inference infrastructure
• vLLM
• CUDA and NVIDIA GPU systems
• inference performance engineering
• distributed inference
• GPU observability
• developer tooling for AI infrastructure
• constrained / edge GPU deployments

GitHub · LinkedIn · llamatelemetry docs

Build for the hardware you have. Measure what actually happens.