HKContact
§ 01 — Biography

Harsh Kaushik

AI Systems · LLM Training · Inference Acceleration

My work centers on the intersection of deep learning algorithms and computer systems. I believe that building models from scratch and reproducing landmark research papers from first principles is the most reliable way to develop deep intuition for architecture scaling limits and performance bottlenecks.

Rather than treating model structures as black boxes, my research focuses on the entire lifecycle—from BPE tokenization choices and data mixture hygiene during pretraining, up to low-rank SFT alignments and double-buffered memory schedules at the hardware boundary during inference serving.

Research & Implementation Philosophy

I hold a data-centric and systems-first outlook: most model quality limits are bound to data curation, while model serving constraints are bound to memory bottlenecks. Hence, grounding SFT targets in concise provision summary structures prevents speculation, while optimizing PCIe expert weight schedules enables large Mixture-of-Experts (MoE) runtimes to operate efficiently on limited hardware.

How I Learn

"My approach to learning is to read research papers, reproduce their core ideas from scratch, analyze structural limitations, and apply those insights to my own systems. Every major project in my portfolio is rooted in literature, but extends beyond passive reading through active implementation, benchmarking, and software engineering."

Long-term Goals

I aim to optimize the layers between transformers and silicon. My long-term goal is to design lightweight, high-utility foundation models and serving systems that make local, private, and latency-optimal inference runtimes accessible on everyday consumer devices.

FocusLocal foundation models & serving runtimes
MethodologyFirst-principles implementation & data-centric alignment
LocationIndia · Open to hybrid/remote setups
§ 01 — Research statement

On the study of language models as systems — from tokenizer to server.

01

Small Language Models

Training compact models that reach outsized quality per parameter through careful data curation and architecture choices.

02

Efficient Inference

KV-cache layout, paged attention, quantization, and batching strategies that make LLMs practical on consumer hardware.

03

Transformer Architecture

Rotary embeddings, grouped-query attention, MoE routing — studying what makes modern decoders scale.

04

Training Language Models

Pretraining pipelines from tokenizer to optimizer state: reproducible, checkpointable, and observability-first.

05

Dataset Engineering

Cleaning, deduplication, and mixture design. Data is the model — most of the quality lives here.

06

Model Compression

LoRA, QLoRA, quantization-aware fine-tuning, and pruning as a systems problem, not just a numerics one.

07

LLM Systems

Serving stacks, throughput/latency trade-offs, memory hierarchies, and the plumbing behind production inference.

08

Research Reproduction

Re-implementing landmark papers from scratch to internalize the design decisions behind them.

§ 06 — Technical Growth Timeline

Progression trajectory

An interactive path showing progression from classical machine learning foundations up to custom pretraining pipelines and inference system runtimes. Click any milestone to expand and inspect detailed research parameters.

  1. Inference SystemsCollapse

    Turbo-LLM — mixture-of-experts acceleration

    Focusing on hardware-software boundaries to run huge models on consumer units. Built SSD ➔ RAM ➔ VRAM sequential expert offloading, double-buffering, and active parameter pinning in GPU cache memory to bypass PCIe bandwidth limits.

    ProjectTurbo-LLM
    Research TopicMemory-efficient inference scheduling & PCIe transfer pipelining
    Key Lesson Learned

    "Double-buffering memory transfers hides weight loading latency behind calculations, boosting throughput of single-GPU offloaded MoE setups by 21x."

    Influential LiteratureMixtral of Experts (MoE) Paper, AirLLM, vLLM
  2. Training from ScratchExpand

    TinyStories-17M — pretraining pipeline

    Trained a 17.2M parameter decoder-only transformer from first principles. Built custom dataset streams, SentencePiece BPE tokenizer, RMSNorm pre-layer normalization, SwiGLU activation blocks, and Rotary Position Embeddings (RoPE).

  3. LLM Fine-tuningExpand

    Indian Legal LLM & MathInstruct — supervised fine-tuning

    Studied SFT post-training workflows. Fine-tuned Qwen 2.5 (0.5B) and Qwen 3 (1.7B) on custom legal and mathematical instruction corpora. Discovered and documented the limits of verbose datasets on hallucination thresholds.

  4. Knowledge DistillationExpand

    Spam Detection Model — model compression

    Developed model compression pipelines. Trained an 80M parameter residual teacher network and successfully distilled its soft logits representation into a lightweight 2M parameter self-attention student model.

  5. Machine LearningExpand

    Foundations — classical modeling

    Established engineering baselines. Built text classifiers using TF-IDF feature extraction, Logistic Regression, and support vector machines (SVM) with class-weighted loss balancing.

§ 03 — Toolkit

Systems, frameworks, and craft

Languages
01
PythonC++CUDABash
Frameworks
02
PyTorchTransformersvLLMPEFTTRL
Training
03
PretrainingSFTLoRAQLoRATokenizer TrainingMoE
Inference
04
KV CachePaged AttentionQuantizationSpeculative DecodingBatching
Data
05
ParquetMinHashMultiprocessingSentencePiece
Platform
06
LinuxHugging FaceDockerWeights & Biases