research · systems · language models

Harsh Kaushik

AI Systems · LLM Training · Efficient Inference

I build language models from scratch, reproduce modern LLM research, optimize inference systems, and explore efficient AI architectures.

View Projects GitHub Hugging Face

17.2M

params trained

2M+

dataset examples

4×

inference speedup

10+

papers reproduced

§ 01 — Research statement

On the study of language models as systems — from tokenizer to server.

Small Language Models

Training compact models that reach outsized quality per parameter through careful data curation and architecture choices.

Efficient Inference

KV-cache layout, paged attention, quantization, and batching strategies that make LLMs practical on consumer hardware.

Transformer Architecture

Rotary embeddings, grouped-query attention, MoE routing — studying what makes modern decoders scale.

Training Language Models

Pretraining pipelines from tokenizer to optimizer state: reproducible, checkpointable, and observability-first.

Dataset Engineering

Cleaning, deduplication, and mixture design. Data is the model — most of the quality lives here.

Model Compression

LoRA, QLoRA, quantization-aware fine-tuning, and pruning as a systems problem, not just a numerics one.

LLM Systems

Serving stacks, throughput/latency trade-offs, memory hierarchies, and the plumbing behind production inference.

Research Reproduction

Re-implementing landmark papers from scratch to internalize the design decisions behind them.

§ 02 — Featured work

Selected projects

4 entries

OngoingFeatured

TurboLLM

Efficient LLM inference on consumer GPUs

A research playground for pushing decoder throughput on a single consumer GPU. Explores paged KV cache, fused kernels, speculative decoding, and mixed-precision serving paths.

›3.8× throughput vs. baseline HF pipeline at 4k context
›Custom paged-KV allocator with < 3% memory overhead
›End-to-end benchmark suite across 6 model families

C++CUDAPyTorchvLLMTriton

Read Case Study

turbollm · bench.log

$ turbollm bench --model llama-3.1-8b --ctx 4096
[init]  paged-kv: 96 blocks · 3.4 GB
[load]  fp16 weights ..... 4.2 s
[warm]  step 8/8 ......... 512 tok/s

baseline (hf transformers)  ...  138 tok/s
turbollm (paged + fused)    ...  527 tok/s
speedup                      ...  3.82×

memory / seq    ↓ 41%
p50 latency     ↓ 63%
p99 latency     ↓ 58%

live · updated nightly

Completed02

TinyStories-17M

A 17.2M-parameter language model, from scratch

Full pretraining stack: custom BPE tokenizer, dataset curation, dataloader, transformer decoder, training loop, and evaluation. Published weights and eval on Hugging Face.

›17.2M-param decoder trained on a 2M-example corpus
›Custom tokenizer + preprocessing pipeline
›Reproducible training config and eval harness

PyTorchSentencePieceParquetHugging Face

Read Case Study ↗

Research03

LLM Post-training

SFT / LoRA / QLoRA across specialized domains

Adapter-based fine-tuning experiments across math, code, and legal corpora. Focus on data mixture, LoRA rank/target module ablations, and eval alignment.

›LoRA/QLoRA sweeps across ranks 4–128
›Domain-specialized checkpoints: math, code, legal
›Eval harness spanning 8 downstream tasks

PyTorchPEFTTRLbitsandbytes

Read Case Study ↗

Ongoing04

Dataset Engineering

Instruction data pipelines at scale

Tooling for large-scale text preprocessing: cleaning, near-duplicate detection, quality filtering, and Parquet-native pipelines that stream from disk.