01
Data platforms
Modernizing Spark infrastructure, helping build Apache Iceberg infrastructure, and making a $6–7M/year Spark fleet’s costs visible.
Yiping Deng
Tech Lead II · AI Data · HubSpot
Making AI work.
At scale.
I build the data platforms behind AI at HubSpot — from distributed systems and vector search to GPU infrastructure and observability.
✳Based in Dublin, Ireland.
Building for the bigger picture.
Building for the bigger picture.
A builder’s perspectiveFig. 01
Behind every intelligent app,
there’s a good system.
there’s a good system.
Selected impact
7B+production ANN vectorsRAG & semantic search
80%less Spark support loadAI-agent job auto-tuning
2 TBingested every dayAI inference & feedback
$90K → $5Kmonthly storage costHBase migration
01 / What I build
The foundations
behind the possibilities.
Reliable platforms. Thoughtful abstractions.
Room for the next big idea.
02
AI infrastructure
Scaling vector search to 7B+ production ANN vectors across 200+ indices, and building inference and feedback pipelines that ingest 2 TB/day.
03
Observable systems
Building OpenTelemetry tracing and storage for all AI agents at HubSpot, and using AI agents to analyze data-entity lineage graphs.
Selected projects / GitHub
Built out of curiosity.
A few things I’ve built outside the day job — from GPU experiments and developer tools to useful little systems for everyday life.
GPU learning lab
Learning CUDA
A hands-on GPU programming notebook: CUDA matrix-multiplication kernels, Triton exercises, and custom PyTorch extensions.
Agent tooling
Codex Tool Mock
Mock shell-tool responses and verify expected commands with reusable fixtures, exact or regex matching, and pytest helpers.
Data tooling
parquet-cat
A Rust CLI for exploring Parquet files on S3, including folders, SQL queries through DataFusion, and AWS profile support.
Developer experience
Codebase Prompt Generator
Turn a Git repository into structured AI context, with a file tree, source contents, include/exclude patterns, and .gitignore support.
Hardware experiment
Kindle LLM Usage
An e-ink dashboard for LLM usage and local coding-agent activity, with touch navigation, cached views, and explicit stale-data states.
Everyday automation
CatLink CLI
A Python CLI for CatLink smart litter boxes: device status, logs, cleaning controls, and authentication backed by the system keyring.
02 / The journey
Built over time.
Jun 2024 — PresentCurrent
HubSpot / Dublin, Ireland
Tech Lead II · AI Data
- Own and modernize Spark infrastructure; introduced AI-agent job auto-tuning, cutting Spark support load by 80%.
- Built cost tracking for a $6–7M/year Spark fleet and helped build HubSpot’s Apache Iceberg infrastructure.
- Built an OpenTelemetry tracing and storage system covering all AI agents at HubSpot; use AI agents to analyze data-entity lineage graphs.
- Scaled vector infrastructure for RAG and semantic search to 7B+ production ANN vectors; current infrastructure supports 200+ indices.
- Built AI inference/feedback pipelines ingesting 2 TB/day; introduced JupyterHub and NVIDIA DGX H100 infrastructure.
- Grew the team to six people and helped establish two spin-out teams focused on JupyterHub and vector databases.
Mar 2023 — Jun 2024
HubSpot / Dublin, Ireland
Tech Lead I · AI Data and MLOps
- Maintained a feature store aggregating internal APIs for AI model serving; implemented compliance workflows and scaled AI data collection across multiple data centres.
Jan 2022 — Mar 2023
HubSpot / Dublin, Ireland
Senior Software Engineer I · AI Infrastructure
- Cut monthly AI inference and feedback storage costs from $90K to $5K by migrating away from HBase.
Oct 2020 — Jan 2022
HubSpot / Dublin, Ireland
Software Engineer · AI Infrastructure
- Maintained Kubernetes GPU infrastructure serving 1B+ AI inferences/day and large-scale HBase storage for inference and feedback data.
- Standardized automated training pipelines for 6+ ML models, retrained weekly using user feedback.
Earlier experience
Sep 2019 — Aug 2020
HubSpot / Dublin, Ireland
Software Engineer Intern · AI Infrastructure
- Implemented automated metrics and performance-tracking pipelines for ML models in the AI Infrastructure team.
Jun 2019 — Aug 2019
Amazon / Luxembourg
Software Development Engineer Intern
- Built an internal UI for technical program managers to review machine-translated product catalogue content.
Feb 2019 — May 2019
Jacobs University Bremen / Bremen, Germany
Research Assistant · Marine and Robotics Systems
- Researched autonomous-driving algorithms using ROS, OpenCV and PyTorch.
Jun 2018 — May 2019
Ubimax / Bremen, Germany
Software Engineer · Working Student
- Built a SqueezeNet training microservice for automotive use cases; deployed models to Google Glass Enterprise and Vuzix M300 with TensorFlow Lite.
03 / A little about me
Curiosity is
the common thread.
My work sits where data infrastructure meets machine learning: the systems that collect, store, retrieve, and make sense of data at scale. At HubSpot, I’ve grown from an AI infrastructure engineer into a tech lead, building platforms and teams along the way.
I’m also a contributor to open-source infrastructure and a co-author of research in formal theorem proving. From mathematical foundations to production systems, I like understanding how things work — and making them work better.
Find me on GitHubBSc Computer Science
Minor in Intelligent Systems
Jacobs University Bremen
Coursework in machine learning, algorithms, data structures, operating systems, and computer vision. Research in formal logic and marine robotics.
Languages & ML
Data & infrastructure
Areas
04 / Beyond the day job
Shared with the community.
Open source
Qdrant
4 merged PRsStrengthened transport security with peer-to-peer mutual TLS, HTTPS client-certificate validation, and live TLS certificate rotation. Added regression coverage for recreating collections in cluster mode.
Open source
Apache Spark
Accepted upstream + 4.0 / 4.1 backportsFixed repeated History Server compaction failures on malformed UTF-8 event logs (SPARK-59747). The server records the evaluated log index and warns once, retrying only after a new file rolls; regression tests preserve application listing and leave malformed logs uncompacted.
Open source
OpenMetadata
Merged PRAdded the Cache Warmup application’s JSON schema to the UI, fixing a configuration editor that displayed undefined. The schema exposes entity selection, batch size, consumer threads, queue size, and the force option.
Research / May 2018
Hilbert Meets Isabelle: Formalisation of the DPRM Theorem in Isabelle
Co-author · EasyChair Preprint 152
Read the publication05 / The notebook
Thinking out loud.
Notes on code, computer science, and the ideas underneath. New thinking, alongside the archive.
Research
Hilbert’s tenth: from machines to equations, and a new rational-case claim
How integer equations encode computation, what formalization checks, and how to read OpenAI’s model-generated claim over the rationals.
16 min read
Web development
React With Higher Order Component
Mixing data-fetching behavior into React components with a reusable higher-order component.
2 min read
Functional programming
Learn Computer Science From Scratch -- Map and Reduce in Function world
A hands-on introduction to map, reduce, and counting characters with Haskell.
3 min read
Good things start with a conversation.
Let’s connect.
Talk systems, trade ideas, or just say hello.
[email protected]Dublin, Ireland