Darpan Jiyani

Data Engineer + AI / San Jose, CA

Darpan Jiyani

I build data pipelines and the AI tools that run on top of them. Right now I'm a Data Engineer at UL Solutions, turning RF test data into Delta tables, dashboards and a review app.

Before that I worked on wafer monitoring at Keysight and boiler sensor telemetry at Cerebulb. MS in Applied Data Intelligence, San Jose State University.

Download resume PDF See projects

What changed after I shipped it

  1. Test-cycle analysis UL Solutions, across 60+ RF test stations
  2. Power BI refresh Keysight, on 2M+ records
  3. Manual test review UL Solutions, on 1M+ RF test records
  4. Text-to-SQL accuracy AI Data Agent, 47 questions, average of 3 runs

Experience

RF test data, semiconductor manufacturing and industrial IoT.

May 2026 to present Fremont, CA

Data Engineer UL Solutions

RF test data for hardware compliance testing

  • Cut end-to-end test-cycle analysis from 2 days of Excel work to 30 minutes by automating RF test-data collection and visualization with Python across 60+ test stations
  • Delivered a Python and React validation app that analyzes 1M+ RF test records and 1,000+ plots per test, cutting manual review from ~30 minutes to ~5 seconds
  • Unified 12+ RF test-case formats with different column sets into one queryable Delta table using Delta Lake mergeSchema, so new test types don't break the pipeline
  • Wrote PySpark transformations and Spark SQL with joins and window functions. Date partitioning cut RF query time by 20%
  • Developed 5+ Tableau dashboards on Delta tables that show RF test results to 3 client teams and 20+ internal engineers
May 2025 to Dec 2025 Santa Rosa, CA

Data Science Intern Keysight Technologies

Wafer monitoring and shift reporting across three production lines

  • Reduced Power BI refresh time from 15 min to under 15 sec on 2M+ records with incremental ETL and timestamp-based change detection, so a 30-person team could monitor wafers in near real time
  • Implemented SQL status-change detection with Python alerts, cutting operator notification from 2 to 3 hours down to under 5 minutes
  • Deployed Dockerized Airflow DAGs on 15-minute schedules with task retries and email failure alerts
  • Designed a Power BI monitoring dashboard showing tool status (up, down, at limit) across three production lines, so operators could spot at-limit equipment before downtime
  • Saved 30+ minutes per shift for 50+ supervisors by automating shift reports with Power Automate, which also removed formatting errors
  • Shipped a Power BI (DAX) dashboard that flags underperforming shifts against rolling baselines, replacing manual review with targeted root-cause investigation
Jan 2022 to Dec 2023 Gujarat, India

Data Engineer Cerebulb

Industrial IoT telemetry from boiler sensors

  • Engineered an EventBridge-scheduled Lambda pipeline that fetches telemetry from 100+ boiler sensor APIs every minute and stores it in S3 for near real-time monitoring
  • Transformed 500GB+ of raw sensor data from S3 to Redshift with daily PySpark jobs on EMR for analyst dashboards
  • Modeled boiler KPIs into a Redshift star schema with fact and dimension tables for Tableau reporting
  • Partnered with plant managers on Tableau monitoring dashboards that flagged failures earlier and cut unplanned downtime

Featured projects

The four I would show first. Each one links to its code.

99.3% text-to-SQL execution accuracy, up from 89.4%

Multi-agent text-to-SQL

AI Data Agent

Ask a data question in plain English and get the answer, a chart and the exact SQL. Three layers of checks keep the database read-only.

questionClaude routerSQL agent or pandas ETL agentsqlglot guard + LLM judgePostgreSQL (read-only)

Build detailsHide details
  • Built a LangGraph multi-agent system where Claude routes plain-English questions to a SQL agent or a pandas ETL agent
  • Raised text-to-SQL execution accuracy from 89.4% to 99.3% on 47 questions, and scored 100% on 2 held-out sets of 20
  • Blocked 22 of 22 SQL write attempts with a sqlglot guard, an LLM judge and a read-only PostgreSQL role
  • Added an Open-Meteo to PostgreSQL ETL agent with human-approved, hash-checked loads. Weather-query accuracy went from 0 of 12 to 11 of 12
  • Cut billed input tokens per question by ~59% with prompt caching, and got follow-ups right in 32 of 32 chats
  • Wrote a 59-question eval harness that scores execution accuracy and LLM-as-a-Judge answers

The live demo runs in demo mode: SQL questions only, a read-only database user and 15 questions per visit. The ride data is synthetic. The app sleeps when idle, so the first visit may need a moment to wake it.

Stack Python, LangGraph, LangChain, Claude, PostgreSQL, sqlglot, pandas, Pydantic, Streamlit

28-second recording: a question, a follow-up, and a delete request the agent refuses.
10 cities of live ride events, merged with a 2,000-ride backfill

Streaming lakehouse

Real-Time Ride Analytics

Ride events stream from a FastAPI app into a Databricks lakehouse, with quality rules, a quarantine table, tests and CI around the pipeline.

FastAPI producerAzure Event Hubs (Kafka)Lakeflow pipelinebronze, silver, gold Deltahourly city KPI view

Build detailsHide details
  • Streamed ride events from a FastAPI app through Azure Event Hubs into a Databricks Lakeflow pipeline over Kafka
  • Added data quality rules that drop bad rides and a quarantine table that keeps unparseable events with their Kafka offset
  • Built SCD Type 1 and 2 dimensions with Auto CDC, ordered by event time so late events never overwrite newer rows
  • Merged a 2,000-ride backfill and the live stream into one table, and added pytest tests and GitHub Actions CI

Started from Ansh Lamba's Uber data engineering tutorial. The CDC sequencing fixes, quality rules, quarantine table, tests, CI and Databricks Asset Bundle deploy are my additions.

Stack Python, FastAPI, Azure Event Hubs, Spark Structured Streaming, Lakeflow Declarative Pipelines, Delta Lake, Unity Catalog, Auto CDC, pytest, GitHub Actions

70% pass@1 on DS-1000 for the fine-tuned code model, 60% on HumanEval

MS capstone, San Jose State, Dec 2025

Simulation of Domain Expert in LLM Models

Five fine-tuned specialist LLMs for math, code and theory questions, served behind a router that sends each question to the right one.

15 LLMs benchmarked5 fine-tuned with QLoRA on GCPFastAPI service on Cloud Runrouter picks the specialist

Build detailsHide details
  • Benchmarked 15 LLMs on GSM8K, MATH, HumanEval and DS-1000 to pick specialist models for math, code and theory
  • Fine-tuned 5 specialist LLMs with QLoRA on GCP. The code model reached 60% pass@1 on HumanEval and 70% on DS-1000
  • Deployed the models behind a FastAPI service on Cloud Run with a router that sends each question to the right specialist
  • Created held-out evaluation datasets and scored fine-tuned models against zero-shot baselines with LLM-as-a-Judge

Stack Python, QLoRA, GCP, Cloud Run, Google Cloud Storage, FastAPI, React

10 city-level views feeding Databricks dashboards and Genie

Streaming ingestion and CDC

Transportation Analytics Lakehouse Pipeline

Trip data for 10 cities lands in Delta tables as a stream, passes quality checks and CDC upserts, then feeds views for each city team.

AWS S3Auto LoaderLakeflow pipelineDelta + Change Data Feedgold fact viewdashboards + Genie

Build detailsHide details
  • Ingested trip data from S3 as a stream with Databricks Auto Loader and Lakeflow into Delta tables with Change Data Feed
  • Added data quality checks and automatic CDC upserts into an SCD Type 1 trips table, plus a holiday-flagged calendar table
  • Published a gold trips fact view and 10 city-level views feeding Databricks dashboards and Genie

Stack Databricks, Auto Loader, Lakeflow Declarative Pipelines, Delta Lake, Unity Catalog, Change Data Feed, AWS S3

More projects

9 more, from lakehouse builds to full-stack apps. Open any row for the details.

Showing 9 of 9
0.96 ROC-AUC with LoRA, training under 5% of the parameters
Multimodal LLM for Cancer Pathology Classification RAG over image captions and a LoRA-tuned image-plus-text model that classify four tissue types. DetailsHide details

RAG and LoRA on public tissue images

tissue imageBLIP captionFAISS indexGPT-3.5 names the tissue type

  • Built a RAG pipeline that captions tissue images with BLIP, indexes them in FAISS and classifies 4 tissue types with GPT-3.5
  • Fused MobileNetV2 image features with caption embeddings in TensorFlow. LoRA reached ~80% accuracy, ROC-AUC 0.96
  • Matched full fine-tuning within ~1 point with 24× fewer trainable parameters, vs 19% zero-shot and 49% chain-of-thought GPT

Uses the public PCam and MHIST datasets. This is not clinical work.

Stack Python, TensorFlow, BLIP, MiniLM, FAISS, GPT-3.5, MobileNetV2, LoRA

7 currencies converted to INR for one BI view
E-Commerce Analytics Platform A legacy on-premises reporting flow rebuilt as a Databricks lakehouse. DetailsHide details

Lakehouse migration

AWS S3bronzesilver (cleaned)gold Deltaone denormalized BI view

  • Moved a legacy on-premises e-commerce reporting flow to a Databricks lakehouse with bronze, silver and gold Delta tables
  • Cleaned duplicate orders, mixed date formats, "$" and "%" symbols and misspelled values with PySpark and Spark SQL
  • Converted revenue in 7 currencies to INR and published one BI view of sales by hour, weekday, brand and category

Stack Databricks, PySpark, Spark SQL, Delta Lake, Unity Catalog, AWS S3

60 s pull interval for the top 10 coins by market cap
Real-Time Cryptocurrency Trend Analysis Coin prices streamed through Kafka, deduped with streaming algorithms and flagged for price moves. DetailsHide details

Streaming and data mining

CoinGecko APIKafkaS3 + HDFSMapReduce and PySpark on EMRprice alerts

  • Streamed the top 10 coins from the CoinGecko API every 60 seconds through Kafka into S3 and HDFS on EMR
  • Implemented Bloom Filter, Flajolet-Martin, Reservoir Sampling and PySpark LSH to dedupe streaming crypto data and find similar coins
  • Explained a Random Forest model with SHAP, added Laplace-noise differential privacy, and flagged ≥5% gains and >3% drops

Stack Python, PySpark, Confluent Kafka, AWS S3, HDFS, AWS EMR, MapReduce, SHAP

60K+ games across 400 clubs and 30K players
Football Data Analytics Match, club and player data modeled in MySQL, Snowflake and MongoDB for coaching dashboards. DetailsHide details

Data modeling across three databases

MySQL schema, 9 dimension tablesSnowflake star schemaPower BI and Atlas Charts dashboards

  • Modeled 60K+ games, 400 clubs and 30K players into a MySQL schema with 9 dimension tables, then a Snowflake star schema
  • Analyzed transfer valuations, manager win rates and late-game goals in Snowflake, MySQL and MongoDB aggregation pipelines
  • Delivered Power BI and MongoDB Atlas Charts dashboards for transfer strategy and coaching reviews

Stack MySQL, Snowflake, MongoDB, PyMongo, Power BI, MongoDB Atlas Charts, AWS S3, Azure DevOps

~75% top-1 accuracy across 345 classes, in under 100 ms
Doodle Recognition Using Deep Learning Draw a sketch and a classifier names it in under 100 ms, then GPT-4 gives drawing tips. DetailsHide details

Deep Learning course project, San Jose State

~345K Quick, Draw! sketchesEfficientNetV2-B0Streamlit drawing appGPT-4 tips

  • Trained EfficientNetV2-B0 on ~345K Quick, Draw! sketches across 345 classes to ~75% top-1 accuracy with under 100 ms responses
  • Compared EfficientNetV2-B0 with a Vision Transformer and dropped the ViT after it overfit on unseen sketches
  • Wrote a Streamlit drawing app that classifies sketches live and sends each prediction to GPT-4 for drawing tips

Stack Python, EfficientNetV2-B0, Vision Transformer, GPT-4, Streamlit, Google Colab

8 dbt models across bronze, silver and gold, plus an SCD Type 2 snapshot
Retail Sales Analytics Platform A bronze, silver and gold dbt project on Databricks with tests and a snapshot. DetailsHide details

dbt on Databricks, tutorial-based build

6 source tables6 bronze modelssilvergoldrevenue by category and gender

  • Set up a bronze-silver-gold dbt project on Databricks with separate dev and prod catalogs and a custom schema macro
  • Added dbt tests (unique, not_null, accepted_values, custom non-negative) and an SCD Type 2 snapshot to track product history

Stack dbt, Databricks, Unity Catalog, Spark SQL, Jinja

2,640 billionaires across 78 countries and 18 industries
Comprehensive Analysis of Billionaires' Wealth Who holds $12.2T of wealth, joined with GDP, CPI, tax and education data by country. DetailsHide details

Exploratory analysis and dashboards

billionaire records + country indicatorsPython joinsexploratory analysisPower BI and Tableau

  • Joined data on 2,640 billionaires across 78 countries with GDP, CPI, tax and education indicators using Python
  • Found that 69% are self-made and Technology holds the largest wealth share (15.4%), ahead of Fashion & Retail and Finance
  • Designed Power BI views (world bubble map, net worth vs. GDP, industry breakdowns) for policy and investment analysis

Stack Python, Tableau, Power BI

K8s app, Kafka and Zookeeper deployed with Helm
Food Delivery Platform A Django REST and React food-delivery app with Kafka events on Kubernetes. DetailsHide details

UberEats-style course lab

React + ReduxDjango REST APIKafka producers and consumersDockerKubernetes with Helm

  • Added Kafka producer and consumer services for event-driven order and user events in a Django REST food-delivery app
  • Containerized the app and deployed it with Kafka and Zookeeper on Kubernetes using Deployments, Services and Helm

Stack Django REST Framework, React 18, Redux, Kafka, Zookeeper, Docker, Kubernetes, Helm, Swagger

JWT login with token blacklisting, for rider and driver roles
Taxi Booking Web App A ride-booking app with rider and driver roles, JWT login and a tested API. DetailsHide details

Full-stack web app

React (Vite) + Tailwind + MapboxDjango REST APISimpleJWT authpytest + OpenAPI docs

  • Built a ride-booking platform with Django REST Framework and React, with separate rider and driver roles
  • Secured the API with JWT login and token blacklisting, and added OpenAPI docs and pytest tests

Stack Django REST Framework, React (Vite), Tailwind CSS, SimpleJWT, drf-spectacular, pytest, Mapbox

Skills

Tools I have used at work or in the projects above.

Languages
Python, SQL, Java, JavaScript
Data engineering
ETL/ELT, PySpark, Spark SQL, Spark Structured Streaming, Kafka, Airflow, Databricks, Delta Lake, dbt
Cloud and databases
AWS (S3, EMR, Redshift, Lambda, EventBridge), Azure Event Hubs, GCP Cloud Run, Snowflake, PostgreSQL, MySQL, SQL Server, MongoDB, ChromaDB
Modeling and optimization
Star schema, medallion architecture, SCD Type 1 and 2, CDC, schema evolution, partitioning, query tuning, data quality checks
AI and LLMs
LangGraph, LangChain, Claude API, OpenAI API, text-to-SQL, RAG, MCP, QLoRA fine-tuning, LLM evaluation, LLM-as-a-Judge
BI and DevOps
Power BI (DAX), Tableau, Streamlit, Power Automate, Git, Docker, GitHub Actions, pytest
Learning now
Airflow 3 (12 practice DAGs with the TaskFlow API and assets), MCP servers with FastMCP, RAG notebooks with LangChain and Chroma

Education

Jan 2024 to Dec 2025

MS, Applied Data Intelligence San Jose State University

Coursework: Big Data Technology, Machine Learning, Deep Learning, Generative AI, Data Visualization

Jul 2019 to May 2023

BE, Computer Engineering LDRP-ITR, Gujarat, India

Coursework: Data Structures and Algorithms, Python Programming, Object-Oriented Programming (Java)

Contact

Email is the fastest way to reach me.