Open to Full-Time Data Engineering Opportunities

Rishikesh Pandey

Data Engineer  |  Lakehouse & Streaming Pipelines

Building production batch & streaming pipelines, ETL/ELT workflows and lakehouse platforms with Spark · PySpark · Databricks · Delta Lake · Kafka · Apache Iceberg · Trino · AWS S3

View Projects Email Me LinkedIn GitHub
~40%
Databricks Cost Cut
543K
Duplicate Rows Traced
2
DE Internships
Live
In Production
Scroll
01Who I Am

About Me

My Story

I'm a Data Engineer building production batch and streaming data pipelines, ETL/ELT workflows and lakehouse platforms on Spark, Databricks and Kafka. As a Data Engineering Intern at HDFC Securities (Jul 2026 – Present), I built a federated query platform on Apache Iceberg and Trino that shifted analytical workloads off Databricks compute, cutting spend by ~40%.

I also traced a 2–3% revenue reconciliation gap to 543K duplicated rows in the silver layer, then rebuilt the bronze-to-gold transaction pipeline end to end in PySpark and Delta Lake — now live in production. Before that, at Lumiq.ai (Feb – Jun 2026), I built a real-time Kafka + Spark Structured Streaming pipeline and a multi-tenant FMCG lakehouse.

Lakehouse Architecture Real-Time Streaming Medallion Architecture Data Quality ETL / ELT Dimensional Modelling
  • HDFC Securities
    Data Engineering Intern · Jul 2026 – Present
  • Bengaluru, India
    Available onsite or remote
  • Spark, Databricks & Delta Lake
    Medallion pipelines in production
  • Iceberg + Trino + Kafka
    Lakehouse & real-time streaming
  • B.Sc. Mathematics
    Siddharth University · CGPA 8.6/10
  • rishikesh8pandey
    4+ public repos · Open source
02Career

Work Experience

Data Engineering Intern

HDFC Securities

Jul 2026 – Present Bengaluru, India

Federated Query Lakehouse

  • Build and operate a production federated query lakehouse on Docker, federating SQL across Apache Iceberg tables on AWS S3 and relational sources (Oracle, PostgreSQL) through a Trino coordinator-worker cluster with Apache Superset BI and Apache Ranger authorization.
  • Deployed the stack to an air-gapped RHEL 8.8 production host with all images and RPMs version-pinned and pre-staged ahead of network cutoff; automated log rotation, resource alerting and backups via Cron.
  • Cut Databricks compute spend by ~40% by routing reporting and ad-hoc analytical queries through Trino on the lakehouse instead of source systems and dedicated job clusters.

Revenue Reconciliation & Pipeline Rebuild

  • Investigated a 2–3% revenue and turnover mismatch between the Oracle source and the gold layer, proving 543,354 duplicated rows in silver — a single full-day double-load affecting 18,175 clients, plus a scattered pattern across six other dates.
  • Traced parallel gold-layer aggregation defects to raw order_ts and sub_product_type used as GROUP BY keys, SUM() applied to per-share price fields, and inconsistent grouping keys across three contract-note schemas.
  • Validated a trade-key deduplication fix that realigned row count, quantity, revenue and turnover with the Oracle source from bronze through gold, verified using the pipeline’s own reporting formulas.
  • Rebuilt the pipeline end to end in PySpark and Delta Lake — YAML-driven append-only bronze ingestion, silver deduplication and typing of ~70 columns, gold aggregation on eight normalized keys with quantity-weighted average pricing — orchestrated on Databricks Workflows and now live in production.
Apache IcebergTrinoAWS S3OraclePostgreSQLApache SupersetApache RangerDockerRHELPySparkDelta LakeDatabricks Workflows

Data Engineering Intern

Lumiq.ai

Feb 2026 – Jun 2026 Noida, India

Real-Time Crypto Analytics Pipeline

  • Built a Dockerised real-time analytics pipeline ingesting live Binance WebSocket trade data into Apache Kafka.
  • Processed the stream with PySpark Structured Streaming using schema validation, watermarking and windowed aggregations to handle late and out-of-order events in live market data.
  • Persisted raw, cleaned and aggregated layers to PostgreSQL and surfaced sub-minute Grafana dashboards over the aggregated layer.

Multi-Tenant FMCG Lakehouse

  • Built a multi-tenant FMCG lakehouse on Delta Lake consolidating sales, customer and product data from multiple child companies into parent-level analytics.
  • Implemented bronze, silver and gold layers with full and incremental loads and data quality checks at each stage.
  • Designed a dimensional model — fact_sales with customer, product and date dimensions — powering consolidated group reporting.
PythonApache KafkaSpark Structured StreamingPostgreSQLGrafanaDockerDelta LakeDimensional Modelling
03Academics

Education

2022 – 2025

Bachelor in Mathematics

Siddharth University

CGPA: 8.6 / 10
2021 – 2022

Intermediate (Science)

TRC Memorial Public School

88%
2019 – 2020

High School

Central Academy Senior Sec. School

91%
04Verified Credentials

Certifications & Achievements

Earned through hands-on problem solving and platform assessments

SQL · Basic

SQL Basic Certificate

HackerRank

Verified Credential
SQL · Intermediate

SQL Intermediate Certificate

HackerRank

Verified Credential
Python · Basic

Python (Basic) Certificate

HackerRank

Verified Credential
DSA · Basic

Problem Solving Basic

HackerRank

Verified Credential
SQL · 50 Problems

SQL 50 Badge

LeetCode

Badge Earned
05Portfolio

Featured Projects

Production-oriented data engineering systems

data flow
Lakehouse · HDFC Securities

Federated Query Lakehouse Platform

Production lakehouse federating SQL across Apache Iceberg tables on AWS S3 and relational sources (Oracle, PostgreSQL) through a Trino coordinator-worker cluster, with Superset BI and Ranger authorization.

  • ~40% Databricks compute spend cut by routing analytics through Trino
  • Air-gapped RHEL 8.8 deployment, version-pinned images & RPMs
  • Cron-automated log rotation, resource alerting & backups
Apache IcebergTrinoAWS S3OraclePostgreSQLSupersetRangerDocker
Live in Production · Proprietary
data flow
Data Pipeline · HDFC Securities

Transaction Pipeline Reconciliation & Rebuild

Traced a 2–3% revenue and turnover mismatch between the Oracle source and gold layer to 543,354 duplicated silver rows, then rebuilt the bronze-to-gold pipeline end to end in PySpark and Delta Lake.

  • Found a full-day double-load hitting 18,175 clients + 6 other dates
  • Fixed gold-layer GROUP BY keys & SUM() on per-share prices
  • YAML-driven bronze, ~70-column silver, 8-key gold with qty-weighted pricing
PySparkDelta LakeDatabricks WorkflowsOracleSQLMedallion Architecture
Live in Production · Proprietary
data flow
Real-Time Streaming

Real-Time Crypto Analytics Platform

Dockerised real-time pipeline ingesting live Binance WebSocket trades into Apache Kafka, processed with PySpark Structured Streaming and served through sub-minute Grafana dashboards.

  • Schema validation, watermarking & windowed aggregations
  • Handles late and out-of-order market events
  • Raw, cleaned & aggregated layers in PostgreSQL
PythonKafkaSpark Structured StreamingPostgreSQLGrafanaDocker
View Source Code
data flow
E-Commerce Analytics

Real-Time Kafka E-Commerce Analytics

End-to-end e-commerce analytics platform with Kafka event streaming, Spark transformation and Snowflake warehousing for scalable, low-latency business insights.

  • Kafka producer & consumer pipeline
  • Snowflake cloud data warehousing
  • Business KPI analytics dashboard
PythonKafkaPySparkSnowflakeDockerSQL
View Source Code
data flow
Lakehouse

Multi-Tenant FMCG Lakehouse

Delta Lake lakehouse consolidating sales, customer and product data from multiple child companies into parent-level analytics for consolidated group reporting.

  • Bronze / silver / gold medallion layers
  • Full & incremental loads with data quality checks
  • Dimensional model: fact_sales + customer, product, date dims
PythonSQLPySparkDelta LakeDatabricksDimensional Modelling
View Source Code
app flow
Web Application

Talent Scout Hub — HR Hiring System

Intelligent recruitment platform streamlining candidate screening, hiring workflows and talent management with profile tracking and workflow automation.

  • Candidate profile management
  • Recruitment workflow automation
  • Backend API & dashboard interface
PythonFlaskSQLHTML/CSSJavaScript
View Source Code
06Toolkit

Technical Skills

Technologies I build with every day

Languages

PythonSQLShell Scripting

Data Engineering

ETL/ELTBatch ProcessingReal-Time StreamingChange Data Capture (CDC)Data ModellingDimensional ModellingMedallion ArchitectureData QualityIncremental Loads

Big Data & Lakehouse

Apache SparkPySparkSpark Structured StreamingApache KafkaDatabricksDelta LakeApache IcebergTrinoOLakeHadoop

Cloud & Orchestration

AWS S3DockerRHEL/LinuxGit/GitHubDatabricks WorkflowsApache RangerCron

Databases & BI

PostgreSQLOracleMongoDBApache SupersetGrafanaPandasNumPy

Core Expertise

  • Lakehouse Platforms (Iceberg, Delta)
  • Federated SQL with Trino
  • Batch & Streaming Pipelines
  • Data Reconciliation & Quality
  • Air-Gapped Production Deployment
07Open Source

GitHub Stats

68 Total Contributions Sep 11, 2023 - Present Week Streak Jun 14 - Jun 28 3 4 Longest Week Streak Mar 15 - Apr 5
08Get In Touch

Let's Build Data-Driven Solutions

Open to Data Engineering, Big Data and Real-Time Analytics opportunities. Let's connect and build something impactful.

Send Email LinkedIn GitHub
rishikeshpandey274@gmail.com Bengaluru, India