Sohail Shaik
Sohail Shaik
Resume
Sohail Shaik
Founding Data & AI Engineer

Sohail Shaik

Data & AI Engineer

I design high-throughput data pipelines and dimensional warehouses that don't fall over at 3 AM. Specializing in deep SQL, idempotent Airflow DAGs, Spark, Star Schema (SCD Type 1/2), and low-latency pgvector RAG systems.

Zero-Loss

Idempotent ETL

SCD Type 1/2

Immutable Audit History

p99 < 100ms

Real-time Telemetry

3+ Yrs

Production Systems

About Me

Deterministic Pipelines & Distributed ETL

Pipelines must be idempotent by default. If a DAG fails halfway through 50M records, rerunning it should never create duplicates or orphan staging tables. I orchestrate with Airflow, process with Spark/Databricks, and build self-healing backfills with stored watermarks.

Dimensional Warehousing & Deep SQL

Postgres and Oracle tuning at the execution plan level. I design Star Schemas where fact tables record granular transactional truth and dimensions track history via SCD Type 1/2. No leaky abstractions: PgBouncer pooling, composite partial indexing, and sub-10ms analytic queries.

Governed RAG & Agent Systems

RAG is only as good as the underlying data layer. I bridge dimensional data stores to LLMs using pgvector semantic search and Model Context Protocol (MCP) servers with strict read-only execution guardrails—so agents query verified numbers, not hallucinations.

Technical Skills

Data Engineering

8 competencies
SQL & PL/SQL
Informatica IICS/IDMC
Apache Airflow
Apache Spark
Databricks
Star Schema Modeling
SCD Type 1/2
Incremental ETL / ELT

AI & LLM

6 competencies
RAG Pipelines
LLM Orchestration
pgvector
Model Context Protocol (MCP)
LangChain
OpenAI / Anthropic

Databases

5 competencies
Oracle Database
PostgreSQL
PgBouncer
Redis
MongoDB

Cloud & Infrastructure

6 competencies
Python
Docker
Kubernetes / EKS
AWS
CI/CD Pipelines
REST APIs & WebSockets

Work Experience

Jan 2026 – Present
Remote — San Francisco, USA
PrettiFlow

PrettiFlow

Founding Data Engineer

  • Engineered the distributed execution pipeline converting natural-language requirements into containerized running services, scaling from zero to 4,000+ beta users and 500+ DAU.
  • Cut end-to-end pipeline build latency from 15m to 8m (46% improvement) by profiling task bottlenecks, parallelizing topological dependency graphs, and aggressively caching intermediate layers.
  • Reduced container cold-start latency from ~3s to ~100ms by architecting pre-warmed pool orchestration and low-latency Redis Pub/Sub event streaming.
  • Automated developer infrastructure provisioning workflows and designed a cross-platform CLI for rapid local-to-cloud environment replication.
  • Implemented execution-layer data governance and safety guardrails: AST-level query sanitization, action-level blocking, and infinite loop traps to prevent runaway compute.
PythonAirflowPostgreSQLRedisDockerAWSWebSocketsCLI
Jan 2023 – Dec 2025
Bengaluru, India
OmniqAI

OmniqAI

Data Engineer – ETL & Data Warehousing

  • Built and maintained mission-critical Informatica IICS/IDMC mappings and taskflows integrating high-volume transactional streams into enterprise dimensional warehouses.
  • Authored complex Oracle PL/SQL stored procedures and source-to-target mappings, managing multi-tier staging, fact, and dimension loads with strict SCD Type 1 & 2 history preservation.
  • Engineered automated SQL reconciliation suites comparing source-vs-target row counts, check sums, and unmatched settlement anomalies—eliminating 95% of downstream reporting discrepancies.
  • Tuned bottleneck queries across multi-million row tables: eliminated sequential table scans via targeted partial indexing and connection pool starvation using PgBouncer, cutting hot-path latency by 40%.
  • Engineered enterprise RAG pipelines over dimensional PostgreSQL schemas, utilizing LangChain and custom prompt-grounding workflows for automated analytical reporting.
  • Architected internal Model Context Protocol (MCP) server endpoints allowing autonomous LLMs to safely query operational schemas with parameterized validation.
Informatica IICS/IDMCSQL & PL/SQLOraclePostgreSQLpgvectorLangChainMCPRedisDockerAWS

Featured Projects

Fault-tolerant distributed pipelines, deterministic star schemas with SCD Type 1/2, and grounded multi-agent graph execution.

Project MODULE
SERVICE ACTIVE
01
Banking Transaction Pipeline & Financial RAG

High-Throughput Financial Ingestion, SCD Type 2 Star Schema & Grounded RAG

Architecture Pipeline Flow
Raw Transactions→Spark Deduplication→Airflow Batch DAG→PostgreSQL Star Schema (SCD 2)→pgvector RAG Layer
  • Ingested simulated high-concurrency payment feeds and clearinghouse settlement files into isolated staging schemas with schema-on-read validation.
  • Enforced deterministic incremental processing via stored watermark tracking and atomic database upserts, guaranteeing crash recovery with zero duplicate insertions.
  • Architected a dimensional Star Schema in PostgreSQL with fact_transactions and dim_accounts / dim_merchants, implementing SCD Type 2 history tracking (valid_from, valid_to, is_current) for complete regulatory auditability.
  • Built automated SQL reconciliation gates that assert record count parity and financial volume balance between source and warehouse, automatically quarantining mismatched records for investigation.
  • Orchestrated self-healing Airflow DAGs with configurable task retry policies, automated backfilling, and failure alert webhooks.
SQL & PL/SQLApache AirflowApache SparkPostgreSQLStar SchemaSCD Type 1/2PythonpgvectorLangChainDocker
View GitHub
Project MODULE
DEPLOYED / VERIFIED
02
OctaClaw — Multi-Agent Task Orchestration Engine

Autonomous Goal Decomposition & Fault-Tolerant DAG Execution

Architecture Pipeline Flow
Goal Decomposition→Topological DAG Graph→Parallel Agent Workers→MCP Tool Execution
  • Architected a distributed multi-agent engine that decomposes complex engineering goals into topological dependency graphs (DAGs) and executes subtasks in parallel across specialized agents.
  • Implemented transactional state persistence and checkpointing in PostgreSQL, allowing long-running execution graphs to survive process crashes and safely resume from the last known state.
  • Engineered sandboxed Model Context Protocol (MCP) integrations for filesystem, HTTP, email, relational databases, and Jira with action-level authorization and dry-run validation.
PythonTypeScriptMulti-Agent SystemsDAG OrchestrationPostgreSQLMCP Tooling
View GitHub