Data-intensive SaaS companies and AI platforms moving away from vendor-locked data warehouses toward open table formats frequently search to hire a freelance Apache Iceberg, Polars, PyArrow, Parquet & Data Lakehouse programmer. I am Aldian Fazrihady, a Senior Freelance Data & AI Engineer available via Upwork.
Core Open Data Lakehouse & High-Speed Python Data Capabilities
- Pandas-to-Polars & PyArrow Zero-Copy Acceleration: Refactoring memory-bound single-threaded Pandas ETL pipelines into lazy, multi-threaded Rust-powered Polars and PyArrow workflows—cutting memory usage by 80% and speeding up execution 10–30x.
- Apache Iceberg & S3/GCS Open Lakehouses: Architecting ACID-compliant Apache Iceberg tables over S3/Cloudflare R2 with schema evolution, time-travel snapshots, and hidden partitioning queried via DuckDB, Trino, BigQuery, or Spark.
- Streaming Parquet Compaction: Preventing the "small files problem" on S3 using buffered Arrow record batches and automated Iceberg compaction jobs.
Frequently Asked Questions
Why are our Python Pandas scripts crashing with Out-Of-Memory (OOM) errors on 5GB CSV/Parquet files?
Pandas eagerly materializes intermediate copies that consume 5–10x the raw file size in RAM. Rewriting the pipeline with `polars.scan_parquet(...)` lazy streaming processes multi-gigabyte datasets in a few hundred megabytes of RAM.
How can we hire you as a freelance Polars & Data Lakehouse programmer?
Message me on Upwork.