List: Data Engineering | Curated by Kasichennupati

Dec 3, 2024
111 stories
Data Engineering
Yingjun Wu
Apache Iceberg: Built for Big Data, Ready for Small?Originally built for massive data lakes, Apache Iceberg is catching the attention of small teams. But can it really fit?
Sep 26, 2024
9
Sep 26, 2024
9
Mikkel Dengsøe
How data observability fits into the different stages in the data pipeline
Jun 28, 2024
Jun 28, 2024
Arup Nanda
Data Architecture The Right WayLearn Various Data Architecture Approaches and How to Structure the Data Architects for Each Approach
Jul 3, 2024
3
Jul 3, 2024
3
Boonjira Angsumalee (Blue)
Which Data File Format to Use? CSV, JSON, Parquet, Avro, ORCWhen discussing the file formats we choose for storing data, whether in a database or for a Data Scientist’s further use, many might think…
Jun 30, 2024
7
Jun 30, 2024
7
In
Art of Data Engineering
by
Pavan Emani
Mastering Architecture Diagrams and Technical Presentations: A Data Architect’s GuideHow to create impactful technical presentations and diagrams that effectively communicate complex ideas with precision and storytelling
Sep 20, 2024
13
Sep 20, 2024
13
In
Data Engineer Things
by
Vu Trinh
How does Notion handle 200 billion data entities?From PostgreSQL → Data Lake
Aug 6, 2024
15
Aug 6, 2024
15
In
Data Engineer Things
by
Thanh Enc
PoC Data Platform project utilizing modern data stack (Airflow, Spark, DBT, Trino, Hive metastore…PoC Data Platform is an innovative project designed to demonstrate how data from AdventureWorks can be extracted, loaded, and transformed…
Oct 8, 2024
4
Oct 8, 2024
4
In
Data Engineer Things
by
Yingjun Wu
Apache Iceberg Won the Future — What’s Next for 2025?RBAC, CDC, Materialized Views, and More: Everything You Need to Know About Apache Iceberg in 2025.
Nov 18, 2024
12
Nov 18, 2024
12
In
TDS Archive
by
Thomas Reid
PySpark Explained: Delta TablesLearn how to use the building blocks of Delta Lakes.
Aug 3, 2024
2
Aug 3, 2024
2
In
Orchestra’s Data Release Pipeline Blog
by
Hugo Lu
Why Apache Iceberg is heralding a new era of change in Data Engineering“Bring Your Own Storage (BYOS)” has never been cooler
Mar 26, 2024
4
Mar 26, 2024
4
Lewis Gavin
Why Zero-ETL is Revolutionizing Data EngineeringAre you ready for a zero-ETL future?
Mar 18, 2024
14
Mar 18, 2024
14
 This story is no longer available
In
datamindedbe
by
Niels Claeys
How we reduced our docker build times by 40%This post describes two ways to speed up building your Docker images: caching build info remotely, using the link option when copying files
Oct 4, 2023
18
Oct 4, 2023
18
 This story is no longer available
In
Towards AI
by
Lazar Gugleta
Why Polars Destroys Pandas in All Possible Ways for Data Scientists?One of the first Data Science libraries, Pandas, has been improving the lives of many developers across the globe, but Polars shows it is…
Aug 8, 2024
13
Aug 8, 2024
13
Avin Kohale
Spark — Beyond basics: Required Spark memory to process 100GB fileProcessing 100GBs file is a cake walk for spark ONLY if you know how to assign spark memory efficiently! Read to know more.
Aug 1, 2024
11
Aug 1, 2024
11
Gaurav Kumar
Data Engineering Roadmap1. Foundational Knowledge
Apr 1, 2024
6
Apr 1, 2024
6
Thoren Lederer
This Open Source Dashboard Solution With 59k Stars on GitHub astonished me — Apache SupersetI have searched for a free and easy-to-use Dashboard solution and found Superset. Here is my an installation tutorial for kubernetes.
May 26, 2024
6
May 26, 2024
6
In
Analytics Vidhya
by
Suraj Patel
Configure Cassandra With DjangoCassandra is (also known as Apache Cassandra) an open-source distributed database management system. It was developed to handle large…
Mar 30, 2020
1
Mar 30, 2020
1
Yeung WONG
Unlocking the Power of PostgreSQL: A Guide to Python Database InteractionsStep by Step Walkthrough from Installation of PostgreSQL to Application with SQLAlchemy and psycopg2
Nov 25, 2023
Nov 25, 2023