Apache Hadoop

Three strengths

• Built to expect failure. Its file system treats hardware failure as "the norm rather than the exception", so a broken machine doesn't stop the work. 2
• Scales to thousands. The same software is designed to grow from a single server to thousands of machines. 1
• Work moves to the data. Running a computation near the data it works on is much more efficient than moving the data to the computation. 2

What it is

When data grows past what one computer can hold, you have to spread it across many. The hard part is keeping that reliable: with enough machines, some are always failing. Apache Hadoop is open source software built for exactly this. It's software for reliable, scalable, distributed computing, designed to handle failures at the application layer and not rely on the hardware never breaking. 1

It has four parts. HDFS, the Hadoop Distributed File System, stores files across many machines and gives high-throughput access to them. YARN schedules jobs and manages the cluster's resources. MapReduce is the system for processing large datasets in parallel. Common holds the shared utilities that support the rest. 1 MapReduce's name comes from its two steps: a map step that works on pieces of the data separately, and a reduce step that merges the results. 4

Hadoop grew out of technologies developed by Yahoo, Google and other web pioneers in the mid-2000s. 3 Google's own 2004 paper described the MapReduce model as one that automatically parallelises work across a large cluster of commodity machines, handling failures along the way. 4 By its 1.0 release in January 2012, Apache reported that Yahoo ran the world's largest known Hadoop production environment, spanning more than 42,000 nodes. 3

It also shaped how data is stored today. Researchers describe the data lake, a low-cost store for raw data in open file formats, as an approach that started with the Apache Hadoop movement, using HDFS for cheap storage. From 2015 onwards, cloud storage services began replacing HDFS for that job. 5 So Hadoop is established and proven, but newer options now cover much of what it once did alone.

We choose it when the job calls for it and weigh it against those newer options each time.

How we use it

We've used Hadoop on Data Solutions work for a transport client, on Azure. It let us spread the processing of very large datasets across many compute nodes (machines working in parallel). The data sat in a data lake as plain files, including Parquet, an open file format built for analysis. For you, it means data too large for an ordinary database can still be stored and processed reliably. For scheduled and bulk work we more often use Spark and Airflow, and for larger or more varied data, Databricks.

References

1. The Apache Software Foundation, "Apache Hadoop", hadoop.apache.org. https://hadoop.apache.org/ (accessed 29 October 2026)
2. The Apache Software Foundation, "HDFS Architecture", Apache Hadoop documentation. https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html (accessed 29 October 2026)
3. The Apache Software Foundation, "The Apache Software Foundation Announces Apache Hadoop v1.0", The ASF Blog, 4 January 2012. https://news.apache.org/foundation/entry/the_apache_software_foundation_announces21 (accessed 29 October 2026)
4. Jeffrey Dean and Sanjay Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters", OSDI '04: Sixth Symposium on Operating System Design and Implementation, 2004, pp. 137-150. https://research.google/pubs/mapreduce-simplified-data-processing-on-large-clusters/ (accessed 29 October 2026)
5. Michael Armbrust, Ali Ghodsi, Reynold Xin and Matei Zaharia, "Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics", 11th Conference on Innovative Data Systems Research (CIDR), 2021. https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf (accessed 29 October 2026)