HADOOP HDFS/MR ARCHITECTURE:

  • Each data node has its own storage and has a replication factor of 3, meaning that each block of data has 2 other copies on other data nodes. This ensures fault-tolerance via redundancy.
  • A master-slave environment is used.

  • One of the main advantages of this architecture is that Hadoop will ship code to where the relevant data is. This will prevent us from having to run code on machines that don't contain the relevant data. Also, this means we don't have to ship large quantities of data to where the code is, saving network costs.

  • An HDFS cluster consists of a single NameNode, a master server that manages the file system namespace and regulates access to files by clients.

  • In addition, there are a number of DataNodes, usually one per node in the cluster, which manage storage attached to the nodes that they run on.

  • Internally, a file is split into one or more blocks and these blocks are stored in a set of DataNodes.

  • The NameNode executes file system namespace operations like opening, closing, and renaming files and directories. It also determines the mapping of blocks to DataNodes. The DataNodes are responsible for serving read and write requests from the file system’s clients. The DataNodes also perform block creation, deletion, and replication upon instruction from the NameNode.

  • These machines typically run a GNU/Linux operating system (OS)

  • A typical deployment has a dedicated machine that runs only the NameNode software.

  • Each of the other machines in the cluster runs one instance of the DataNode software. The architecture does not preclude running multiple DataNodes on the same machine but in a real deployment that is rarely the case.

  • HDFS supports a traditional hierarchical file organization.

  • The number of copies of a file is called the replication factor of that file. This information is stored by the NameNode.

  • The NameNode makes all decisions regarding replication of blocks. It periodically receives a Heartbeat and a Blockreport from each of the DataNodes in the cluster.

  • Receipt of a Heartbeat implies that the DataNode is functioning properly.

  • A Blockreport contains a list of all blocks on a DataNode.

  • Optimizing replica placement distinguishes HDFS from most other distributed file systems

  • For the common case, when the replication factor is three, HDFS’s placement policy is toput one replica on one node in the local rack, another on a different node in the local rack, and the last on a different node in a different rack. This policy cuts the inter-rack write traffic which generally improves write performance. The chance of rack failure is far less than that of node failure; this policy does not impact data reliability and availability guarantees. However, it does reduce the aggregate network bandwidth used when reading data since a block is placed in only two unique racks rather than three. With this policy, the replicas of a file do not evenly distribute across the racks. One third of replicas are on one node, two thirds of replicas are on one rack, and the other third are evenly distributed across the remaining racks. This policy improves write performance without compromising data reliability or read performance.

  • To minimize global bandwidth consumption and read latency, HDFS tries to satisfy a read request from a replica that is closest to the reader. If there exists a replica on the same rack as the reader node, then that replica is preferred to satisfy the read request. If HDFS cluster spans multiple data centers, then a replica that is resident in the local data center is preferred over any remote replica.

  • Startup Mode: On startup, the NameNode enters a special state called Safemode. Replication of data blocks does not occur when the NameNode is in the Safemode state. The NameNode receives Heartbeat and Blockreport messages from the DataNodes. A Blockreport contains the list of data blocks that a DataNode is hosting. Each block has a specified minimum number of replicas. A block is considered safely replicated when the minimum number of replicas of that data block has checked in with the NameNode. After a configurable percentage of safely replicated data blocks checks in with the NameNode (plus an additional 30 seconds), the NameNode exits the Safemode state. It then determines the list of data blocks (if any) that still have fewer than the specified number of replicas. The NameNode then replicates these blocks to other DataNodes.

results matching ""

    No results matching ""