重温Hadoop（2）-- MapReduce流程及partition

1.map(K1, V1) –> list (K2, V2)                    // 对输入数据进行抽取过滤排序等操作

 

2.combine(K2, list(V2)) –> list(K2, V2)        // 为了减少reduce的输入，需要在map端对输出进行预处理，类似3.reduce。不是所有的reduce都在部分数据集上有效，比如求平均值就不能简单用于combine

 

4.partition(K2, V2) –> integer                //将key划分配到不同reduce分区，返回分区索引号。分区内的key会排序，相同的键的所有值会合成一个组（list(V2)）

 

5.reduce(K2, list(V2)) –> list(K3, V3)   //每个reduce会处理具有某些特性的键，每个键上都有值的序列，是通过对所有map输出的值进行统计得来的；当获得一个分区后，tasktracker会对每条记录调用reduce。

 

默认的map和reduce函数是IdentityMapper和IdentityReducer，均是泛型类型，简单的将所有输入写到输出中。默认的 partitioner是HashPartitioner，对每天记录的键进行哈希操作以决定该记录属于那个分区让reduce处理。

 

 

partition:

/**
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.hadoop.mapreduce;

/** 
 * Partitions the key space.
 * 
 * <p><code>Partitioner</code> controls the partitioning of the keys of the 
 * intermediate map-outputs. The key (or a subset of the key) is used to derive
 * the partition, typically by a hash function. The total number of partitions
 * is the same as the number of reduce tasks for the job. Hence this controls
 * which of the <code>m</code> reduce tasks the intermediate key (and hence the 
 * record) is sent for reduction.</p>
 * 
 * @see Reducer
 */
public abstract class Partitioner<KEY, VALUE> {
  
  /** 
   * Get the partition number for a given key (hence record) given the total 
   * number of partitions i.e. number of reduce-tasks for the job.
   *   
   * <p>Typically a hash function on a all or a subset of the key.</p>
   *
   * @param key the key to be partioned.
   * @param value the entry value.
   * @param numPartitions the total number of partitions.
   * @return the partition number for the <code>key</code>.
   */
  public abstract int getPartition(KEY key, VALUE value, int numPartitions);
  
}

  可以看到只有一个方法，自定义pratition的时候，只需实现getPartition方法，getPartition返回分配到的reducer的号，大小从0到reducer数。