org.apache.hadoop.mapred

Class CollectiveMapper<KEYIN,VALUEIN,KEYOUT,VALUEOUT>

java.lang.Object

<any>

org.apache.hadoop.mapred.CollectiveMapper<KEYIN,VALUEIN,KEYOUT,VALUEOUT>

Type Parameters:

KEYIN - Input key

VALUEIN - Input value

KEYOUT - Output key

VALUEOUT - Output value

public class CollectiveMapper<KEYIN,VALUEIN,KEYOUT,VALUEOUT>
extends <any>

CollectiveMapper is extended from original mapper in Hadoop. It includes new APIs for in-memory collective communication.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	CollectiveMapper.KeyValReader A Key-Value reader to read key-value inputs for this worker.

Field Summary

Fields

Modifier and Type	Field and Description
protected static Log	LOG

Constructor Summary

Constructors

Constructor and Description
CollectiveMapper()

Method Summary

Modifier and Type	Method and Description
<P extends Simple> boolean	allgather(java.lang.String contextName, java.lang.String operationName, Table<P> table) Allgather partitions of the tables to all the local tables.
<P extends Simple> boolean	allreduce(java.lang.String contextName, java.lang.String operationName, Table<P> table) Allreduce partitions of the tables to all the local tables.
boolean	barrier(java.lang.String contextName, java.lang.String operationName) Synchronize workers through a barrier
<P extends Simple> boolean	broadcast(java.lang.String contextName, java.lang.String operationName, Table<P> table, int bcastWorkerID, boolean useMSTBcast) Broadcast the partitions of the table on a worker to other workers.
protected void	cleanup(Context context) Called once at the end of the task.
protected void	freeConn() Free the connections cached in the pool.
protected void	freeMemory() Free the objects cached in the pool.
Event	getEvent() Get an event from the event queue.
int	getMasterID() Get the ID of the master worker.
int	getMaxID() Get the maximum worker ID
int	getMinID() Get the minimum worker ID.
int	getNumWorkers() Get the total number of workers.
int	getSelfID() Get the ID of this worker.
boolean	isMaster() Check if this worker is the master worker.
protected void	logGCTime() Log Garbage collection time
protected void	logMemUsage() Log memory usage
protected void	mapCollective(CollectiveMapper.KeyValReader reader, Context context) Most applications should override this.
<P extends Simple> boolean	pull(java.lang.String contextName, java.lang.String operationName, Table<P> localTable, Table<P> globalTable, boolean useBcast) Pull partitions in the global table to the local tables.
<P extends Simple,PT extends Partitioner> boolean	push(java.lang.String contextName, java.lang.String operationName, Table<P> localTable, Table<P> globalTable, PT partitioner) Push the partitions of local tables to the global table.
<P extends Simple> boolean	reduce(java.lang.String contextName, java.lang.String operationName, Table<P> table, int reduceWorkerID) Reduce the partitions of the tables to one of them.
<P extends Simple,PT extends Partitioner> boolean	regroup(java.lang.String contextName, java.lang.String operationName, Table<P> table, PT partitioner) Regroup the partitions of the tables based on a partitioner.
<P extends Simple> boolean	rotate(java.lang.String contextName, java.lang.String operationName, Table<P> globalTable, Int2IntMap rotateMap) The rotation communication.
void	run(Context context) Override this method to support collective communications among Mappers
boolean	sendEvent(Event event) Send an event to the local (local event), to a remote worker (message event), or to the rest workers (collective event).
protected void	setup(Context context) Called once at the beginning of the task.
Event	waitEvent() Wait for an event from the queue

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final Log LOG

Constructor Detail

CollectiveMapper

public CollectiveMapper()

Method Detail

getSelfID

public int getSelfID()

Get the ID of this worker.

Returns:

Worker ID

getMasterID

public int getMasterID()

Get the ID of the master worker.

Returns:

Master ID

isMaster

public boolean isMaster()

Check if this worker is the master worker.

Returns:

is this the master worker ID?

getNumWorkers

public int getNumWorkers()

Get the total number of workers.

Returns:

the number of workers

getMinID

public int getMinID()

Get the minimum worker ID.

Returns:

worker ID

getMaxID

public int getMaxID()

Get the maximum worker ID

Returns:

barrier

public boolean barrier(java.lang.String contextName,
                       java.lang.String operationName)

Synchronize workers through a barrier

Parameters:

contextName -

operationName -

Returns:

a boolean tells if the operation succeeds

broadcast

public <P extends Simple> boolean broadcast(java.lang.String contextName,
                                            java.lang.String operationName,
                                            Table<P> table,
                                            int bcastWorkerID,
                                            boolean useMSTBcast)

Broadcast the partitions of the table on a worker to other workers.

Parameters:

contextName - the name of the operation context

operationName - the name of the operation

table - the table used to hold the partitions

bcastWorkerID - the worker ID of broadcasting data

useMSTBcast - if minimum-spanning tree algorithm is used

Returns:

a boolean tells if the operation succeeds

reduce

public <P extends Simple> boolean reduce(java.lang.String contextName,
                                         java.lang.String operationName,
                                         Table<P> table,
                                         int reduceWorkerID)

Reduce the partitions of the tables to one of them.

Parameters:

contextName - the name of the operation context

operationName - the name of the operation

table - the table to hold the partitions

reduceWorkerID - the worker ID for reduce operation

Returns:

a boolean tells if the operation succeeds

allgather

public <P extends Simple> boolean allgather(java.lang.String contextName,
                                            java.lang.String operationName,
                                            Table<P> table)

Allgather partitions of the tables to all the local tables.

Parameters:

contextName - the name of the operation context

operationName - the name of the operation

table - the table to hold the partitions

Returns:

a boolean tells if the operations succeeds

allreduce

public <P extends Simple> boolean allreduce(java.lang.String contextName,
                                            java.lang.String operationName,
                                            Table<P> table)

Allreduce partitions of the tables to all the local tables.

Parameters:

contextName - the name of the operation context

operationName - the name of the operation

table - the table to hold the partitions

Returns:

a boolean tells if the operation succeeds

regroup

public <P extends Simple,PT extends Partitioner> boolean regroup(java.lang.String contextName,
                                                                 java.lang.String operationName,
                                                                 Table<P> table,
                                                                 PT partitioner)

Regroup the partitions of the tables based on a partitioner.

Parameters:

contextName - the name of the operation context

operationName - the name of the operation

table - the table to hold the partitions

partitioner - a boolean tells if the operations succeeds

Returns:

pull

public <P extends Simple> boolean pull(java.lang.String contextName,
                                       java.lang.String operationName,
                                       Table<P> localTable,
                                       Table<P> globalTable,
                                       boolean useBcast)

Pull partitions in the global table to the local tables. If any partition ID conflicts with the partition ID in the local table, combines them.

Parameters:

contextName - the name of the operation context

operationName - the name of the operation

localTable - the local tables

globalTable - the global table, acts like a distributed dataset

useBcast - if using broadcast in scattering the partitions

Returns:

a boolean tells if the operation succeeds

push

public <P extends Simple,PT extends Partitioner> boolean push(java.lang.String contextName,
                                                              java.lang.String operationName,
                                                              Table<P> localTable,
                                                              Table<P> globalTable,
                                                              PT partitioner)

Push the partitions of local tables to the global table.

Parameters:

contextName - the name of the operation context

operationName - the name of operation

localTable - the local tables

globalTable - the global table which acts like a distributed dataset, each partition in this table is unique

partitioner - when a partiitoner is used, the local partitions are sent to the global table even without partiiton ID association

Returns:

a boolean tells if the operation succeeds

rotate

public <P extends Simple> boolean rotate(java.lang.String contextName,
                                         java.lang.String operationName,
                                         Table<P> globalTable,
                                         Int2IntMap rotateMap)

The rotation communication. Transfer the partitions from worker to worker. After rotation, every worker sends its partitions to one of the other worker, and receive partitions from one of the other workers

Parameters:

contextName - the name of the operation context

operationName - the name of operation

globalTable - the global table which acts like a distributed dataset, each partition in this table is unique

rotateMap - the map from worker to worker, defines how to rotate the data

Returns:

true if succeeded, false if not

getEvent

public Event getEvent()

Get an event from the event queue.

Returns:

an event object, null if the queue is empty

waitEvent

public Event waitEvent()

Wait for an event from the queue

Returns:

the event object

sendEvent

public boolean sendEvent(Event event)

Send an event to the local (local event), to a remote worker (message event), or to the rest workers (collective event).

Parameters:

event - an event

Returns:

true if succeeded, false if not

freeMemory

protected void freeMemory()

Free the objects cached in the pool.

freeConn

protected void freeConn()

Free the connections cached in the pool.

logMemUsage

protected void logMemUsage()

Log memory usage

logGCTime

protected void logGCTime()

Log Garbage collection time

setup

protected void setup(Context context)
              throws java.io.IOException,
                     java.lang.InterruptedException

Called once at the beginning of the task.

Throws:

java.io.IOException

java.lang.InterruptedException

mapCollective

protected void mapCollective(CollectiveMapper.KeyValReader reader,
                             Context context)
                      throws java.io.IOException,
                             java.lang.InterruptedException

Most applications should override this.

Throws:

java.io.IOException

java.lang.InterruptedException

cleanup

protected void cleanup(Context context)
                throws java.io.IOException,
                       java.lang.InterruptedException

Called once at the end of the task.

Throws:

java.io.IOException

java.lang.InterruptedException

run

public void run(Context context)
         throws java.io.IOException,
                java.lang.InterruptedException

Override this method to support collective communications among Mappers

Parameters:

context - the context

Throws:

java.io.IOException

java.lang.InterruptedException