|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.apache.hadoop.mapred.TaskTracker
public class TaskTracker
TaskTracker is a process that starts and tracks MR Tasks in a networked environment. It contacts the JobTracker for Task assignments and reporting results.
Nested Class Summary | |
---|---|
static class |
TaskTracker.Child
The main() for child processes. |
static class |
TaskTracker.MapOutputServlet
This class is used in TaskTracker's Jetty to serve the map outputs to other nodes. |
class |
TaskTracker.TaskTrackerMetrics
|
Field Summary | |
---|---|
static int |
CLUSTER_INCREMENT
|
static long |
COUNTER_UPDATE_INTERVAL
|
static int |
FILE_NOT_FOUND
|
static int |
HEARTBEAT_INTERVAL_MIN
|
static org.apache.commons.logging.Log |
LOG
|
static String |
MAP_OUTPUT_LENGTH
The custom http header used for the map output length. |
static float |
MAX_INMEM_FILESIZE_FRACTION
Constant denoting the max size (in terms of the fraction of the total size of the filesys) of a map output file that we will try to keep in mem. |
static float |
MAX_INMEM_FILESYS_USE
Constant denoting when a merge of in memory files will be triggered |
static int |
SUCCESS
|
static String |
TEMP_DIR_NAME
Temporary directory name |
static long |
versionID
Changed the version to 2, since we have a new method getMapOutputs Changed version to 3 to have progress() return a boolean Changed the version to 4, since we have replaced TaskUmbilicalProtocol.progress(String, float, String, org.apache.hadoop.mapred.TaskStatus.Phase, Counters) with statusUpdate(String, TaskStatus)
Version 5 changed counters representation for HADOOP-2248
Version 6 changes the TaskStatus representation for HADOOP-2208
Version 7 changes the done api (via HADOOP-3140). |
Constructor Summary | |
---|---|
TaskTracker(JobConf conf)
Start with the local machine name, and the default JobTracker |
Method Summary | |
---|---|
void |
cleanupStorage()
Removes all contents of temporary storage. |
void |
close()
Close down the TaskTracker and all its components. |
void |
done(String taskid,
boolean shouldPromote)
The task is done. |
void |
fsError(String taskId,
String message)
A child task had a local filesystem error. |
org.apache.hadoop.mapred.InterTrackerProtocol |
getJobClient()
The connection to the JobTracker, used by the TaskRunner for locating remote files. |
TaskCompletionEvent[] |
getMapCompletionEvents(String jobId,
int fromEventId,
int maxLocs)
Called by a reduce task to get the map output locations for finished maps. |
long |
getProtocolVersion(String protocol,
long clientVersion)
Return protocol version corresponding to protocol interface. |
org.apache.hadoop.mapred.Task |
getTask(String taskid)
Called upon startup by the child process, to fetch Task data. |
TaskTracker.TaskTrackerMetrics |
getTaskTrackerMetrics()
|
InetSocketAddress |
getTaskTrackerReportAddress()
Return the port at which the tasktracker bound to |
boolean |
isIdle()
Is this task tracker idle? |
static void |
main(String[] argv)
Start the TaskTracker, point toward the indicated JobTracker |
void |
mapOutputLost(String taskid,
String errorMsg)
A completed map task's output has been lost. |
boolean |
ping(String taskid)
Child checking to see if we're alive. |
void |
reportDiagnosticInfo(String taskid,
String info)
Called when the task dies before completion, and we want to report back diagnostic info |
void |
run()
The server retry loop. |
void |
shuffleError(String taskId,
String message)
A reduce-task failed to shuffle the map-outputs. |
void |
shutdown()
|
boolean |
statusUpdate(String taskid,
org.apache.hadoop.mapred.TaskStatus taskStatus)
Called periodically to report Task progress, from 0.0 to 1.0. |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
public static final org.apache.commons.logging.Log LOG
public static final int HEARTBEAT_INTERVAL_MIN
public static final int CLUSTER_INCREMENT
public static final long COUNTER_UPDATE_INTERVAL
public static final float MAX_INMEM_FILESYS_USE
public static final float MAX_INMEM_FILESIZE_FRACTION
public static final int SUCCESS
public static final int FILE_NOT_FOUND
public static final String MAP_OUTPUT_LENGTH
public static final String TEMP_DIR_NAME
public static final long versionID
statusUpdate(String, TaskStatus)
Version 5 changed counters representation for HADOOP-2248
Version 6 changes the TaskStatus representation for HADOOP-2208
Version 7 changes the done api (via HADOOP-3140). It now expects whether
or not the task's output needs to be promoted.
Constructor Detail |
---|
public TaskTracker(JobConf conf) throws IOException
IOException
Method Detail |
---|
public TaskTracker.TaskTrackerMetrics getTaskTrackerMetrics()
public long getProtocolVersion(String protocol, long clientVersion) throws IOException
VersionedProtocol
protocol
- The classname of the protocol interfaceclientVersion
- The version of the protocol that the client speaks
IOException
public void cleanupStorage() throws IOException
IOException
public void shutdown() throws IOException
IOException
public void close() throws IOException
IOException
public org.apache.hadoop.mapred.InterTrackerProtocol getJobClient()
public InetSocketAddress getTaskTrackerReportAddress()
public void run()
run
in interface Runnable
public org.apache.hadoop.mapred.Task getTask(String taskid) throws IOException
IOException
public boolean statusUpdate(String taskid, org.apache.hadoop.mapred.TaskStatus taskStatus) throws IOException
taskid
- task-id of the childtaskStatus
- status of the child
IOException
public void reportDiagnosticInfo(String taskid, String info) throws IOException
taskid
- the id of the task involvedinfo
- the text to report
IOException
public boolean ping(String taskid) throws IOException
IOException
public void done(String taskid, boolean shouldPromote) throws IOException
taskid
- task's idshouldPromote
- whether to promote the task's output or not
IOException
public void shuffleError(String taskId, String message) throws IOException
IOException
public void fsError(String taskId, String message) throws IOException
IOException
public TaskCompletionEvent[] getMapCompletionEvents(String jobId, int fromEventId, int maxLocs) throws IOException
fromEventId
- the index starting from which the locations should be
fetchedmaxLocs
- the max number of locations to fetch
IOException
public void mapOutputLost(String taskid, String errorMsg) throws IOException
IOException
public boolean isIdle()
public static void main(String[] argv) throws Exception
Exception
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |