org.apache.hadoop.mapred
Class JobTracker

java.lang.Object
  extended by org.apache.hadoop.mapred.JobTracker
All Implemented Interfaces:
VersionedProtocol, JobSubmissionProtocol

public class JobTracker
extends Object
implements JobSubmissionProtocol

JobTracker is the central location for submitting and tracking MR jobs in a network environment.


Nested Class Summary
static class JobTracker.IllegalStateException
          A client tried to submit a job before the Job Tracker was ready.
static class JobTracker.State
           
 
Field Summary
static int CLUSTER_INCREMENT
           
static long COUNTER_UPDATE_INTERVAL
           
static int FILE_NOT_FOUND
           
static int HEARTBEAT_INTERVAL_MIN
           
static org.apache.commons.logging.Log LOG
           
static String MAP_OUTPUT_LENGTH
          The custom http header used for the map output length.
static float MAX_INMEM_FILESIZE_FRACTION
          Constant denoting the max size (in terms of the fraction of the total size of the filesys) of a map output file that we will try to keep in mem.
static float MAX_INMEM_FILESYS_USE
          Constant denoting when a merge of in memory files will be triggered
static int SUCCESS
           
static String TEMP_DIR_NAME
          Temporary directory name
static int TRACKERS_OK
           
static int UNKNOWN_TASKTRACKER
           
static long versionID
          version 3 introduced to replace emitHearbeat/pollForNewTask/pollForTaskWithClosedJob with heartbeat(TaskTrackerStatus, boolean, boolean, short) version 4 changed TaskReport for HADOOP-549.
 
Fields inherited from interface org.apache.hadoop.mapred.JobSubmissionProtocol
versionID
 
Method Summary
 Vector<org.apache.hadoop.mapred.JobInProgress> completedJobs()
           
 Vector<org.apache.hadoop.mapred.JobInProgress> failedJobs()
           
static InetSocketAddress getAddress(Configuration conf)
           
 JobStatus[] getAllJobs()
          Get all the jobs submitted.
 String getAssignedTracker(String taskId)
          Get tracker name for a given task id.
 ClusterStatus getClusterStatus()
          Get the current status of the cluster
 String getFilesystemName()
          Grab the local fs name
 int getInfoPort()
           
 org.apache.hadoop.mapred.JobInProgress getJob(String jobid)
           
 Counters getJobCounters(String jobid)
          Grab the current job counters
 JobProfile getJobProfile(String jobid)
          Grab a handle to a job that is already known to the JobTracker.
 JobStatus getJobStatus(String jobid)
          Grab a handle to a job that is already known to the JobTracker.
 String getJobTrackerMachine()
           
static String getLocalJobFilePath(String jobId)
          Get the localized job file path on the job trackers local file system
 TaskReport[] getMapTaskReports(String jobid)
          Grab a bunch of info on the map tasks that make up the job
 String getNewJobId()
          Allocates a new JobId string.
 Node getNode(String name)
          Return the Node in the network topology that corresponds to the hostname
 Collection<Node> getNodesAtMaxLevel()
          Returns a collection of nodes at the max level
 int getNumResolvedTaskTrackers()
           
 int getNumTaskCacheLevels()
           
static Node getParentNode(Node node, int level)
           
 long getProtocolVersion(String protocol, long clientVersion)
          Return protocol version corresponding to protocol interface.
 TaskReport[] getReduceTaskReports(String jobid)
          Grab a bunch of info on the reduce tasks that make up the job
 List<org.apache.hadoop.mapred.JobInProgress> getRunningJobs()
          Version that is called from a timer thread, and therefore needs to be careful to synchronize.
 long getStartTime()
           
 TaskCompletionEvent[] getTaskCompletionEvents(String jobid, int fromEventId, int maxEvents)
          Get task completion events for the jobid, starting from fromEventId.
 String[] getTaskDiagnostics(String jobId, String tipId, String taskId)
          Get the diagnostics for a given task
 org.apache.hadoop.mapred.TaskTrackerStatus getTaskTracker(String trackerID)
           
 int getTotalSubmissions()
           
 String getTrackerIdentifier()
          Get the unique identifier (ie.
 int getTrackerPort()
           
 org.apache.hadoop.mapred.HeartbeatResponse heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus status, boolean initialContact, boolean acceptNewTasks, short responseId)
          The periodic heartbeat mechanism between the TaskTracker and the JobTracker.
 JobStatus[] jobsToComplete()
          Get the jobs that are not completed and not failed
 void killJob(String jobid)
          Kill the indicated job
 boolean killTask(String taskid, boolean shouldFail)
          Mark a Task to be killed
static void main(String[] argv)
          Start the JobTracker process.
 void offerService()
          Run forever
 void reportTaskTrackerError(String taskTracker, String errorClass, String errorMessage)
          Report a problem to the job tracker.
 Node resolveAndAddToTopology(String name)
           
 Vector<org.apache.hadoop.mapred.JobInProgress> runningJobs()
           
static JobTracker startTracker(JobConf conf)
          Start the JobTracker with given configuration.
 void stopTracker()
           
 JobStatus submitJob(String jobId)
          JobTracker.submitJob() kicks off a new job.
 Collection taskTrackers()
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

HEARTBEAT_INTERVAL_MIN

public static final int HEARTBEAT_INTERVAL_MIN
See Also:
Constant Field Values

CLUSTER_INCREMENT

public static final int CLUSTER_INCREMENT
See Also:
Constant Field Values

COUNTER_UPDATE_INTERVAL

public static final long COUNTER_UPDATE_INTERVAL
See Also:
Constant Field Values

MAX_INMEM_FILESYS_USE

public static final float MAX_INMEM_FILESYS_USE
Constant denoting when a merge of in memory files will be triggered

See Also:
Constant Field Values

MAX_INMEM_FILESIZE_FRACTION

public static final float MAX_INMEM_FILESIZE_FRACTION
Constant denoting the max size (in terms of the fraction of the total size of the filesys) of a map output file that we will try to keep in mem. Ideally, this should be a factor of MAX_INMEM_FILESYS_USE

See Also:
Constant Field Values

SUCCESS

public static final int SUCCESS
See Also:
Constant Field Values

FILE_NOT_FOUND

public static final int FILE_NOT_FOUND
See Also:
Constant Field Values

MAP_OUTPUT_LENGTH

public static final String MAP_OUTPUT_LENGTH
The custom http header used for the map output length.

See Also:
Constant Field Values

TEMP_DIR_NAME

public static final String TEMP_DIR_NAME
Temporary directory name

See Also:
Constant Field Values

versionID

public static final long versionID
version 3 introduced to replace emitHearbeat/pollForNewTask/pollForTaskWithClosedJob with heartbeat(TaskTrackerStatus, boolean, boolean, short) version 4 changed TaskReport for HADOOP-549. version 5 introduced that removes locateMapOutputs and instead uses getTaskCompletionEvents to figure finished maps and fetch the outputs version 6 adds maxTasks to TaskTrackerStatus for HADOOP-1245 version 7 replaces maxTasks by maxMapTasks and maxReduceTasks in TaskTrackerStatus for HADOOP-1274 Version 8: HeartbeatResponse is added with the next heartbeat interval. version 9 changes the counter representation for HADOOP-2248 version 10 changes the TaskStatus representation for HADOOP-2208

See Also:
Constant Field Values

TRACKERS_OK

public static final int TRACKERS_OK
See Also:
Constant Field Values

UNKNOWN_TASKTRACKER

public static final int UNKNOWN_TASKTRACKER
See Also:
Constant Field Values
Method Detail

startTracker

public static JobTracker startTracker(JobConf conf)
                               throws IOException,
                                      InterruptedException
Start the JobTracker with given configuration. The conf will be modified to reflect the actual ports on which the JobTracker is up and running if the user passes the port as zero.

Parameters:
conf - configuration for the JobTracker.
Throws:
IOException
InterruptedException

stopTracker

public void stopTracker()
                 throws IOException
Throws:
IOException

getProtocolVersion

public long getProtocolVersion(String protocol,
                               long clientVersion)
                        throws IOException
Description copied from interface: VersionedProtocol
Return protocol version corresponding to protocol interface.

Specified by:
getProtocolVersion in interface VersionedProtocol
Parameters:
protocol - The classname of the protocol interface
clientVersion - The version of the protocol that the client speaks
Returns:
the version that the server will speak
Throws:
IOException

getAddress

public static InetSocketAddress getAddress(Configuration conf)

offerService

public void offerService()
                  throws InterruptedException
Run forever

Throws:
InterruptedException

getTotalSubmissions

public int getTotalSubmissions()

getJobTrackerMachine

public String getJobTrackerMachine()

getTrackerIdentifier

public String getTrackerIdentifier()
Get the unique identifier (ie. timestamp) of this job tracker start.

Returns:
a string with a unique identifier

getTrackerPort

public int getTrackerPort()

getInfoPort

public int getInfoPort()

getStartTime

public long getStartTime()

runningJobs

public Vector<org.apache.hadoop.mapred.JobInProgress> runningJobs()

getRunningJobs

public List<org.apache.hadoop.mapred.JobInProgress> getRunningJobs()
Version that is called from a timer thread, and therefore needs to be careful to synchronize.


failedJobs

public Vector<org.apache.hadoop.mapred.JobInProgress> failedJobs()

completedJobs

public Vector<org.apache.hadoop.mapred.JobInProgress> completedJobs()

taskTrackers

public Collection taskTrackers()

getTaskTracker

public org.apache.hadoop.mapred.TaskTrackerStatus getTaskTracker(String trackerID)

resolveAndAddToTopology

public Node resolveAndAddToTopology(String name)

getNodesAtMaxLevel

public Collection<Node> getNodesAtMaxLevel()
Returns a collection of nodes at the max level


getParentNode

public static Node getParentNode(Node node,
                                 int level)

getNode

public Node getNode(String name)
Return the Node in the network topology that corresponds to the hostname


getNumTaskCacheLevels

public int getNumTaskCacheLevels()

getNumResolvedTaskTrackers

public int getNumResolvedTaskTrackers()

heartbeat

public org.apache.hadoop.mapred.HeartbeatResponse heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus status,
                                                            boolean initialContact,
                                                            boolean acceptNewTasks,
                                                            short responseId)
                                                     throws IOException
The periodic heartbeat mechanism between the TaskTracker and the JobTracker. The JobTracker processes the status information sent by the TaskTracker and responds with instructions to start/stop tasks or jobs, and also 'reset' instructions during contingencies.

Parameters:
status - the status update
initialContact - true if this is first interaction since 'refresh', false otherwise.
acceptNewTasks - true if the TaskTracker is ready to accept new tasks to run.
responseId - the last responseId successfully acted upon by the TaskTracker.
Returns:
a HeartbeatResponse with fresh instructions.
Throws:
IOException

getFilesystemName

public String getFilesystemName()
                         throws IOException
Grab the local fs name

Specified by:
getFilesystemName in interface JobSubmissionProtocol
Throws:
IOException

reportTaskTrackerError

public void reportTaskTrackerError(String taskTracker,
                                   String errorClass,
                                   String errorMessage)
                            throws IOException
Report a problem to the job tracker.

Parameters:
taskTracker - the name of the task tracker
errorClass - the kind of error (eg. the class that was thrown)
errorMessage - the human readable error message
Throws:
IOException - if there was a problem in communication or on the remote side

getNewJobId

public String getNewJobId()
                   throws IOException
Allocates a new JobId string.

Specified by:
getNewJobId in interface JobSubmissionProtocol
Returns:
a unique job name for submitting jobs.
Throws:
IOException

submitJob

public JobStatus submitJob(String jobId)
                    throws IOException
JobTracker.submitJob() kicks off a new job. Create a 'JobInProgress' object, which contains both JobProfile and JobStatus. Those two sub-objects are sometimes shipped outside of the JobTracker. But JobInProgress adds info that's useful for the JobTracker alone. We add the JIP to the jobInitQueue, which is processed asynchronously to handle split-computation and build up the right TaskTracker/Block mapping.

Specified by:
submitJob in interface JobSubmissionProtocol
Throws:
IOException

getClusterStatus

public ClusterStatus getClusterStatus()
Description copied from interface: JobSubmissionProtocol
Get the current status of the cluster

Specified by:
getClusterStatus in interface JobSubmissionProtocol
Returns:
summary of the state of the cluster

killJob

public void killJob(String jobid)
Description copied from interface: JobSubmissionProtocol
Kill the indicated job

Specified by:
killJob in interface JobSubmissionProtocol

getJobProfile

public JobProfile getJobProfile(String jobid)
Description copied from interface: JobSubmissionProtocol
Grab a handle to a job that is already known to the JobTracker.

Specified by:
getJobProfile in interface JobSubmissionProtocol
Returns:
Profile of the job, or null if not found.

getJobStatus

public JobStatus getJobStatus(String jobid)
Description copied from interface: JobSubmissionProtocol
Grab a handle to a job that is already known to the JobTracker.

Specified by:
getJobStatus in interface JobSubmissionProtocol
Returns:
Status of the job, or null if not found.

getJobCounters

public Counters getJobCounters(String jobid)
Description copied from interface: JobSubmissionProtocol
Grab the current job counters

Specified by:
getJobCounters in interface JobSubmissionProtocol

getMapTaskReports

public TaskReport[] getMapTaskReports(String jobid)
Description copied from interface: JobSubmissionProtocol
Grab a bunch of info on the map tasks that make up the job

Specified by:
getMapTaskReports in interface JobSubmissionProtocol

getReduceTaskReports

public TaskReport[] getReduceTaskReports(String jobid)
Description copied from interface: JobSubmissionProtocol
Grab a bunch of info on the reduce tasks that make up the job

Specified by:
getReduceTaskReports in interface JobSubmissionProtocol

getTaskCompletionEvents

public TaskCompletionEvent[] getTaskCompletionEvents(String jobid,
                                                     int fromEventId,
                                                     int maxEvents)
                                              throws IOException
Get task completion events for the jobid, starting from fromEventId. Returns empty aray if no events are available.

Specified by:
getTaskCompletionEvents in interface JobSubmissionProtocol
Parameters:
jobid - job id
fromEventId - event id to start from.
maxEvents - the max number of events we want to look at
Returns:
array of task completion events.
Throws:
IOException

getTaskDiagnostics

public String[] getTaskDiagnostics(String jobId,
                                   String tipId,
                                   String taskId)
                            throws IOException
Get the diagnostics for a given task

Specified by:
getTaskDiagnostics in interface JobSubmissionProtocol
Parameters:
jobId - the id of the job
tipId - the id of the tip
taskId - the id of the task
Returns:
an array of the diagnostic messages
Throws:
IOException

killTask

public boolean killTask(String taskid,
                        boolean shouldFail)
                 throws IOException
Mark a Task to be killed

Specified by:
killTask in interface JobSubmissionProtocol
Parameters:
taskid - the id of the task to kill.
shouldFail - if true the task is failed and added to failed tasks list, otherwise it is just killed, w/o affecting job failure status.
Throws:
IOException

getAssignedTracker

public String getAssignedTracker(String taskId)
Get tracker name for a given task id.

Parameters:
taskId - the name of the task
Returns:
The name of the task tracker

jobsToComplete

public JobStatus[] jobsToComplete()
Description copied from interface: JobSubmissionProtocol
Get the jobs that are not completed and not failed

Specified by:
jobsToComplete in interface JobSubmissionProtocol
Returns:
array of JobStatus for the running/to-be-run jobs.

getAllJobs

public JobStatus[] getAllJobs()
Description copied from interface: JobSubmissionProtocol
Get all the jobs submitted.

Specified by:
getAllJobs in interface JobSubmissionProtocol
Returns:
array of JobStatus for the submitted jobs

getJob

public org.apache.hadoop.mapred.JobInProgress getJob(String jobid)

getLocalJobFilePath

public static String getLocalJobFilePath(String jobId)
Get the localized job file path on the job trackers local file system

Parameters:
jobId - id of the job
Returns:
the path of the job conf file on the local file system

main

public static void main(String[] argv)
                 throws IOException,
                        InterruptedException
Start the JobTracker process. This is used only for debugging. As a rule, JobTracker should be run as part of the DFS Namenode process.

Throws:
IOException
InterruptedException


Copyright © 2006 The Apache Software Foundation