2014年2月25日 星期二

[筆記] Hadoop 2.x YARN Test Case如何執行?放在哪裡?



Hadoop 2.x 開始有太多指令和API 都不太一樣了,而目前網路上能搜尋到的幾乎都是 1.x 版本的資訊,所以如果遇到無法執行的,還是回到官網看文件最實在。關於Hadoop 內建的Benchmark 程式原本釋放在hadoop-common-test,那現在搬到哪裡去了呢?答案是:

/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.2.0.2.0.6.0-101-tests.jar



執行以下指令,就可以看到有哪些Benchmark 的方法可以使用:

# hadoop jar  hadoop-mapreduce-client-jobclient-2.2.0.2.0.6.0-101-tests.jar

An example program must be given as the first argument.
Valid program names are:
  • DFSCIOTest: Distributed i/o benchmark of libhdfs.
  • DistributedFSCheck: Distributed checkup of the file system consistency.
  • JHLogAnalyzer: Job History Log analyzer.
  • MRReliabilityTest: A program that tests the reliability of the MR framework by injecting faults/failures
  • SliveTest: HDFS Stress Test and Live Data Verification.
  • TestDFSIO: Distributed i/o benchmark.
  • fail: a job that always fails
  • filebench: Benchmark SequenceFile(Input|Output)Format (block,record compressed and uncompressed), Text(Input|Output)Format (compressed and uncompressed)
  • loadgen: Generic map/reduce load generator
  • mapredtest: A map/reduce test check.
  • minicluster: Single process HDFS and MR cluster.
  • mrbench: A map/reduce benchmark that can create many small jobs
  • nnbench: A benchmark that stresses the namenode.
  • sleep: A job that sleeps at each map and reduce task.
  • testbigmapoutput: A map/reduce program that works on a very big non-splittable file and does identity map/reduce
  • testfilesystem: A test for FileSystem read/write.
  • testmapredsort: A map/reduce program that validates the map-reduce framework's sort.
  • testsequencefile: A test for flat files of binary key value pairs.
  • testsequencefileinputformat: A test for sequence file input format.
  • testtextinputformat: A test for text input format.
  • threadedmapbench: A map/reduce benchmark that compares the performance of maps with multiple spills over maps with 1 spill

除了 hadoop-mapreduce-client-jobclient 所顯示的test case 外,另外一個有名的Tera sort 則仍然是放在hadoop-mapreduce-examples*.jar裡面,指令如下:

hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.2.0.2.0.6.0-101.jar teragen

hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.2.0.2.0.6.0-101.jar terasort



張貼留言