Hadoop 2.x 開始有太多指令和API 都不太一樣了，而目前網路上能搜尋到的幾乎都是 1.x 版本的資訊，所以如果遇到無法執行的，還是回到官網看文件最實在。關於Hadoop 內建的Benchmark 程式原本釋放在hadoop-common-test，那現在搬到哪裡去了呢？答案是：
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.2.0.2.0.6.0-101-tests.jar
執行以下指令，就可以看到有哪些Benchmark 的方法可以使用：
# hadoop jar hadoop-mapreduce-client-jobclient-2.2.0.2.0.6.0-101-tests.jar
An example program must be given as the first argument.
Valid program names are:
- DFSCIOTest: Distributed i/o benchmark of libhdfs.
- DistributedFSCheck: Distributed checkup of the file system consistency.
- JHLogAnalyzer: Job History Log analyzer.
- MRReliabilityTest: A program that tests the reliability of the MR framework by injecting faults/failures
- SliveTest: HDFS Stress Test and Live Data Verification.
- TestDFSIO: Distributed i/o benchmark.
- fail: a job that always fails
- filebench: Benchmark SequenceFile(Input|Output)Format (block,record compressed and uncompressed), Text(Input|Output)Format (compressed and uncompressed)
- loadgen: Generic map/reduce load generator
- mapredtest: A map/reduce test check.
- minicluster: Single process HDFS and MR cluster.
- mrbench: A map/reduce benchmark that can create many small jobs
- nnbench: A benchmark that stresses the namenode.
- sleep: A job that sleeps at each map and reduce task.
- testbigmapoutput: A map/reduce program that works on a very big non-splittable file and does identity map/reduce
- testfilesystem: A test for FileSystem read/write.
- testmapredsort: A map/reduce program that validates the map-reduce framework's sort.
- testsequencefile: A test for flat files of binary key value pairs.
- testsequencefileinputformat: A test for sequence file input format.
- testtextinputformat: A test for text input format.
- threadedmapbench: A map/reduce benchmark that compares the performance of maps with multiple spills over maps with 1 spill
除了 hadoop-mapreduce-client-jobclient 所顯示的test case 外，另外一個有名的Tera sort 則仍然是放在hadoop-mapreduce-examples*.jar裡面，指令如下：
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.2.0.2.0.6.0-101.jar teragen
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.2.0.2.0.6.0-101.jar terasort
沒有留言 :
張貼留言