Hadoop??????????????
???????????? ???????[ 2016/3/23 10:10:28 ] ?????????????????? ???ù???
????fs.datanode.handler.count
????datanode?????????RPC?????????????3?????????????????Щ??????8????????????????????????????????????
????tasktracker.http.threads
????HTTP server???????????????????TaskTracker??????????map task????????????????????40~50??
????dfs.replication
?????????????????????3??????????
????dfs.block.size
????HDFS??????block??С??????64M????????128MB
????mapred.local.dir
???????? mapreduce ?м????洢??
????dfs.data.dir
????HDFS Block ?洢???
???????? ??? ??? - LZO (?????? map)
????FileInputFormat.setMaxInputSplitSize(job?? 64 *1024 * 1024);
????{map/reduce}.tasks.maximum
????????????TaskTracker????map/reduce task????
????io.sort.factor
?????????map task?????????????????(mapred.local.dir)??????spill?????merge sort ????Щ??????????????merge sort?????????????????spill?????ò????????????????????????merge sort?????????????????????????????
????mapred.child.java.opts
????????JVM?????????棬?????ó???????????á?
????io.sort.mb
????Map task????????????????????buffer???С??
????mapred.compress.map.output/ Mapred.output.compress
?????м???????????????????????????????????????Mapred.compress.map.output.codec/ Mapred.output.compress.codec??????????LZO?????Intel???????????????δ????????LZO????? TeraSort?????????????60%???????????Zlib?????
????reduce task???????
????Reduce shuffle???copier????????????5?????????????????16~25??
??????
???·???
??????????????????
2023/3/23 14:23:39???д?ò??????????
2023/3/22 16:17:39????????????????????Щ??
2022/6/14 16:14:27??????????????????????????
2021/10/18 15:37:44???????????????
2021/9/17 15:19:29???·???????·
2021/9/14 15:42:25?????????????
2021/5/28 17:25:47??????APP??????????
2021/5/8 17:01:11