{"id":355,"date":"2018-03-14T09:50:29","date_gmt":"2018-03-14T01:50:29","guid":{"rendered":"http:\/\/www.max-shu.com\/blog\/?p=355"},"modified":"2018-03-14T09:50:29","modified_gmt":"2018-03-14T01:50:29","slug":"hbase%e5%92%8chadoop%e5%a2%9e%e5%8a%a0-snappy%e5%8e%8b%e7%bc%a9","status":"publish","type":"post","link":"http:\/\/www.max-shu.com\/blog\/?p=355","title":{"rendered":"HBase\u548cHadoop\u589e\u52a0 Snappy\u538b\u7f29"},"content":{"rendered":"<p># tar -zxvf\u00a0snappy-1.0.5.tar.gz<br \/>\n# .\/configure<br \/>\n# make<br \/>\n# make install<\/p>\n<p># vi \/etc\/ld.so.conf.d\/snappy.conf<br \/>\n\/usr\/local\/lib<\/p>\n<p># ldconfig<br \/>\n# ldconfig -p |grep snappy<br \/>\n# ldconfig -v |grep snappy<\/p>\n<p><span style=\"color: #ff00ff;\"><b>\u4e0b\u9762\u4e24\u884c\u53ef\u4ee5\u9632\u6b62\u5728\u4e0d\u540c\u7684\u64cd\u4f5c\u7cfb\u7edf\u4e0bhadoop\u539f\u6765\u81ea\u5e26\u7684snappy native\u5e93\u4e0d\u80fd\u4f7f\u7528\u7684\u95ee\u9898\uff1a<\/b><\/span><br \/>\n# cp -P \/usr\/local\/lib\/libsnappy* $HADOOP_HOME\/lib\/native\/Linux-amd64-64\/<br \/>\n# cp -P \/usr\/local\/lib\/libsnappy* $HBASE_HOME\/lib\/native\/Linux-amd64-64\/<\/p>\n<p># vi $HADOOP_HOME\/conf\/core-site.xml<br \/>\n&lt;property&gt;<br \/>\n&lt;name&gt;io.compression.codecs&lt;\/name&gt;<br \/>\n&lt;value&gt;org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,<span style=\"color: #ff0000;\">org.apache.hadoop.io.compress.SnappyCodec<\/span>&lt;\/value&gt;<br \/>\n&lt;\/property&gt;<\/p>\n<p># vi $HADOOP_HOME\/conf\/mapred.xml<br \/>\n&lt;property&gt;<br \/>\n&lt;name&gt;mapred.output.compression.codec&lt;\/name&gt;<br \/>\n&lt;value&gt;<span style=\"color: #ff0000;\">org.apache.hadoop.io.compress.SnappyCodec<\/span>&lt;\/value&gt;<br \/>\n&lt;description&gt;If the job outputs are compressed, how should they be compressed?<br \/>\n&lt;\/description&gt;<br \/>\n&lt;\/property&gt;<\/p>\n<p>&lt;property&gt;<br \/>\n&lt;name&gt;mapred.compress.map.output&lt;\/name&gt;<br \/>\n&lt;value&gt;true&lt;\/value&gt;<br \/>\n&lt;description&gt;Should the outputs of the maps be compressed before being<br \/>\nsent across the network. Uses SequenceFile compression.<br \/>\n&lt;\/description&gt;<br \/>\n&lt;\/property&gt;<\/p>\n<p>&lt;property&gt;<br \/>\n&lt;name&gt;mapred.map.output.compression.codec&lt;\/name&gt;<br \/>\n&lt;value&gt;<span style=\"color: #ff0000;\">org.apache.hadoop.io.compress.SnappyCodec<\/span>&lt;\/value&gt;<br \/>\n&lt;description&gt;If the map outputs are compressed, how should they be<br \/>\ncompressed?<br \/>\n&lt;\/description&gt;<br \/>\n&lt;\/property&gt;<\/p>\n<p>&lt;property&gt;<br \/>\n&lt;name&gt;mapred.output.compress&lt;\/name&gt;<br \/>\n&lt;value&gt;true&lt;\/value&gt;<br \/>\n&lt;description&gt;Should the job outputs be compressed?<br \/>\n&lt;\/description&gt;<br \/>\n&lt;\/property&gt;<\/p>\n<p>\u8981\u628a$HADOOP_HOME\/conf\u52a0\u5165hbase-env.sh\uff0cHADOOP_HOME\u5fc5\u987b\u662f\u7edd\u5bf9\u8def\u5f84\uff1a<br \/>\n# vi hbase-env.sh<br \/>\n<span style=\"color: #ff0000;\">export HADOOP_HOME=\/data1\/oHive\/hadoop-0.20.2-cdh3u2<br \/>\nexport HBASE_CLASSPATH=${HADOOP_HOME}\/conf<\/span><\/p>\n<p>\u6d4b\u8bd5\uff1a<br \/>\n#\u00a0<b><span style=\"color: #ff0000;\">.\/hbase org.apache.hadoop.hbase.util.CompressionTest file:\/\/\/tmp\/aaa snappy<\/span><\/b><\/p>\n<p># .\/hbase shell<br \/>\nhbase(main):003:0&gt;\u00a0<b><span style=\"color: #ff0000;\">create &#8216;t1&#8217;, { NAME =&gt; &#8216;cf1&#8217;, COMPRESSION =&gt; &#8216;SNAPPY&#8217; }<\/span><\/b><br \/>\nhbase(main):004:0&gt; describe &#8216;t1&#8217;<br \/>\nDESCRIPTION\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ENABLED<br \/>\n{NAME =&gt; &#8216;t1&#8217;, FAMILIES =&gt; [{NAME =&gt; &#8216;cf1&#8217;, BLOOMFILTER =&gt; &#8216;NONE&#8217;, REPLICATION_SCOPE true<br \/>\n=&gt; &#8216;0&#8217;,\u00a0<span style=\"color: #ff0000;\">COMPRESSION =&gt; &#8216;SNAPPY&#8217;<\/span>, VERSIONS =&gt; &#8216;3&#8217;, TTL =&gt; &#8216;2147483647&#8217;, BLOCKSIZE =&gt;<br \/>\n&#8216;65536&#8217;, IN_MEMORY =&gt; &#8216;false&#8217;, BLOCKCACHE =&gt; &#8216;true&#8217;}]}<br \/>\n1 row(s) in 0.5370 seconds<br \/>\nhbase(main):014:0&gt; put &#8216;t1&#8217;, &#8216;r1&#8217;, &#8216;cf1&#8217;, &#8216;1234&#8217;<br \/>\nhbase(main):020:0&gt; put &#8216;t1&#8217;, &#8216;r2&#8217;, &#8216;cf1&#8217;, &#8216;1234&#8217;<br \/>\nhbase(main):016:0&gt; get &#8216;t1&#8217;, &#8216;r1&#8217;<br \/>\nCOLUMN\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 CELL<br \/>\ncf1:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 timestamp=1347353707019, value=1234<br \/>\n1 row(s) in 0.0900 seconds<br \/>\nhbase(main):021:0&gt; get &#8216;t1&#8217;, &#8216;r2&#8217;<br \/>\nCOLUMN\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 CELL<br \/>\ncf1:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 timestamp=1347353737455, value=1234<br \/>\n1 row(s) in 0.0870 seconds<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p># tar -zxvf\u00a0snappy-1.0.5.tar.gz # .\/configure # make #  &hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[63],"tags":[80,198,275,276],"class_list":["post-355","post","type-post","status-publish","format-standard","hentry","category-63","tag-hadoop","tag-hbase","tag-snappy","tag-276"],"views":1503,"_links":{"self":[{"href":"http:\/\/www.max-shu.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/355","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.max-shu.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.max-shu.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.max-shu.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.max-shu.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=355"}],"version-history":[{"count":1,"href":"http:\/\/www.max-shu.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/355\/revisions"}],"predecessor-version":[{"id":356,"href":"http:\/\/www.max-shu.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/355\/revisions\/356"}],"wp:attachment":[{"href":"http:\/\/www.max-shu.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=355"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.max-shu.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=355"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.max-shu.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=355"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}