Pig 安装



  • 安装

    本章说明如何在系统中下载,安装和设置Apache Pig。
  • 先决条件

    在选择Apache Pig之前,必须在系统上安装HadoopJava。因此,在安装Apache Pig之前,请按照以下链接中给出的步骤安装Hadoop和Java-
  • 下载Apache Pig

    首先,从以下网站下载最新版本的Apache Pig- https://pig.apache.org/
    第1步 - 打开Apache Pig网站的主页。在新闻部分下,单击链接发布页面,如以下快照所示。
    pig
    第2步 - 单击指定的链接后,您将被重定向到Apache Pig Releases页面。在此页面的“Download”部分。
    pig
    第3步 - 然后您将被重定向到具有一组镜像的页面。选择并单击清华大学的镜像(更快一些这个镜像),如下所示。
    pig
    步骤4 - 这些镜像将带您到Pig Releases页面。该页面包含各种版本的Apache Pig。单击其中的最新版本。
    pig
    第5步 -在这些文件夹中,您将获得各种发行版的Apache Pig的源文件和二进制文件。下载Apache Pig 0.17,Pig0.17.0-src.tar.gz和pig-0.17.0.tar.gz的源文件和二进制文件的tar文件。
    pig
  • 安装Apache Pig

    下载Apache Pig软件之后,请按照以下步骤在Linux环境中安装它。
    第1步 - 在安装了Hadoop,Java和其他软件的安装目录的目录中,创建一个名为Pig的目录。(在我们的教程中,我们在名为Hadoop的用户中创建了Pig目录)。
    
    $ mkdir Pig
    
    
    第2步 - 提取下载的tar文件,如下所示。(我们只下载了二进制包)
    
    $ cd Downloads/  
    $ tar zxvf pig-0.17.0.tar.gz 
    
    
    第3步 - 将pig-0.17.0.tar.gz文件的内容移动到之前创建的Pig目录中,如下所示。
    
    $ mv pig-0.17.0/* /home/Hadoop/Pig/
    
    
  • 配置Apache Pig

    安装Apache Pig之后,我们必须对其进行配置。要进行配置,我们需要编辑两个文件-bashrc和pig.properties。
    .bashrc文件
    在.bashrc文件中,设置以下变量-
    • PIG_HOME文件夹到Apache Pig的安装文件夹,
    • PATH环境变量到bin文件夹,以及
    • PIG_CLASSPATH环境变量到Hadoop安装的etc(配置)文件夹(包含core-site.xml,hdfs-site.xml和mapred-site.xml文件的目录)中。
    
    export PIG_HOME = /home/Hadoop/Pig
    export PATH  = $PATH:/home/Hadoop/pig/bin
    export PIG_CLASSPATH = $HADOOP_HOME/conf
    
    
    pig.properties文件
    在Pig的conf文件夹中,我们有一个名为pig.properties的文件。在pig.properties文件中,您可以设置以下各种参数。
    
    pig -h properties 
    
    
    支持以下属性-
    
    Logging: verbose = true|false; default is false. This property is the same as -v
           switch brief=true|false; default is false. This property is the same 
           as -b switch debug=OFF|ERROR|WARN|INFO|DEBUG; default is INFO.             
           This property is the same as -d switch aggregate.warning = true|false; default is true. 
           If true, prints count of warnings of each type rather than logging each warning.       
           
    Performance tuning: pig.cachedbag.memusage=<mem fraction>; default is 0.2 (20% of all memory).
           Note that this memory is shared across all large bags used by the application.         
           pig.skewedjoin.reduce.memusagea=<mem fraction>; default is 0.3 (30% of all memory).
           Specifies the fraction of heap available for the reducer to perform the join.
           pig.exec.nocombiner = true|false; default is false.
               Only disable combiner as a temporary workaround for problems.         
           opt.multiquery = true|false; multiquery is on by default.
               Only disable multiquery as a temporary workaround for problems.
           opt.fetch=true|false; fetch is on by default.
               Scripts containing Filter, Foreach, Limit, Stream, and Union can be dumped without MR jobs.         
           pig.tmpfilecompression = true|false; compression is off by default.             
               Determines whether output of intermediate jobs is compressed.         
           pig.tmpfilecompression.codec = lzo|gzip; default is gzip.
               Used in conjunction with pig.tmpfilecompression. Defines compression type.         
           pig.noSplitCombination = true|false. Split combination is on by default.
               Determines if multiple small files are combined into a single map.         
               
           pig.exec.mapPartAgg = true|false. Default is false.             
               Determines if partial aggregation is done within map phase, before records are sent to combiner.         
           pig.exec.mapPartAgg.minReduction=<min aggregation factor>. Default is 10.             
               If the in-map partial aggregation does not reduce the output num records by this factor, it gets disabled.
               
    Miscellaneous: exectype = mapreduce|tez|local; default is mapreduce. This property is the same as -x switch
           pig.additional.jars.uris=<comma seperated list of jars>. Used in place of register command.
           udf.import.list=<comma seperated list of imports>. Used to avoid package names in UDF.
           stop.on.failure = true|false; default is false. Set to true to terminate on the first error.         
           pig.datetime.default.tz=<UTC time offset>. e.g. +08:00. Default is the default timezone of the host.
               Determines the timezone used to handle datetime datatype and UDFs.
    Additionally, any Hadoop property can be specified.
    
    
  • 验证安装

    通过键入version命令来验证Apache Pig的安装。如果安装成功,您将获得如下所示的Apache Pig版本。
    
    $ pig –version 
    
    
    
    Apache Pig version 0.17.0 (r1682971)  
    compiled Jun 01 2015, 11:44:35