Pig explain 运算符



  • explain 运算符

    explain 运算符用于显示的关系的逻辑,物理和MapReduce的执行计划。
    句法
    下面给出了explain运算符的语法。
    
    grunt> explain Relation_name;
    
    
    例子
    假设我们在HDFS中有一个具有以下内容的文件Student_data.txt。
    
    001,Rajiv,Reddy,9848022337,Hyderabad
    002,siddarth,Battacharya,9848022338,Kolkata
    003,Rajesh,Khanna,9848022339,Delhi
    004,Preethi,Agarwal,9848022330,Pune
    005,Trupthi,Mohanthy,9848022336,Bhuwaneshwar
    006,Archana,Mishra,9848022335,Chennai.
    
    
    如下所示,我们已使用LOAD运算符将其读入关系student。
    
    grunt> student = LOAD 'hdfs://localhost:9000/pig_data/student_data.txt' USING PigStorage(',')
       as ( id:int, firstname:chararray, lastname:chararray, phone:chararray, city:chararray );
    
    
    现在,让我们使用解释运算符解释名为student的关系,如下所示。
    
    grunt> explain student;
    
    
    输出
    它将产生以下输出。
    
    $ explain student;
    
    2020-10-05 11:32:43,660 [main]
    2020-10-05 11:32:43,660 [main] INFO  org.apache.pig.newplan.logical.optimizer
    .LogicalPlanOptimizer -
    {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator,
    GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, 
    MergeForEach, PartitionFilterOptimizer, PredicatePushdownOptimizer,
    PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter]}  
    #-----------------------------------------------
    # New Logical Plan: 
    #-----------------------------------------------
    student: (Name: LOStore Schema:
    id#31:int,firstname#32:chararray,lastname#33:chararray,phone#34:chararray,city#
    35:chararray)
    | 
    |---student: (Name: LOForEach Schema:
    id#31:int,firstname#32:chararray,lastname#33:chararray,phone#34:chararray,city#
    35:chararray)
        |   |
        |   (Name: LOGenerate[false,false,false,false,false] Schema:
    id#31:int,firstname#32:chararray,lastname#33:chararray,phone#34:chararray,city#
    35:chararray)ColumnPrune:InputUids=[34, 35, 32, 33,
    31]ColumnPrune:OutputUids=[34, 35, 32, 33, 31]
        |   |   | 
        |   |   (Name: Cast Type: int Uid: 31) 
        |   |   |     |   |   |---id:(Name: Project Type: bytearray Uid: 31 Input: 0 Column: (*))
        |   |   |     
        |   |   (Name: Cast Type: chararray Uid: 32)
        |   |   | 
        |   |   |---firstname:(Name: Project Type: bytearray Uid: 32 Input: 1
    Column: (*))
        |   |   |
        |   |   (Name: Cast Type: chararray Uid: 33)
        |   |   |
        |   |   |---lastname:(Name: Project Type: bytearray Uid: 33 Input: 2
        Column: (*))
        |   |   | 
        |   |   (Name: Cast Type: chararray Uid: 34)
        |   |   |  
        |   |   |---phone:(Name: Project Type: bytearray Uid: 34 Input: 3 Column:
    (*))
        |   |   | 
        |   |   (Name: Cast Type: chararray Uid: 35)
        |   |   |  
        |   |   |---city:(Name: Project Type: bytearray Uid: 35 Input: 4 Column:
    (*))
        |   | 
        |   |---(Name: LOInnerLoad[0] Schema: id#31:bytearray)
        |   |  
        |   |---(Name: LOInnerLoad[1] Schema: firstname#32:bytearray)
        |   |
        |   |---(Name: LOInnerLoad[2] Schema: lastname#33:bytearray)
        |   |
        |   |---(Name: LOInnerLoad[3] Schema: phone#34:bytearray)
        |   | 
        |   |---(Name: LOInnerLoad[4] Schema: city#35:bytearray)
        |
        |---student: (Name: LOLoad Schema: 
    id#31:bytearray,firstname#32:bytearray,lastname#33:bytearray,phone#34:bytearray
    ,city#35:bytearray)RequiredFields:null 
    #-----------------------------------------------
    # Physical Plan: #-----------------------------------------------
    student: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-36
    | 
    |---student: New For Each(false,false,false,false,false)[bag] - scope-35
        |   |
        |   Cast[int] - scope-21
        |   |
        |   |---Project[bytearray][0] - scope-20
        |   |  
        |   Cast[chararray] - scope-24
        |   |
        |   |---Project[bytearray][1] - scope-23
        |   | 
        |   Cast[chararray] - scope-27
        |   |  
        |   |---Project[bytearray][2] - scope-26 
        |   |  
        |   Cast[chararray] - scope-30 
        |   |  
        |   |---Project[bytearray][3] - scope-29
        |   |
        |   Cast[chararray] - scope-33
        |   | 
        |   |---Project[bytearray][4] - scope-32
        | 
        |---student: Load(hdfs://localhost:9000/pig_data/student_data.txt:PigStorage(',')) - scope19
    2020-10-05 11:32:43,682 [main]
    INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - 
    File concatenation threshold: 100 optimistic? false
    2020-10-05 11:32:43,684 [main]
    INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOp timizer - 
    MR plan size before optimization: 1 2020-10-05 11:32:43,685 [main]
    INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.
    MultiQueryOp timizer - MR plan size after optimization: 1 
    #--------------------------------------------------
    # Map Reduce Plan                                   
    #--------------------------------------------------
    MapReduce node scope-37
    Map Plan
    student: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-36
    |
    |---student: New For Each(false,false,false,false,false)[bag] - scope-35
        |   |
        |   Cast[int] - scope-21 
        |   |
        |   |---Project[bytearray][0] - scope-20
        |   |
        |   Cast[chararray] - scope-24
        |   |
        |   |---Project[bytearray][1] - scope-23
        |   |
        |   Cast[chararray] - scope-27
        |   | 
        |   |---Project[bytearray][2] - scope-26 
        |   | 
        |   Cast[chararray] - scope-30 
        |   |  
        |   |---Project[bytearray][3] - scope-29 
        |   | 
        |   Cast[chararray] - scope-33
        |   | 
        |   |---Project[bytearray][4] - scope-32 
        |  
        |---student:
    Load(hdfs://localhost:9000/pig_data/student_data.txt:PigStorage(',')) - scope
    19-------- Global sort: false
     ----------------