|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.apache.hadoop.mapred.MapReduceBase
com.ebay.erl.mobius.core.datajoin.DataJoinMapper<IK,IV,org.apache.hadoop.io.WritableComparable<?>,org.apache.hadoop.io.WritableComparable<?>>
com.ebay.erl.mobius.core.mapred.AbstractMobiusMapper<IK,IV>
IK
- input key type.IV
- input value type.public abstract class AbstractMobiusMapper<IK,IV>
Base class for implementing a customized Mobius mapper.
Extends this class if the built-in mappers,
TSVMapper
and
SequenceFileMapper
,
does not meet the needs.
This class provides filtering (by taking user specified
tuple_criteria
), compute computedColumns
,
and updating counters.
Override the parse(Object, Object)
method to convert
the K-V objects into a tuple, then the underlying data source
can be processed by mobius.
This product is licensed under the Apache License, Version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0. This product contains portions derived from Apache hadoop which is licensed under the Apache License, Version 2.0, available at http://hadoop.apache.org. © 2007 – 2012 eBay Inc., Evan Chiu, Woody Zhou, Neel Sundaresan
Field Summary | |
---|---|
static long |
_100MB
|
protected long |
_COUNTER_FILTERED_RECORD
Counts for the number of filtered records, filtered by user specified tuple_criteria . |
protected long |
_COUNTER_INPUT_RECORD
Counts for the number of input records. |
protected long |
_COUNTER_INVALIDATE_FORMAT_RECORD
Counts for invalidate format records. |
protected long |
_COUNTER_OUTPUT_RECORD
Counts for the number of outputted records. |
protected boolean |
_IS_MAP_ONLY_JOB
|
protected java.util.List<ComputedColumns> |
computedColumns
ComputedColumns specified by user. |
protected com.ebay.erl.mobius.core.mapred.CounterUpdateThread |
counterThread
A background thread responsible for updating the Hadoop counters. |
protected java.lang.String |
currentDatasetID
The current dataset ID. |
protected java.lang.String |
dataset_display_id
The normalized name of the dataset been processed by this mapper currently, it is used as counter ID to update the corresponding Hadoop counters for this dataset. |
protected java.lang.String[] |
key_columns
columns to be emitted as key of this Mapper |
protected java.lang.String[] |
projection_order
Output column names for map only job, ex: listing. |
protected boolean |
reporterSet
|
protected TupleCriterion |
tuple_criteria
filters |
protected java.lang.String[] |
value_columns
columns to be emitted as value of this Mapper |
Fields inherited from class com.ebay.erl.mobius.core.datajoin.DataJoinMapper |
---|
conf, hasReducer |
Constructor Summary | |
---|---|
AbstractMobiusMapper()
|
Method Summary | |
---|---|
void |
close()
close Mapper |
void |
configure(org.apache.hadoop.mapred.JobConf conf)
Setup Mapper. |
protected java.lang.Object |
get(java.lang.String key)
Get object from JobConf , assuming the value
is Base64 encoded, and can be decoded back to Java
object. |
java.lang.String |
getDatasetID()
Get the current dataset ID. |
void |
joinmap(IK key,
IV value,
org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.WritableComparable<?>,org.apache.hadoop.io.WritableComparable<?>> output,
org.apache.hadoop.mapred.Reporter reporter)
map() |
protected void |
outputRecords(Tuple key,
Tuple value,
org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.WritableComparable<?>,org.apache.hadoop.io.WritableComparable<?>> output)
|
abstract Tuple |
parse(IK inkey,
IV invalue)
Parse the input key and input value into Tuple |
protected void |
updateCounter(java.lang.String group,
java.lang.String couter,
long number)
update certain counter |
Methods inherited from class com.ebay.erl.mobius.core.datajoin.DataJoinMapper |
---|
extractSortValueKeyword, getSortValueComparator, map |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
protected TupleCriterion tuple_criteria
protected java.lang.String[] key_columns
Mapper
protected java.lang.String[] value_columns
Mapper
protected java.lang.String[] projection_order
protected java.lang.String currentDatasetID
protected java.lang.String dataset_display_id
The name is normalized from the currentDatasetID
by
removing the serial number part.
protected com.ebay.erl.mobius.core.mapred.CounterUpdateThread counterThread
protected long _COUNTER_INPUT_RECORD
#INPUT_RECORDS = #FILTERED_RECORDS + #OUTPUT_RECORDS.
protected long _COUNTER_OUTPUT_RECORD
protected long _COUNTER_FILTERED_RECORD
tuple_criteria
.
protected long _COUNTER_INVALIDATE_FORMAT_RECORD
protected java.util.List<ComputedColumns> computedColumns
ComputedColumns
specified by user.
protected boolean _IS_MAP_ONLY_JOB
public static final long _100MB
protected boolean reporterSet
Constructor Detail |
---|
public AbstractMobiusMapper()
Method Detail |
---|
public void configure(org.apache.hadoop.mapred.JobConf conf)
Override this method if there is extra initial settings need to be done.
Make sure to call super.configure(JobConf)
when overriding.
configure
in interface org.apache.hadoop.mapred.JobConfigurable
configure
in class DataJoinMapper<IK,IV,org.apache.hadoop.io.WritableComparable<?>,org.apache.hadoop.io.WritableComparable<?>>
public void joinmap(IK key, IV value, org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.WritableComparable<?>,org.apache.hadoop.io.WritableComparable<?>> output, org.apache.hadoop.mapred.Reporter reporter) throws java.io.IOException
joinmap
in class DataJoinMapper<IK,IV,org.apache.hadoop.io.WritableComparable<?>,org.apache.hadoop.io.WritableComparable<?>>
java.io.IOException
protected void outputRecords(Tuple key, Tuple value, org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.WritableComparable<?>,org.apache.hadoop.io.WritableComparable<?>> output) throws java.io.IOException
java.io.IOException
public void close() throws java.io.IOException
close
in interface java.io.Closeable
close
in class org.apache.hadoop.mapred.MapReduceBase
java.io.IOException
public abstract Tuple parse(IK inkey, IV invalue) throws java.lang.IllegalArgumentException, java.io.IOException
Tuple
java.lang.IllegalArgumentException
java.io.IOException
protected final void updateCounter(java.lang.String group, java.lang.String couter, long number)
public final java.lang.String getDatasetID()
getDatasetID
in class DataJoinMapper<IK,IV,org.apache.hadoop.io.WritableComparable<?>,org.apache.hadoop.io.WritableComparable<?>>
protected final java.lang.Object get(java.lang.String key) throws java.io.IOException
JobConf
, assuming the value
is Base64 encoded, and can be decoded back to Java
object.
If the value from JobConf
for the given
key
is null or empty, null is returned.
java.io.IOException
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |