org.apache.spark.sql.execution

Class EvaluatePython

Object

org.apache.spark.sql.catalyst.trees.TreeNode<PlanType>

org.apache.spark.sql.catalyst.plans.QueryPlan<org.apache.spark.sql.catalyst.plans.logical.LogicalPlan>

org.apache.spark.sql.catalyst.plans.logical.LogicalPlan

org.apache.spark.sql.catalyst.plans.logical.UnaryNode

org.apache.spark.sql.execution.EvaluatePython

All Implemented Interfaces:

java.io.Serializable, Logging, org.apache.spark.sql.catalyst.trees.UnaryNode<org.apache.spark.sql.catalyst.plans.logical.LogicalPlan>, scala.Equals, scala.Product

public class EvaluatePython
extends org.apache.spark.sql.catalyst.plans.logical.UnaryNode
implements scala.Product, scala.Serializable

:: DeveloperApi :: Evaluates a PythonUDF, appending the result to the end of the input tuple.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
EvaluatePython(PythonUDF udf, org.apache.spark.sql.catalyst.plans.logical.LogicalPlan child, org.apache.spark.sql.catalyst.expressions.AttributeReference resultAttribute)

Method Summary

Methods

Modifier and Type	Method and Description
static EvaluatePython	apply(PythonUDF udf, org.apache.spark.sql.catalyst.plans.logical.LogicalPlan child)
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan	child()
static Object	fromJava(Object obj, org.apache.spark.sql.catalyst.types.DataType dataType)
scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.Attribute>	output()
org.apache.spark.sql.catalyst.expressions.AttributeReference	resultAttribute()
static Object[]	rowToArray(org.apache.spark.sql.catalyst.expressions.Row row, scala.collection.Seq<org.apache.spark.sql.catalyst.types.DataType> fields) Convert Row into Java Array (for pickled into Python)
static Object	toJava(Object obj, org.apache.spark.sql.catalyst.types.DataType dataType) Helper for converting a Scala object to a java suitable for pyspark serialization.
PythonUDF	udf()

Methods inherited from class org.apache.spark.sql.catalyst.plans.logical.UnaryNode

children

Methods inherited from class org.apache.spark.sql.catalyst.plans.logical.LogicalPlan

childrenResolved, cleanArgs, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$Logging$$log__$eq, org$apache$spark$Logging$$log_, resolve, resolve, resolveChildren, resolved, sameResult, statePrefix, statistics

Methods inherited from class org.apache.spark.sql.catalyst.plans.QueryPlan

expressions, inputSet, missingInput, org$apache$spark$sql$catalyst$plans$QueryPlan$$transformExpressionDown$1, org$apache$spark$sql$catalyst$plans$QueryPlan$$transformExpressionUp$1, outputSet, printSchema, references, schema, schemaString, simpleString, transformAllExpressions, transformExpressions, transformExpressionsDown, transformExpressionsUp

Methods inherited from class org.apache.spark.sql.catalyst.trees.TreeNode

apply, argString, asCode, collect, fastEquals, flatMap, foreach, generateTreeString, getNodeNumbered, makeCopy, map, mapChildren, nodeName, numberedTreeString, otherCopyArgs, stringArgs, toString, transform, transformChildrenDown, transformChildrenUp, transformDown, transformUp, treeString, withNewChildren

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface scala.Product

productArity, productElement, productIterator, productPrefix

Methods inherited from interface scala.Equals

canEqual, equals

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, log_

Constructor Detail

EvaluatePython

public EvaluatePython(PythonUDF udf,
              org.apache.spark.sql.catalyst.plans.logical.LogicalPlan child,
              org.apache.spark.sql.catalyst.expressions.AttributeReference resultAttribute)

Method Detail

apply

public static EvaluatePython apply(PythonUDF udf,
                   org.apache.spark.sql.catalyst.plans.logical.LogicalPlan child)

toJava

public static Object toJava(Object obj,
            org.apache.spark.sql.catalyst.types.DataType dataType)

Helper for converting a Scala object to a java suitable for pyspark serialization.

rowToArray

public static Object[] rowToArray(org.apache.spark.sql.catalyst.expressions.Row row,
                  scala.collection.Seq<org.apache.spark.sql.catalyst.types.DataType> fields)

Convert Row into Java Array (for pickled into Python)

fromJava

public static Object fromJava(Object obj,
              org.apache.spark.sql.catalyst.types.DataType dataType)

udf

public PythonUDF udf()

child

public org.apache.spark.sql.catalyst.plans.logical.LogicalPlan child()

Specified by:

child in interface org.apache.spark.sql.catalyst.trees.UnaryNode<org.apache.spark.sql.catalyst.plans.logical.LogicalPlan>

resultAttribute

public org.apache.spark.sql.catalyst.expressions.AttributeReference resultAttribute()

output

public scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.Attribute> output()

Specified by:

output in class org.apache.spark.sql.catalyst.plans.QueryPlan<org.apache.spark.sql.catalyst.plans.logical.LogicalPlan>