I have found a user defined function to convert the ip to long in Spark, but I need to write it in pyspark. I am new to both of them and I do not know how to do that. Could someone help? Here is the function:
val ipToLong = udf((ip: String) => {
if (ip != null && ip != "" && ip.contains(".")) {
val atoms: Array[Long] = ip.split("\\.").map(java.lang.Long.parseLong(_))
val result: Long = (3 to 0 by -1).foldLeft(0L)(
(result, position) => result | (atoms(3 - position) << position * 8))
result & 0xFFFFFFFF
} else {
-1
}
})
User contributions licensed under CC BY-SA 3.0