且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

在pyspark数据框中的两个日期之间生成每月时间戳记

更新时间:2022-04-14 09:11:17

假设您具有以下DataFrame:

Suppose you had the following DataFrame:

data = [("2000-01-01","2002-12-01")]
df = spark.createDataFrame(data, ["minDate", "maxDate"])
df.show()
#+----------+----------+
#|   minDate|   maxDate|
#+----------+----------+
#|2000-01-01|2002-12-01|
#+----------+----------+

您可以按照与/a>.

只需将pyspark.sql.functions.datediff替换为pyspark.sql.functions.months_between,然后使用add_months而不是date_add:

Just replace pyspark.sql.functions.datediff with pyspark.sql.functions.months_between, and use add_months instead of date_add:

import pyspark.sql.functions as f

df.withColumn("monthsDiff", f.months_between("maxDate", "minDate"))\
    .withColumn("repeat", f.expr("split(repeat(',', monthsDiff), ',')"))\
    .select("*", f.posexplode("repeat").alias("date", "val"))\
    .withColumn("date", f.expr("add_months(minDate, date)"))\
    .select('date')\
    .show(n=50)
#+----------+
#|      date|
#+----------+
#|2000-01-01|
#|2000-02-01|
#|2000-03-01|
#|2000-04-01|
# ...skipping some rows...
#|2002-10-01|
#|2002-11-01|
#|2002-12-01|
#+----------+