Skip to content
DC' Blog
Go back

03-polars Series

Tip

所有的代码分析基于 polars 1.43.0 版本

分析polars的Series

Series 是 Polars 的核心列式数据结构,上接 Python API,下接 Arrow 内存格式和 ChunkedArray。本文分析其从 Python 到 Rust 的完整架构和设计亮点。

架构总览

flowchart TD subgraph Python A[series.py: Series class] A --> B["sequence_to_pyseries / numpy_to_pyseries"] end subgraph pyo3 C["crates/polars-python: PySeries(RwLock<Series>)"] end subgraph Rust Core D["crates/polars-core: Series(pub Arc<dyn SeriesTrait>)"] E["SeriesTrait - 所有操作的trait"] F["SeriesWrap<ChunkedArray<T>> - 每个类型的实现"] G["ChunkedArray<T> - 多 chunk 数据容器"] H["Arrow ArrayRef - 底层内存"] end B --> C C --> D D --> E E --> F F --> G G --> H

1. Python 层:用户入口

Python 的 Series 类定义在:

py-polars/src/polars/series/series.pyGitHub
展开折叠代码 (173-253 行,共 81 行)
@expr_dispatch
class Series:
    """
    A Series represents a single column in a Polars DataFrame.

    Parameters
    ----------
    name : str, default None
        Name of the Series. Will be used as a column name when used in a DataFrame.
        When not specified, name is set to an empty string.
    values : ArrayLike, default None
        One-dimensional data in various forms. Supported are: Sequence, Series,
        pyarrow Array, and numpy ndarray.
    dtype : DataType, default None
        Data type of the resulting Series. If set to `None` (default), the data type is
        inferred from the `values` input. The strategy for data type inference depends
        on the `strict` parameter:

        - If `strict` is set to True (default), the inferred data type is equal to the
          first non-null value, or `Null` if all values are null.
        - If `strict` is set to False, the inferred data type is the supertype of the
          values, or :class:`Object` if no supertype can be found. **WARNING**: A full
          pass over the values is required to determine the supertype.
        - If no values were passed, the resulting data type is :class:`Null`.

    strict : bool, default True
        Throw an error if any value does not exactly match the given or inferred data
        type. If set to `False`, values that do not match the data type are cast to
        that data type or, if casting is not possible, set to null instead.
    nan_to_null : bool, default False
        In case a numpy array is used to create this Series, indicate how to deal
        with np.nan values. (This parameter is a no-op on non-numpy data).

    Examples
    --------
    Constructing a Series by specifying name and values positionally:

    >>> s = pl.Series("a", [1, 2, 3])
    >>> s
    shape: (3,)
    Series: 'a' [i64]
    [
        1
        2
        3
    ]

    Notice that the dtype is automatically inferred as a polars Int64:

    >>> s.dtype
    Int64

    Constructing a Series with a specific dtype:

    >>> s2 = pl.Series("a", [1, 2, 3], dtype=pl.Float32)
    >>> s2
    shape: (3,)
    Series: 'a' [f32]
    [
        1.0
        2.0
        3.0
    ]

    It is possible to construct a Series with values as the first positional argument.
    This syntax considered an anti-pattern, but it can be useful in certain
    scenarios. You must specify any other arguments through keywords.

    >>> s3 = pl.Series([1, 2, 3])
    >>> s3
    shape: (3,)
    Series: '' [i64]
    [
        1
        2
        3
    ]
    """

    # NOTE: This `= None` is needed to generate the docs with sphinx_accessor.
    _s: PySeries = None  # type: ignore[assignment]

关键设计:

2. Python ↔ Rust 桥接

crates/polars-python/src/series/mod.rsGitHub
#[pyclass(frozen, from_py_object)]
#[repr(transparent)]
pub struct PySeries {
    pub series: RwLock<Series>,
}

impl Clone for PySeries {
    fn clone(&self) -> Self {
        Self {
            series: RwLock::new(self.series.read().clone()),
        }
    }
}

impl From<Series> for PySeries {
    fn from(series: Series) -> Self {
        Self::new(series)
    }
}

impl PySeries {
    pub(crate) fn new(series: Series) -> Self {
        PySeries {
            series: RwLock::new(series),
        }
    }
}

PySeries 是对 Rust Series 的简单封装:

ToSeries / ToPySeries trait 提供 Vec<PySeries>Vec<Series> 之间的双向转换,简化批量操作。

3. Rust 核心:Series + SeriesTrait

3.1 Series 定义

crates/polars-core/src/series/mod.rsGitHub
#[derive(Clone)]
#[must_use]
pub struct Series(pub Arc<dyn SeriesTrait>);

核心设计模式:类型擦除 + 虚方法调用 (trait object)

Series 本身只是一个 Arc<dyn SeriesTrait> 的包装,不感知具体类型。所有操作通过 SeriesTrait 的虚函数分派到具体实现。

flowchart LR Series -- Arc --> SeriesTrait SeriesTrait --> SeriesWrap_Int32 SeriesTrait --> SeriesWrap_Float64 SeriesTrait --> SeriesWrap_String SeriesTrait --> SeriesWrap_Datetime SeriesWrap_Int32 -- 持有 --> ChunkedArray_Int32 SeriesWrap_Float64 -- 持有 --> ChunkedArray_Float64 SeriesWrap_String -- 持有 --> ChunkedArray_String

3.2 Deref 到 SeriesTrait

crates/polars-core/src/series/mod.rsGitHub
impl Deref for Series {
    type Target = dyn SeriesTrait;

    fn deref(&self) -> &Self::Target {
        self.0.as_ref()
    }
}

通过 Deref,Series 可以直接调用 SeriesTrait 的所有方法(如 filtertakecast 等),无需手动解引用。

3.3 SeriesTrait

crates/polars-core/src/series/series_trait.rsGitHub
pub trait SeriesTrait:
    Send + Sync + private::PrivateSeries + private::PrivateSeriesNumeric
{

SeriesTrait 约定了所有类型必须实现的操作接口:

类别方法
元数据name()dtype()len()null_count()chunks()
切片/过滤slice()filter()take()head()tail()
排序/唯一sort_with()unique()arg_unique()
聚合sum_reduce()min_reduce()max_reduce()mean_reduce()
类型转换cast()to_physical_repr()
空值处理is_null()is_not_null()drop_nulls()
遍历get()get_unchecked()

PrivateSeries 子 trait 提供内部使用的操作(agg、hash、arithmetic 等),对外部隐藏。

设计亮点:所有方法都有默认实现(polars_bail! 或返回全 Null),每个类型只需覆盖自己支持的操作。新类型加入时,只需实现需要的方法,其余自动 fallback。

4. 具体类型的实现

4.1 SeriesWrap + impl_dyn_series! 宏

crates/polars-core/src/series/implementations/mod.rsGitHub
#[repr(transparent)]
pub(crate) struct SeriesWrap<T>(pub T);

impl<T: PolarsDataType> From<ChunkedArray<T>> for SeriesWrap<ChunkedArray<T>> {
    fn from(ca: ChunkedArray<T>) -> Self {
        SeriesWrap(ca)
    }
}

SeriesWrap<T> 是一个 #[repr(transparent)] 的新类型包装,所有 ChunkedArray<T> 通过它来实现 SeriesTrait

impl_dyn_series! 宏为每种物理类型生成 PrivateSeries + SeriesTrait 的实现,包括:

impl_dyn_series!(Int32Chunked, Int32Type);
impl_dyn_series!(UInt32Chunked, UInt32Type);
impl_dyn_series!(Float64Chunked, Float64Type);
impl_dyn_series!(BooleanChunked, BooleanType);
// ...等

核心方法实现模式——委派给 ChunkedArray 的同名方法

fn filter(&self, filter: &BooleanChunked) -> PolarsResult<Series> {
    ChunkFilter::filter(&self.0, filter).map(|ca| ca.into_series())
}
fn sort_with(&self, options: SortOptions) -> PolarsResult<Series> {
    Ok(ChunkSort::sort_with(&self.0, options).into_series())
}

4.2 浮点类型的独立实现

浮动类型(Float16/Float32/Float64)的 impl_dyn_series!floats.rs 而非 mod.rs,因为它们的 mean_reducemedian_reduce 返回原地类型而非 Float64

// mod.rs (整数类型):
fn mean_reduce(&self) -> PolarsResult<Scalar> {
    Ok(Scalar::new(DataType::Float64, self.mean().into()))
}

// floats.rs (浮点类型):
fn mean_reduce(&self) -> PolarsResult<Scalar> {
    let mean = self.mean().map(AsPrimitive::as_);
    Ok(Scalar::new(self.dtype().clone(), mean.into()))
}

4.3 Boolean 的独立实现

Boolean 的算术操作只有 add_to(因为布尔值支持按位逻辑运算),且 and_reduce/or_reduce/xor_reduce 使用了 polars_compute::bitwise::BitwiseKernel 进行位运算聚合。

4.4 逻辑类型的实现(Datetime 示例)

对于 DatetimeChunked 这样的逻辑类型,实现模式是所有操作在物理层完成,再封装回逻辑类型

fn filter(&self, filter: &BooleanChunked) -> PolarsResult<Series> {
    self.0.physical().filter(filter).map(|ca| {
        ca.into_datetime(self.0.time_unit(), self.0.time_zone().clone())
            .into_series()
    })
}

算术操作也遵循类型语义:

crates/polars-core/src/series/implementations/datetime.rsGitHub
    fn subtract(&self, rhs: &Series) -> PolarsResult<Series> {
        match (self.dtype(), rhs.dtype()) {
            (DataType::Datetime(tu, tz), DataType::Datetime(tur, tzr)) => {
                assert_eq!(tu, tur);
                assert_eq!(tz, tzr);
                let lhs = self.cast(&DataType::Int64, CastOptions::NonStrict).unwrap();
                let rhs = rhs.cast(&DataType::Int64).unwrap();
                Ok(lhs.subtract(&rhs)?.into_duration(*tu).into_series())
            },
            (DataType::Datetime(tu, tz), DataType::Duration(tur)) => {
                assert_eq!(tu, tur);
                let lhs = self.cast(&DataType::Int64, CastOptions::NonStrict).unwrap();
                let rhs = rhs.cast(&DataType::Int64).unwrap();
                Ok(lhs
                    .subtract(&rhs)?
                    .into_datetime(*tu, tz.clone())
                    .into_series())
            },
            (dtl, dtr) => polars_bail!(opq = sub, dtl, dtr),
        }
    }
    fn add_to(&self, rhs: &Series) -> PolarsResult<Series> {
展开折叠代码 (134-146 行,共 13 行)
        match (self.dtype(), rhs.dtype()) {
            (DataType::Datetime(tu, tz), DataType::Duration(tur)) => {
                assert_eq!(tu, tur);
                let lhs = self.cast(&DataType::Int64, CastOptions::NonStrict).unwrap();
                let rhs = rhs.cast(&DataType::Int64).unwrap();
                Ok(lhs
                    .add_to(&rhs)?
                    .into_datetime(*tu, tz.clone())
                    .into_series())
            },
            (dtl, dtr) => polars_bail!(opq = add, dtl, dtr),
        }
    }

4.5 NullChunked — 特殊的独立实现

NullChunked 不是一个 SeriesWrap<ChunkedArray<T>>,而是一个独立的结构体,直接实现 SeriesTrait

crates/polars-core/src/series/implementations/null.rsGitHub
#[derive(Clone)]
pub struct NullChunked {
    pub(crate) name: PlSmallStr,
    length: usize,
    // we still need chunks as many series consumers expect
    // chunks to be there
    chunks: Vec<ArrayRef>,
}

所有操作都是空操作或返回空结果:

5. 从 Arrow 数据构造 Series

crates/polars-core/src/series/from.rsGitHub
    pub unsafe fn from_chunks_and_dtype_unchecked(
        name: PlSmallStr,
        chunks: Vec<ArrayRef>,
        dtype: &DataType,
    ) -> Self {
        use DataType::*;
        match dtype {
            Int8 => Int8Chunked::from_chunks(name, chunks).into_series(),
            Int16 => Int16Chunked::from_chunks(name, chunks).into_series(),
            Int32 => Int32Chunked::from_chunks(name, chunks).into_series(),
            Int64 => Int64Chunked::from_chunks(name, chunks).into_series(),
            UInt8 => UInt8Chunked::from_chunks(name, chunks).into_series(),
            UInt16 => UInt16Chunked::from_chunks(name, chunks).into_series(),
            UInt32 => UInt32Chunked::from_chunks(name, chunks).into_series(),
            UInt64 => UInt64Chunked::from_chunks(name, chunks).into_series(),
展开折叠代码 (75-158 行,共 84 行)
            #[cfg(feature = "dtype-i128")]
            Int128 => Int128Chunked::from_chunks(name, chunks).into_series(),
            #[cfg(feature = "dtype-u128")]
            UInt128 => UInt128Chunked::from_chunks(name, chunks).into_series(),
            #[cfg(feature = "dtype-date")]
            Date => Int32Chunked::from_chunks(name, chunks)
                .into_date()
                .into_series(),
            #[cfg(feature = "dtype-time")]
            Time => Int64Chunked::from_chunks(name, chunks)
                .into_time()
                .into_series(),
            #[cfg(feature = "dtype-duration")]
            Duration(tu) => Int64Chunked::from_chunks(name, chunks)
                .into_duration(*tu)
                .into_series(),
            #[cfg(feature = "dtype-datetime")]
            Datetime(tu, tz) => Int64Chunked::from_chunks(name, chunks)
                .into_datetime(*tu, tz.clone())
                .into_series(),
            #[cfg(feature = "dtype-decimal")]
            Decimal(precision, scale) => Int128Chunked::from_chunks(name, chunks)
                .into_decimal_unchecked(*precision, *scale)
                .into_series(),
            #[cfg(feature = "dtype-array")]
            Array(_, _) => {
                ArrayChunked::from_chunks_and_dtype_unchecked(name, chunks, dtype.clone())
                    .into_series()
            },
            List(_) => ListChunked::from_chunks_and_dtype_unchecked(name, chunks, dtype.clone())
                .into_series(),
            String => StringChunked::from_chunks(name, chunks).into_series(),
            Binary => BinaryChunked::from_chunks(name, chunks).into_series(),
            #[cfg(feature = "dtype-categorical")]
            dt @ (Categorical(_, _) | Enum(_, _)) => {
                with_match_categorical_physical_type!(dt.cat_physical().unwrap(), |$C| {
                    let phys = ChunkedArray::from_chunks(name, chunks);
                    CategoricalChunked::<$C>::from_cats_and_dtype_unchecked(phys, dt.clone()).into_series()
                })
            },
            Boolean => BooleanChunked::from_chunks(name, chunks).into_series(),
            #[cfg(feature = "dtype-f16")]
            Float16 => Float16Chunked::from_chunks(name, chunks).into_series(),
            Float32 => Float32Chunked::from_chunks(name, chunks).into_series(),
            Float64 => Float64Chunked::from_chunks(name, chunks).into_series(),
            BinaryOffset => BinaryOffsetChunked::from_chunks(name, chunks).into_series(),
            #[cfg(feature = "dtype-extension")]
            Extension(typ, storage) => ExtensionChunked::from_storage(
                typ.clone(),
                Series::from_chunks_and_dtype_unchecked(name, chunks, storage),
            )
            .into_series(),
            #[cfg(feature = "dtype-struct")]
            Struct(_) => {
                let mut ca =
                    StructChunked::from_chunks_and_dtype_unchecked(name, chunks, dtype.clone());
                StructChunked::propagate_nulls_mut(&mut ca);
                ca.into_series()
            },
            #[cfg(feature = "object")]
            Object(_) => {
                if let Some(arr) = chunks[0].as_any().downcast_ref::<FixedSizeBinaryArray>() {
                    assert_eq!(chunks.len(), 1);
                    // SAFETY:
                    // this is highly unsafe. it will dereference a raw ptr on the heap
                    // make sure the ptr is allocated and from this pid
                    // (the pid is checked before dereference)
                    {
                        let pe = PolarsExtension::new(arr.clone());
                        let s = pe.get_series(&name);
                        pe.take_and_forget();
                        s
                    }
                } else {
                    unsafe { get_object_builder(name, 0).from_chunks(chunks) }
                }
            },
            Null => new_null(name, &chunks),
            Unknown(_) => {
                panic!("dtype is unknown; consider supplying data-types for all operations")
            },
            #[allow(unreachable_patterns)]
            _ => unreachable!(),
        }
    }

from_chunks_and_dtype_unchecked 是系列构造的统一入口。它接受 Vec<ArrayRef>DataType,据此分派到对应的 ChunkedArray::from_chunks

此外还提供了 _try_from_arrow_unchecked 方法,可以直接从 Arrow 的 DataType 创建 Series,自动处理 Arrow ↔ Polars 类型的映射(如 ArrowDataType::Date32DataType::Int32 + into_date())。

6. 设计哲学:Python 不可变 vs Rust 可变

PySeries 用了 #[pyclass(frozen)](Python 层面不可变)+ RwLock<Series>(内部可变),这和直觉相反,却是刻意的设计:

1. 跟计算模型一致

Polars 的查询引擎基于表达式 + 惰性计算,数据流本质上是函数式的:s.sort() 返回新 Series,不改原值。Python 暴露可变接口会破坏查询优化的推理(谓词下推、公共子表达式消除等)。

2. 线程安全

Rust 侧执行时会释放 GIL 做并行计算。如果 PySeries 本身可变,另一个线程通过 DataFrame 引用同一块数据就可能产生数据竞争。frozen + RwLock 让 Python 层面安全共享,Rust 内部可控地修改。

3. 跟 Arrow 内存模型对齐

Arrow 数组本身不可变,“追加” 是追加新 chunk 而非修改已有内存。Rust 侧的可变性服务于:追加 chunk、改名字、标记排序状态等元数据操作,大块数据从不原地修改。

4. 接口更干净

Python 用户不需要区分 s.sort()(原地)和 s.sorted()(返回新值),永远只有 s.sort() 返回新 Series。

一句话:对外 (Python) 承诺不可变让代码安全可预测;对内 (Rust) 保留可变能力满足实际需求。

设计亮点总结

  1. 类型擦除 + 虚方法分派Series(pub Arc<dyn SeriesTrait>) 是核心,一个统一类型包装所有具体类型,通过 trait object 实现动态分派

  2. 宏驱动实现impl_dyn_series! 宏为每种物理类型生成 ~400 行样板代码,只需一两行宏调用;特殊类型(float、boolean、datetime)覆盖宏生成的默认实现

  3. 物理/逻辑类型分离 — 逻辑类型(Date、Datetime、Duration)在物理类型(Int32、Int64)之上添加语义,通过 self.0.physical() 获取底层数据操作,结果再封装回逻辑类型

  4. NullChunked 零成本表示 — Null 类型不存储实际数据,只需要 lengthchunks 字段,所有操作都是 O(1)

  5. Arc 共享 + COWSeries 内部用 Arc 共享数据,_get_inner_mut 在需要修改时才 clone(写时复制),保证零拷贝切片的高效性

  6. Python↔Rust 零拷贝 — 数据始终在 Rust 端,Python Series.__init__ 的所有路径最终都调用 Rust 的构造方法,不经过中间序列化



Previous Post
04-polars DataFrame
Next Post
02-polars数据类型