Move Spark data into Cassandra.

What it does

The connector reads Cassandra tables as Spark data and writes Spark data back to Cassandra. It supports DataFrames, RDDs, arbitrary CQL, server-side filtering, replica-aware partitioning, collections, UDTs, vector types, and Scala 2.12/2.13 builds for Spark 3.5.

Project coordinates use the io.teknek.arizona.spark namespace. The 10.0.1 work is aimed at making streaming updates and integration testing practical.

Structured Streaming Update mode

Stateful Spark aggregations do not only produce a final answer when a window closes. In Update mode, changed aggregate rows are written on each microbatch. Cassandra receives those rows as upserts while the window is active.

val metrics = events
  .withWatermark("event_time", "10 minutes")
  .groupBy(window(col("event_time"), "10 minutes"), col("content_id"))
  .count()

metrics.writeStream
  .format("org.apache.spark.sql.cassandra")
  .option("keyspace", "streaming_metrics")
  .option("table", "clickstream_metrics")
  .option("checkpointLocation", "/var/lib/spark/checkpoints/clickstream-metrics")
  .outputMode(OutputMode.Update())
  .start()

The Cassandra primary key must identify the aggregate row, usually the grouping key plus the window start. Later updates replace the earlier value for that key.

Docker-based integration tests

The test workflow uses Docker to provide Cassandra and keeps the test environment reproducible. In Airlock, use the nested rootless Docker daemon rather than the host Docker socket.

export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
export PATH="$JAVA_HOME/bin:$PATH"

./sbt/sbt test
./sbt/sbt it:test

For a targeted Testcontainers integration test inside Airlock:

start-rootless-docker env \
  JAVA_HOME=/usr/lib/jvm/java-17-openjdk \
  PATH=/usr/lib/jvm/java-17-openjdk/bin:$PATH \
  TESTCONTAINERS_HOST_OVERRIDE=127.0.0.1 \
  ./sbt/sbt \
  "connector/IntegrationTest/testOnly io.teknek.arizona.spark.connector.sql.CassandraStreamingSinkSpec"

Continue into the project

Read the streaming guide, the developer notes, and the change history. The connector is an Apache-licensed fork maintained under the Arizona project namespace.