Releases: spotify/scio
Releases · spotify/scio
v0.14.9
What's changed
Includes Beam 2.60.0 support.
🚀 Enhancements
- Add missing fanout SCollection API (#5497) @RustedBones
- Add latest API to SCollection (#5495) @RustedBones
- Add top to collections with fanout (#5487) @RustedBones
🐛 Bug Fixes
- Fix RemoteFileUtil to download in parallel as expected. (#5515) @psobot
- Fix file batch operation (#5519) @RustedBones
📗 Documentation
- Prepare for release 0.14.9 (#5522) @RustedBones
🏗️ Build Improvements
- Remove deprecated usage of tlSonatypeUseLegacyHost in build (#5517) @RustedBones
- Update sbt-bom to 1.0.16 (#5501) @scala-steward
- Update sbt-mdoc to 2.6.1 (#5496) @scala-steward
- Update sbt, sbt-dependency-tree to 1.10.2 (#5490) @scala-steward
🔧 Refactorings
- Consistent implementation for reduce operation (#5498) @RustedBones
🌱 Dependency Updates
- Update beam to 2.60 (#5516) @RustedBones
- Update socco-ng to 0.1.13 (#5518) @RustedBones
- Update scala-compiler, scala-library, ... to 2.13.15 (#5504) @scala-steward
- Update elasticsearch-java to 8.15.2 (#5503) @scala-steward
- Update jedis to 5.2.0 (#5507) @scala-steward
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.21.0 (#5509) @scala-steward
- Update metrics-core to 4.2.28 (#5511) @scala-steward
- Update scalacheck to 1.18.1 (#5491) @scala-steward
- Update jna to 5.15.0 (#5489) @scala-steward
- Update munit to 1.0.2 (#5492) @scala-steward
Contributors to this release
v0.14.8
What's changed
Includes Beam 2.59.0 support.
🐛 Bug Fixes
- (fix #5472) Set desiredBundleSizeBytes to Long.MaxValue BinaryIO reads to prevent file subrange-splitting (#5473) @clairemcginty
- use noOp cache instead of null when instantiating typedSparkeySideInput (#5469) @pgoggijr
📗 Documentation
- Prepare release 0.14.8 (#5486) @RustedBones
- Fix redis examples (#5482) @kellen
- Fixes a minor formatting issue in the FAQs (#5467) @luciomartinez
🧪 Test Improvements
- Add test for empty string arg (#5485) @RustedBones
- Support pubsub epoch timestamp attribute in test (#5484) @RustedBones
🏗️ Build Improvements
- Update sbt-typelevel to 0.7.3 (#5480) @scala-steward
- Update sbt-scoverage to 2.2.0 (#5479) @scala-steward
- Update sbt-avro to 3.5.0 (#5461) @scala-steward
🌱 Dependency Updates
- Update beam to 2.59 (#5471) @RustedBones
- Update elasticsearch-java to 8.15.1 (#5474) @scala-steward
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.20.1 (#5475) @scala-steward
- Update circe-core, circe-generic, ... to 0.14.10 (#5476) @scala-steward
- Update scala-compiler, scala-library, ... to 2.12.20 (#5478) @scala-steward
- Update parquet-avro, parquet-column, ... to 1.14.2 (#5477) @scala-steward
- Update metrics-core to 4.2.27 (#5463) @scala-steward
- Update mssql-jdbc to 12.8.1.jre11 (#5462) @scala-steward
- Update jedis to 5.1.5 (#5465) @scala-steward
Contributors to this release
@RustedBones, @clairemcginty, @kellen, @luciomartinez, @pgoggijr and @scala-steward
v0.14.7
What's changed
Includes Beam 2.58.1 support.
🚀 Enhancements
🐛 Bug Fixes
- Fix Async DoFn to emit element with restored metadata (#5441) @RustedBones
📜 Scalafix Migrations
- Improve accuracy of FixQuery rule (#5438) @clairemcginty
📗 Documentation
- Update version matrix with scio 0.14.7 (#5460) @RustedBones
🏗️ Build Improvements
- Ignore updates from zstd-jni and sync with beam's version (#5457) @RustedBones
- Update scalac-options to 0.1.7 (#5454) @scala-steward
- Update scalafmt-core to 3.8.3 (#5453) @scala-steward
- Move files in folder matching package (#5445) @RustedBones
- Enforce javafmt on CI (#5442) @RustedBones
- Disable MiMa for scio-bom (#5439) @RustedBones
🌱 Dependency Updates
- Update beam to 2.58.1 (#5459) @RustedBones
- Update elasticsearch-java to 8.15.0 (#5446) @scala-steward
- Update jedis to 5.1.4 (#5455) @scala-steward
- Update munit to 1.0.1 (#5452) @scala-steward
- Update voyager to 2.0.9 (#5450) @scala-steward
- Update mssql-jdbc to 12.8.0.jre11 (#5449) @scala-steward
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.20.0 (#5448) @scala-steward
- Update beam to 2.58 (#5436) @RustedBones
Contributors to this release
v0.14.6
What's changed
Includes Beam 2.57.0 support.
🚀 Enhancements
- Publish scio-bom (#5423) @RustedBones
- Support BigQuery JSON column type (#5416) @turb
🐛 Bug Fixes
- fix: use fully qualified types in macros (#5434) @RustedBones
- [gcp] Fix BQ BIGNUMERIC type support (#5428) @RustedBones
- Use hash join when writing sparkey (#5402) @aslotnick
📗 Documentation
- Prepare release v0.14.6 (#5437) @RustedBones
- doc: Update doc regarding avro version and logical-type support (#5435) @RustedBones
🏗️ Build Improvements
- Update sbt, sbt-dependency-tree to 1.10.1 (#5426) @scala-steward
- Update sbt-mdoc to 2.5.4 (#5427) @scala-steward
- Update sbt-bom to 1.0.14 (#5425) @scala-steward
- Use sbt-bom (#5414) @RustedBones
- Update sbt-scoverage to 2.1.0 (#5422) @scala-steward
- Update scalafmt-core to 3.8.2 (#5399) @scala-steward
- Derive artifact dependent versions (#5395) @RustedBones
🌱 Dependency Updates
- Update magnolify to 0.7.4 (#5432) @RustedBones
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.19.1 (#5424) @scala-steward
- Update elasticsearch-java to 8.14.3 (#5429) @scala-steward
- Update beam to 2.57.0 (#5413) @RustedBones
- Use default beam avro version (#5415) @RustedBones
- Update circe-core, circe-generic, ... to 0.14.9 (#5420) @scala-steward
- Update mysql-connector-j to 9.0.0 (#5419) @scala-steward
- Update neo4j-java-driver to 4.4.18 (#5412) @scala-steward
- Update scalatest to 3.2.19 (#5408) @scala-steward
- Update mssql-jdbc to 12.6.3.jre11 (#5403) @scala-steward
- Update parquet-avro, parquet-column, ... to 1.14.1 (#5405) @scala-steward
- Update testcontainers-scala-elasticsearch, ... to 0.41.4 (#5398) @scala-steward
- Update metrics-core to 4.2.26 (#5394) @scala-steward
Contributors to this release
v0.14.5
What's changed
Includes Beam 2.56.0 support.
warning: This release transitively pulls newer avro and jackson versions. If you need to depend on previous versions, you'll need to pin them in your build.
🚀 Enhancements
- (#5296) Support Parquet predicates/projections in tests (#5309) @clairemcginty
- Add support for Zstd coders (#5321) @kellen
- [scio-core](feature) Sample SCollection with max weight (#5352) @RustedBones
- [scio-core](feature) Add readFiles and readFilesWithPath apis (#5350) @RustedBones
🐛 Bug Fixes
- Fix sparkey loading issue for unknown keys (#5387) @RustedBones
- (fix)[avro] Use correct DatumWriter constructor (#5371) @RustedBones
- fix FixQuery scalafix rule for macro-generated classes (#5356) @clairemcginty
📜 Scalafix Migrations
- Refactor FixAvroCoder case application (#5353) @clairemcginty
- Update FixAvroCoder to match transformed Avro SCollections (#5351) @clairemcginty
📗 Documentation
- Update release table for 0.14.5 (#5392) @RustedBones
- Prefer waitUndilDone over waitUntilFinish in test and examples (#5349) @RustedBones
🧪 Test Improvements
- Drop slow test tag (#5391) @RustedBones
🏗️ Build Improvements
- Update scalac-options to 0.1.5 (#5385) @scala-steward
- Fix assembly conflict (#5382) @RustedBones
- Update munit to 1.0.0 (#5378) @scala-steward
- [avro] Fix CI test for avro latest (#5372) @RustedBones
- Update sbt, sbt-dependency-tree to 1.10.0 (#5365) @scala-steward
- Update sbt-typelevel to 0.7.1 (#5366) @scala-steward
- Update socco-ng to 0.11.12 (#5368) @RustedBones
- Update sbt-scalafix to 0.12.1 (#5357) @scala-steward
- Update sbt-scoverage to 2.0.12 (#5363) @scala-steward
- (chore) Update to codecov-action (#5354) @RustedBones
- Enable mima for test artifacts (#5343) @RustedBones
- Update scalacheck to 1.18.0 (#5344) @scala-steward
🌱 Dependency Updates
- Sync flink artifact (#5393) @RustedBones
- Update elasticsearch-java to 7.17.21 (#5390) @RustedBones
- Update elasticsearch-java to 8.14.0 (#5388) @scala-steward
- Update neo4j-java-driver to 4.4.17 (#5384) @scala-steward
- Update algebra, cats-core, cats-kernel to 2.12.0 (#5383) @scala-steward
- Update magnolify to 0.7.3 (#5381) @RustedBones
- Update mssql-jdbc to 12.6.2.jre11 (#5380) @scala-steward
- Update jedis to 5.1.3 (#5376) @scala-steward
- Update shapeless to 2.3.12 (#5374) @scala-steward
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.18.1 (#5369) @scala-steward
- Update scala-compiler, scala-library, ... to 2.13.14 (#5362) @scala-steward
- Update mysql-connector-j to 8.4.0 (#5359) @scala-steward
- Update beam to 2.56 (#5346) @RustedBones
- Update circe-core, circe-generic, ... to 0.14.7 (#5360) @scala-steward
Contributors to this release
v0.14.4
Includes Beam 2.55.1 support.
🚀 Enhancements
- Robust throwable kryo coder (#5318) @RustedBones
- Use JacksonJsonpMapper as default for Elasticsearch (#5306) @kellen
- Allow String key type to transform SMB sources with CharSequence key (#5297) @clairemcginty
🐛 Bug Fixes
- [scio-avro] fix: allow conversions field in record (#5332) @RustedBones
- Add all avro logical type conversions to model (#5301) @RustedBones
- fix: safer implementation of distinctBy (#5299) @RustedBones
📜 Scalafix Migrations
- scalafix: Rule to migrate deprecated query to queryRaw (#5302) @RustedBones
📗 Documentation
- Prepare release v0.14.4 (#5342) @RustedBones
- [doc] Fix scaladoc link to class declared in package objects (#5316) @RustedBones
- Add staging dir to REPL docs (#5308) @kellen
🧪 Test Improvements
- Move test code in respective projects (#5330) @RustedBones
- [integration] Use literals for fromSchemaFile integration test (#5334) @RustedBones
- [integration] Fix avro integration test (#5333) @RustedBones
🏗️ Build Improvements
- Update sbt-site, sbt-site-paradox to 1.7.0 (#5324) @scala-steward
- Update sbt-paradox to 0.10.7 (#5325) @scala-steward
- Update scalafmt-core to 3.8.1 (#5320) @scala-steward
- Update sbt-buildinfo to 0.12.0 (#5310) @scala-steward
- Update sbt-assembly to 2.2.0 (#5304) @scala-steward
🌱 Dependency Updates
- Update scalacheck to 1.17.1 (#5341) @scala-steward
- Update cassandra-all to 3.11.17 (#5340) @scala-steward
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.18.0 (#5339) @scala-steward
- Update pprint to 0.9.0 (#5336) @scala-steward
- Update scala-collection-compat to 2.12.0 (#5337) @scala-steward
- Update beam to 2.55.1 (#5322) @RustedBones
- Update elasticsearch-java to 8.13.2 (#5323) @scala-steward
- Update neo4j-java-driver to 4.4.15 (#5328) @scala-steward
- Update magnolify-avro, magnolify-bigtable, ... to 0.7.2 (#5319) @scala-steward
- Update elasticsearch v7 to 7.17.19 (#5317) @RustedBones
- Update beam to 2.55 (#5307) @RustedBones
- Update sbt-site, sbt-site-paradox to 1.6.0 (#5311) @scala-steward
- Update voyager to 2.0.6 (#5313) @scala-steward
Contributors to this release
v0.14.3
🐛 Bug Fixes
- fix: Execute SmbIO output assertions (#5289) @RustedBones
- (fix #5290) Support empty input in TransformOverride.ofSource (#5293) @clairemcginty
- (Fix #5285) Allow String key type to read SMB sources written with CharSequence key type (#5291) @clairemcginty
🌱 Dependency Updates
- Update jedis to 5.1.2 (#5292) @scala-steward
- Update neo4j-java-driver to 4.4.14 (#5288) @scala-steward
🏗️ Build Improvements
- Update sbt-scalafix to 0.12.0 (#5287) @scala-steward
Contributors to this release
v0.14.2
Contains a bugfix for duplicated SMB Transforms.
🐛 Bug Fixes
- (fix) Remove duplicated transform application (#5283) @RustedBones
🚀 Enhancements
- Make checkVersion more dynamic (#5282) @RustedBones
- Improve implicit coder not found message (#5281) @RustedBones
📗Documentation
- (doc) fix 0.14 migration guide broken links (#5278) @RustedBones
🏗️ Build Improvements
- Update scala to 2.13.13 and 2.12.19 (#5276) @RustedBones
Contributors to this release
v0.14.1
Includes Beam 2.54.0 support.
🚀 Enhancements
- Add layer for low priority coder conflict (#5274) @RustedBones
- Add support of BIGNUMERIC in BigQueryIO (#5225) @shnapz
- Change dataflow runner check for parquet splittable do fn (#5264) @RustedBones
- Make sure Coder.gen produces informative error (#5258) @RustedBones
- Add default coder for ResourceId (#5244) @clairemcginty
- Add type bound for parquet-avro SCollection ops (#5230) @RustedBones
🐛 Bug Fixes
- (fix) Update SmbIO to support absolute path in test (#5277) @RustedBones
- Simplify BucketedInput serialization (#5270) @clairemcginty
- Fix byte[] equality issue for MockByteArraySparkeyReader (#5262) @clairemcginty
- Fix scio-repl assembly after bream 2.54 upgrade (#5263) @RustedBon
- Always set Parquet-Avro projection (#5234) @clairemcginty
- Fix SmbIO's testId naming (#5228) @clairemcginty
- Sort Path IDs in SmbIO (#5242) @clairemcginty
- Fix SmbIO side input support (fix #5240) (#5241) @clairemcginty
📜 Scalafix Migrations
- Make FixLogicalTypeSupplier more permissive (#5233) @clairemcginty
- Fix MatchError for FixAvroCoder (#5269) @clairemcginty
- Broaden use cases to FixAvroCoder (#5267) @clairemcginty
- Update FixAvroCoder to include SMB Avro reads (#5245) @clairemcginty
- Fix Scalafix rule for Avro package import (#5239) @clairemcginty
- Add Scalafix rule for SMB CharSequence key (#5236) @clairemcginty
- Scalafix: Add Avro coder import for SpecificRecord JobTest IOs (#5237) @clairemcginty
- Check options in scalafix (#5231) @kellen
- Inline scalafix (#5229) @kellen
📗 Documentation
- Update copyright year for scio site (#5261) @RustedBones
🧪 Test Improvements
- Fix populate it data (#5260) @RustedBones
- Add SortMergeBucketParityIT case for sortMergeTransform (#5224) @clairemcginty
🏗️ Build Improvements
- Update sbt-typelevel to 0.6.7 (#5275) @RustedBones
- Update sbt, sbt-dependency-tree to 1.9.9 (#5272) @scala-steward
- Update sbt-scoverage to 2.0.11 (#5273) @scala-steward
- Use released 0.14.0 scio version in scalafix (#5259) @RustedBones
- Update sbt-paradox-material-theme to 0.7.0 (#5251) @scala-steward
- Update scalafmt-core to 3.8.0 (#5254) @scala-steward
- Update scalactic to 3.2.18 (#5253) @scala-steward
- Update scalatest to 3.2.18 (#5255) @scala-steward
- Update sbt-scoverage to 2.0.10 (#5256) @scala-steward
- Cache TensorFlow Metadata proto files (#5246) @Duhemm
- Fix cache step for scalafix GHA (#5238) @RustedBones
- Bump release-drafter/release-drafter from 5 to 6 (#5226) @dependabot
🌱 Dependency Updates
- Update beam to 2.54.0 (#5235) @RustedBones
- Update elasticsearch-java to 8.12.2 (#5271) @scala-steward
- Update mssql-jdbc to 12.6.1.jre11 (#5268) @scala-steward
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.16.0 (#5249) @scala-steward
- Update testcontainers-scala-elasticsearch, ... to 0.41.3 (#5248) @scala-steward
Contributors to this release
@Duhemm, @RustedBones, @clairemcginty, @dependabot, @dependabot[bot], @kellen, @scala-steward and @shnapz
v0.14.0
What's Changed
Includes Beam 2.53.0 support.
Breaking Changes
- avro removed from
scio-core
. scalafix rules helping: FixAvroCoder, FixAvroSchemasPackage, FixDynamicAvro - some avro API changes . scalafix rules helping: FixGenericAvro.
- fallback kryo coder requires explicit import
- use of official tensorflow metadata
- BigQuery error-info and result handling API change
sio-smb
module not pulling implementation dependenciessio-smb
inJobTest
expectingSmbIO
test input/output
See the Migration Guide for more information.
🚀 Enhancements
- improve testing framework by @RustedBones in #4962
- fs.gs.inputstream.fadvise defaults to SEQUENTIAL by @farzad-sedghi in #5132
- (scio-smb) Support mixed FileOperations per BucketedInput by @clairemcginty in #5064
- Support Tap for SMB writes (addresses #5080) by @clairemcginty in #5144
- feat: add save dynamic csv by @klDen in #5130
- Make Sparkey testable by @kellen in #5128
- Integrate avro datum factory in scio-avro by @RustedBones in #5152
- Handle BQ write result as ClosedTap side output by @RustedBones in #5172
- Support projection in ParquetAvroSortedBucketIO by @clairemcginty in #5173
- Expose bigtable read maxBufferElementCount option by @RustedBones in #5026
- Integrate datum-factory in smb-avro by @RustedBones in #5181
- Require import for kryo implicit fallback coder by @RustedBones in #5199
- Add kryo serializer for GAX api exceptions by @RustedBones in #5198
- Initial Iceberg bucket support by @regadas in #5205
- Add scala enumeration implicit coder by @RustedBones in #5213
- Add SortedBucketTransform counter for records written by @clairemcginty in #5220
- Add PipelineTestUtils helper for Taps by @clairemcginty in #5216
- Add counter for SMB Predicate filtering by @clairemcginty in #5221
🐛 Bug Fixes
- Fix race in sparkey write by @kellen in #4937
- Support null CharSequence keys by @RustedBones in #5113
- Add location to BigQuery LoadOps by @f-loris in #5106
- Fix bigtable option conversion for bulk API by @RustedBones in #5167
- Fix multi-line DML statement detection by @RustedBones in #5169
- (fix #5147) Fix Materialize for elements that match compression encoding signature by @clairemcginty in #5148
- In SMB and ParquetAvroIOTap set GenericDataSupplier and read schemas by @shnapz in #5121
- (bugfix) Set metadata in AvroSortedBucketIO by @clairemcginty in #5184
- (fixes #5193) Serialize BucketMetadata#hashType as String by @clairemcginty in #5194
- Close client in async DoFn by @RustedBones in #5206
- Use non-deprecated version of murmur3_32 by @regadas in #5204
- Use fork-join common pool for async DonFn callbacks by @RustedBones in #5209
📜 Scalafix Migrations
- Add 0.14 scalafix migration for saveAsAvroFile and update avro coder one by @RustedBones in #5215
- Add Scalafix rule for LogicalTypeSupplier removal by @clairemcginty in #5178
📗 Documentation
- Add docs about SMB secondary keys by @kellen in #5095
- Remove references to Spotify FOSS Slack by @BalestraPatrick in #5149
- Convert SortMergeBucketExample to Parquet + update tests by @clairemcginty in #5191
- Fix list in Builtin.md by @kellen in #5214
- Scio 0.14 migration guide by @RustedBones in #5212
- Update scio and beam release table by @RustedBones in #5222
🧪 Test Improvements
- Fix flaky SCollectionTest by @kellen in #5098
- Drop deprecated sbt
IntegrationTest
configuration by @RustedBones in #4971 - Move populate test data to compile scope by @RustedBones in #5138
🏗️ Build Improvements
- Fix compiler warnings by @RustedBones in #4934
- Update sbt-assembly to 2.1.5 by @scala-steward in #5093
- Update sbt-bloop to 1.5.12 by @scala-steward in #5097
- Use sbt-typelevel for build by @RustedBones in #5107
- Ack or fix deprecation warnings by @RustedBones in #5124
- Ack expected non-exhaustive pattern match by @RustedBones in #5126
- Update sbt-jmh to 0.4.7 by @scala-steward in #5166
- Update sbt-typelevel to 0.6.5 by @scala-steward in #5164
- Update sbt-avro to 3.4.4 by @scala-steward in #5157
- Update sbt-mdoc to 2.5.2 by @scala-steward in #5163
- Update sbt, sbt-dependency-tree to 1.9.8 by @scala-steward in #5162
- Build integration test in PRs originating from repo by @RustedBones in #5143
- Remove duplicated scalac option by @RustedBones in #5171
- Handle unused warning as error by @RustedBones in #5180
- Skip dependency check if compile is skipped by @RustedBones in #5188
- Update sbt-protoc to 1.0.7 by @scala-steward in #5196
- Update sbt-paradox to 0.10.6 by @scala-steward in #5210
- Fix jar signing by @RustedBones in #5223
🔧 Refactorings
- Remove avro from scio-core, implement binary file source by @kellen in #4913
- Tensorflow metadata by @RustedBones in #4944
- Use avro builder API by @RustedBones in #5119
- Cleanup deprecated API by @RustedBones in #5134
- Move BQ typed from query to queryRaw by @RustedBones in #5137
- Rename scala SortedBucketIO class to SmbIO by @clairemcginty in #5140
- Rename SortedBucketIOTest to match SmbIO by @clairemcginty in #5146
- Relax versioning regex by @RustedBones in #5139
- Change scio-smb to depend on provided scio io modules by @RustedBones in #5004
- Drop collection compat shim in favor of official scala-collection-compat by @RustedBones in #5069
🌱 Dependency Updates
- Update algebra, cats-core, cats-kernel to 2.10.0 by @scala-steward in #4952
- Update jedis to 5.1.0 by @scala-steward in #5094
- Update metrics-core to 4.2.23 by @scala-steward in #5110
- Update hadoop to v3.2.4 by @RustedBones in #5135
- Update beam to v2.53 by @RustedBones in #5133
- Update scalac-compat-annotation, ... to 0.1.4 by @scala-steward in #5165
- Update neo4j-java-driver to 4.4.13 by @scala-steward in #5161
- Update jna to 5.14.0 by @scala-steward in #5160
- Update magnolify to 0.7.0 by @RustedBones in #5155
- Upgrade Parquet to 0.13.1 by @clairemcginty in #5175
- Update elasticsearch-java to 8.12.0 by @scala-steward in #5185
- Update mysql-connector-j to 8.3.0 by @scala-steward in #5174
- Update cloud-sql-connector-jdbc-sqlserver to 1.15.2 by @scala-steward in #5186
- Update mysql-socket-factory-connector-j-8 to 1.15.2 by @scala-steward in #5187
- Use new vendored Guava version by @clairemcginty in #5195
- Update metrics-core to 4.2.25 by @scala-steward in https://git...