Enable int64 to write as optional timestamp, int32 to write as optional date. #472

DuanWeiFan · 2023-02-06T21:12:57Z

At the moment, schema does not allow to be set from *int32 to date, or *int64 to timestamp.

We would like to add a statement that will redirect to it's Elem() whenever it encounters a reflect.Ptr.

This change was made in the past, but for some reason it was reverted later on. here
Adding similar logic back and update schema_test.go

DuanWeiFan · 2023-03-06T18:49:31Z

Hi, can someone help take a look at this PR?
The main purpose is to support writing pointers as Parquet schema.

DuanWeiFan · 2023-04-27T17:16:11Z

Hi @achille-roussel could you help take a look at this pr?
At the moment parquet SchemaOf does not work for some pointers type. Would like to add this function so that it would work for all different pointers. Thank you.

achille-roussel · 2023-04-27T17:27:20Z

Hello @DuanWeiFan, I left Twilio so I'm currently not part of the maintainer team anymore for parquet-go.

Maybe someone else from the team can pick up the review? @kevinburkesegment @bartleyg

These aren't available outside of the segmentio organization.

ARM64 runners were being used as part of Segment's cloud infrastructure that we don't have access to now. For now, just remove the tests.

Dependabot flagged an issue, and while we are at it update all of the dependencies. This will fail until we rewrite all of the references in this repo to match the new import scheme.

Generated by Craiyon: https://www.craiyon.com/

I noticed that checksum of testdata/issue368.parquet was different meaning we were working with the wrong file. There is no need for further investigation on why we end up with a different file.

use interface{} instead of any

fix: wrong file in testdata

chore: don't skip TestIssue368

fixes segmentio#2 When reconstructing the schema for nested structs we call function `fieldByIndex` to get the pointer to the underlying value of the field that we assign the values we read into. This is the comment on fieldByIndex function ```go // fieldByIndex is like reflect.Value.FieldByIndex but returns the zero-value of // reflect.Value if one of the fields was a nil pointer instead of panicking. ``` In theory if we have a field with value `*Foo` a call to fieldByIndex should do the following when this field is not initialized yet (i.e is Nil) - Create a zero value of `Foo` : `new(Foo)` - Assign the zero value created to the field. Before this change this is what fieldByIndex was doing ```go if v.IsNil() { v = reflect.Value{} break ``` `reflect.Value{}` IS NOT A ZERO VALUE of anything , it is an invalid value. This commit ensures that we correctly initialize zero values of nested pointer fields.

…field fix: zero value of nested field pointer

This reverts commit 4c444d3.

This commit uses sync.Pool and other tricks to ensure `GenericReader` is optimized Here are benchmark results ``` goos: darwin goarch: amd64 pkg: github.com/parquet-go/parquet-go cpu: Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ GenericReader/benchmarkRowType/go1.17-8 220.3µ ± ∞ ¹ 170.5µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/benchmarkRowType/go1.18-8 154.4µ ± ∞ ¹ 106.5µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/booleanColumn/go1.17-8 151.6µ ± ∞ ¹ 116.6µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/booleanColumn/go1.18-8 102.10µ ± ∞ ¹ 75.89µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/int32Column/go1.17-8 141.9µ ± ∞ ¹ 107.7µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/int32Column/go1.18-8 90.40µ ± ∞ ¹ 64.77µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/int64Column/go1.17-8 143.2µ ± ∞ ¹ 108.5µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/int64Column/go1.18-8 92.92µ ± ∞ ¹ 64.91µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/floatColumn/go1.17-8 143.1µ ± ∞ ¹ 109.2µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/floatColumn/go1.18-8 91.16µ ± ∞ ¹ 67.32µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/doubleColumn/go1.17-8 142.8µ ± ∞ ¹ 109.1µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/doubleColumn/go1.18-8 91.87µ ± ∞ ¹ 63.51µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/byteArrayColumn/go1.17-8 190.1µ ± ∞ ¹ 149.6µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/byteArrayColumn/go1.18-8 135.16µ ± ∞ ¹ 98.65µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/fixedLenByteArrayColumn/go1.17-8 153.1µ ± ∞ ¹ 118.3µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/fixedLenByteArrayColumn/go1.18-8 106.51µ ± ∞ ¹ 70.64µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/stringColumn/go1.17-8 176.6µ ± ∞ ¹ 145.6µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/stringColumn/go1.18-8 121.36µ ± ∞ ¹ 92.26µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/indexedStringColumn/go1.17-8 177.4µ ± ∞ ¹ 145.4µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/indexedStringColumn/go1.18-8 123.77µ ± ∞ ¹ 92.53µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/uuidColumn/go1.17-8 151.1µ ± ∞ ¹ 116.8µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/uuidColumn/go1.18-8 99.38µ ± ∞ ¹ 70.54µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/timeColumn/go1.17-8 217.4µ ± ∞ ¹ 191.0µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/timeColumn/go1.18-8 157.1µ ± ∞ ¹ 138.1µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/timeInMillisColumn/go1.17-8 215.3µ ± ∞ ¹ 192.1µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/timeInMillisColumn/go1.18-8 157.1µ ± ∞ ¹ 138.7µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/mapColumn/go1.17-8 2.067m ± ∞ ¹ 2.036m ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/mapColumn/go1.18-8 1.720m ± ∞ ¹ 1.686m ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/decimalColumn/go1.17-8 145.2µ ± ∞ ¹ 111.4µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/decimalColumn/go1.18-8 94.19µ ± ∞ ¹ 71.14µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/contact/go1.17-8 273.4µ ± ∞ ¹ 230.0µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/contact/go1.18-8 201.6µ ± ∞ ¹ 158.1µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/paddedBooleanColumn/go1.17-8 152.5µ ± ∞ ¹ 115.5µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/paddedBooleanColumn/go1.18-8 100.67µ ± ∞ ¹ 75.21µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/optionalInt32Column/go1.17-8 149.0µ ± ∞ ¹ 115.4µ ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/optionalInt32Column/go1.18-8 96.61µ ± ∞ ¹ 69.62µ ± ∞ ¹ ~ (p=1.000 n=1) ² geomean 161.5µ 125.5µ -22.27% ¹ need >= 6 samples for confidence interval at level 0.95 ² need >= 4 samples to detect a difference at alpha level 0.05 │ old.txt │ new.txt │ │ row/s │ row/s vs base │ GenericReader/benchmarkRowType/go1.17-8 4.540M ± ∞ ¹ 5.864M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/benchmarkRowType/go1.18-8 5.889M ± ∞ ¹ 8.535M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/booleanColumn/go1.17-8 6.595M ± ∞ ¹ 8.576M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/booleanColumn/go1.18-8 8.904M ± ∞ ¹ 11.980M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/int32Column/go1.17-8 7.048M ± ∞ ¹ 9.281M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/int32Column/go1.18-8 10.06M ± ∞ ¹ 14.04M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/int64Column/go1.17-8 6.982M ± ∞ ¹ 9.220M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/int64Column/go1.18-8 9.784M ± ∞ ¹ 14.007M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/floatColumn/go1.17-8 6.988M ± ∞ ¹ 9.157M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/floatColumn/go1.18-8 9.972M ± ∞ ¹ 13.505M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/doubleColumn/go1.17-8 7.005M ± ∞ ¹ 9.163M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/doubleColumn/go1.18-8 9.896M ± ∞ ¹ 14.315M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/byteArrayColumn/go1.17-8 5.262M ± ∞ ¹ 6.684M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/byteArrayColumn/go1.18-8 6.726M ± ∞ ¹ 9.216M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/fixedLenByteArrayColumn/go1.17-8 6.533M ± ∞ ¹ 8.454M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/fixedLenByteArrayColumn/go1.18-8 8.535M ± ∞ ¹ 12.870M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/stringColumn/go1.17-8 5.661M ± ∞ ¹ 6.869M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/stringColumn/go1.18-8 7.492M ± ∞ ¹ 9.854M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/indexedStringColumn/go1.17-8 5.636M ± ∞ ¹ 6.876M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/indexedStringColumn/go1.18-8 7.345M ± ∞ ¹ 9.826M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/uuidColumn/go1.17-8 6.616M ± ∞ ¹ 8.559M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/uuidColumn/go1.18-8 9.149M ± ∞ ¹ 12.889M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/timeColumn/go1.17-8 4.599M ± ∞ ¹ 5.235M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/timeColumn/go1.18-8 5.787M ± ∞ ¹ 6.585M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/timeInMillisColumn/go1.17-8 4.645M ± ∞ ¹ 5.205M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/timeInMillisColumn/go1.18-8 5.787M ± ∞ ¹ 6.553M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/mapColumn/go1.17-8 483.7k ± ∞ ¹ 491.2k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/mapColumn/go1.18-8 528.9k ± ∞ ¹ 539.8k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/decimalColumn/go1.17-8 6.888M ± ∞ ¹ 8.976M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/decimalColumn/go1.18-8 9.652M ± ∞ ¹ 12.779M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/contact/go1.17-8 3.658M ± ∞ ¹ 4.348M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/contact/go1.18-8 4.508M ± ∞ ¹ 5.750M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/paddedBooleanColumn/go1.17-8 6.556M ± ∞ ¹ 8.656M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/paddedBooleanColumn/go1.18-8 9.031M ± ∞ ¹ 12.088M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/optionalInt32Column/go1.17-8 6.711M ± ∞ ¹ 8.662M ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/optionalInt32Column/go1.18-8 9.411M ± ∞ ¹ 13.058M ± ∞ ¹ ~ (p=1.000 n=1) ² geomean 5.904M 7.596M +28.65% ¹ need >= 6 samples for confidence interval at level 0.95 ² need >= 4 samples to detect a difference at alpha level 0.05 │ old.txt │ new.txt │ │ B/op │ B/op vs base │ GenericReader/benchmarkRowType/go1.17-8 48000.000 ± ∞ ¹ 4.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/benchmarkRowType/go1.18-8 43656.00 ± ∞ ¹ 13.00 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/booleanColumn/go1.17-8 24000.000 ± ∞ ¹ 2.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/booleanColumn/go1.18-8 21825.000 ± ∞ ¹ 4.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/int32Column/go1.17-8 23999.000 ± ∞ ¹ 2.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/int32Column/go1.18-8 21825.000 ± ∞ ¹ 4.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/int64Column/go1.17-8 23999.000 ± ∞ ¹ 2.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/int64Column/go1.18-8 21825.000 ± ∞ ¹ 4.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/floatColumn/go1.17-8 23999.000 ± ∞ ¹ 2.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/floatColumn/go1.18-8 21824.000 ± ∞ ¹ 4.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/doubleColumn/go1.17-8 23999.000 ± ∞ ¹ 2.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/doubleColumn/go1.18-8 21823.000 ± ∞ ¹ 4.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/byteArrayColumn/go1.17-8 28.851Ki ± ∞ ¹ 5.419Ki ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/byteArrayColumn/go1.18-8 26.235Ki ± ∞ ¹ 4.930Ki ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/fixedLenByteArrayColumn/go1.17-8 24000.000 ± ∞ ¹ 2.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/fixedLenByteArrayColumn/go1.18-8 21825.000 ± ∞ ¹ 4.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/stringColumn/go1.17-8 39.06Ki ± ∞ ¹ 15.64Ki ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/stringColumn/go1.18-8 35.52Ki ± ∞ ¹ 14.22Ki ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/indexedStringColumn/go1.17-8 39.06Ki ± ∞ ¹ 15.64Ki ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/indexedStringColumn/go1.18-8 35.52Ki ± ∞ ¹ 14.22Ki ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/uuidColumn/go1.17-8 23999.000 ± ∞ ¹ 2.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/uuidColumn/go1.18-8 21825.000 ± ∞ ¹ 4.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/timeColumn/go1.17-8 46.87Ki ± ∞ ¹ 23.45Ki ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/timeColumn/go1.18-8 42.62Ki ± ∞ ¹ 21.33Ki ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/timeInMillisColumn/go1.17-8 46.87Ki ± ∞ ¹ 23.45Ki ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/timeInMillisColumn/go1.18-8 42.62Ki ± ∞ ¹ 21.33Ki ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/mapColumn/go1.17-8 728.9Ki ± ∞ ¹ 682.0Ki ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/mapColumn/go1.18-8 460.5Ki ± ∞ ¹ 418.2Ki ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/decimalColumn/go1.17-8 23999.000 ± ∞ ¹ 2.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/decimalColumn/go1.18-8 21825.000 ± ∞ ¹ 4.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/contact/go1.17-8 78.12Ki ± ∞ ¹ 31.28Ki ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/contact/go1.18-8 71.04Ki ± ∞ ¹ 28.45Ki ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/paddedBooleanColumn/go1.17-8 23999.000 ± ∞ ¹ 2.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/paddedBooleanColumn/go1.18-8 21825.000 ± ∞ ¹ 4.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/optionalInt32Column/go1.17-8 24003.000 ± ∞ ¹ 5.000 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/optionalInt32Column/go1.18-8 21828.000 ± ∞ ¹ 7.000 ± ∞ ¹ ~ (p=1.000 n=1) ² geomean 34.41Ki 109.8 -99.69% ¹ need >= 6 samples for confidence interval at level 0.95 ² need >= 4 samples to detect a difference at alpha level 0.05 │ old.txt │ new.txt │ │ allocs/op │ allocs/op vs base │ GenericReader/benchmarkRowType/go1.17-8 1.000k ± ∞ ¹ 0.000k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/benchmarkRowType/go1.18-8 909.0 ± ∞ ¹ 0.0 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/booleanColumn/go1.17-8 1.000k ± ∞ ¹ 0.000k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/booleanColumn/go1.18-8 909.0 ± ∞ ¹ 0.0 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/int32Column/go1.17-8 1.000k ± ∞ ¹ 0.000k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/int32Column/go1.18-8 909.0 ± ∞ ¹ 0.0 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/int64Column/go1.17-8 1.000k ± ∞ ¹ 0.000k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/int64Column/go1.18-8 909.0 ± ∞ ¹ 0.0 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/floatColumn/go1.17-8 1.000k ± ∞ ¹ 0.000k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/floatColumn/go1.18-8 909.0 ± ∞ ¹ 0.0 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/doubleColumn/go1.17-8 1.000k ± ∞ ¹ 0.000k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/doubleColumn/go1.18-8 909.0 ± ∞ ¹ 0.0 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/byteArrayColumn/go1.17-8 1899.0 ± ∞ ¹ 900.0 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/byteArrayColumn/go1.18-8 1727.0 ± ∞ ¹ 818.0 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/fixedLenByteArrayColumn/go1.17-8 1.000k ± ∞ ¹ 0.000k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/fixedLenByteArrayColumn/go1.18-8 909.0 ± ∞ ¹ 0.0 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/stringColumn/go1.17-8 1.999k ± ∞ ¹ 1.000k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/stringColumn/go1.18-8 1818.0 ± ∞ ¹ 909.0 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/indexedStringColumn/go1.17-8 1.999k ± ∞ ¹ 1.000k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/indexedStringColumn/go1.18-8 1818.0 ± ∞ ¹ 909.0 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/uuidColumn/go1.17-8 1.000k ± ∞ ¹ 0.000k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/uuidColumn/go1.18-8 909.0 ± ∞ ¹ 0.0 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/timeColumn/go1.17-8 1.999k ± ∞ ¹ 1.000k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/timeColumn/go1.18-8 1818.0 ± ∞ ¹ 909.0 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/timeInMillisColumn/go1.17-8 1.999k ± ∞ ¹ 1.000k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/timeInMillisColumn/go1.18-8 1818.0 ± ∞ ¹ 909.0 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/mapColumn/go1.17-8 16.98k ± ∞ ¹ 15.98k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/mapColumn/go1.18-8 15.37k ± ∞ ¹ 14.46k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/decimalColumn/go1.17-8 1.000k ± ∞ ¹ 0.000k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/decimalColumn/go1.18-8 909.0 ± ∞ ¹ 0.0 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/contact/go1.17-8 2.999k ± ∞ ¹ 2.000k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/contact/go1.18-8 2.727k ± ∞ ¹ 1.818k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/paddedBooleanColumn/go1.17-8 1.000k ± ∞ ¹ 0.000k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/paddedBooleanColumn/go1.18-8 909.0 ± ∞ ¹ 0.0 ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/optionalInt32Column/go1.17-8 1.000k ± ∞ ¹ 0.000k ± ∞ ¹ ~ (p=1.000 n=1) ² GenericReader/optionalInt32Column/go1.18-8 909.0 ± ∞ ¹ 0.0 ± ∞ ¹ ~ (p=1.000 n=1) ² geomean 1.434k ? ³ ⁴ ¹ need >= 6 samples for confidence interval at level 0.95 ² need >= 4 samples to detect a difference at alpha level 0.05 ³ summaries must be >0 to compute geomean ⁴ ratios must be >0 to compute geomean ```

As far as we are concerned, when we receive the bytes we want, its none of our business if there was an error returned by the interface. The operation is deemed a success when requested bytes are served.

fix error handling when reading from io.ReaderAt

Absolute performance improvement on GenericReader

Drop go1.17

fixes segmentio#23 closes segmentio#24 Multiple layers of buffers were causing active rows memory to be overwritten resulting in data corruption. While merging rows, we buffer reads to speed up processing, however for cases where we need more data we were calling the underlying `RowGroup` multiple times resulting in active buffers being overwritten. Fixing `ReadRows` is a bit challenging. Without buffered reads it becomes a performance bottleneck. Lucky for us there is an interface `RowWriterTo` which allows us to have control over the active buffers. This commit implements `RowWriterTo` for the `MergeRowGroups` implementation. In this case we ensure that values not yet written are ot corrupted.

we already state go 1.20 in go.mod

only run tests on 1.20.x

add preview changelog for `v0.17.0`

move PR link after description

reformat changelog

fix memory corruption in MergeRowGroups

…hangelog remove preview from changelog

I'm neither employed nor paid by twilio to maintain this.

Co-authored-by: Filip Petkovski <[email protected]>

Co-authored-by: Kevin Burke <[email protected]>

Update README to state its community maintained

comunidadio mentioned this pull request Apr 3, 2023

Optional *time.Time panics #497

Open

DuanWeiFan changed the title ~~Allow *int64 to write as optional timestamp, *int32 to write as optional date.~~ Enable *int64 to write as optional timestamp, *int32 to write as optional date. Apr 18, 2023

kevinburkesegment and others added 25 commits July 12, 2023 10:07

LICENSE: replace boilerplate with Twilio

6d5ef31

.github: test with Go v1.20

db0f39a

.github: remove the self-hosted runners

9d4b1ba

These aren't available outside of the segmentio organization.

.github: remove ARM64 runners

7d13da4

ARM64 runners were being used as part of Segment's cloud infrastructure that we don't have access to now. For now, just remove the tests.

all: replace references to segmentio

8a36f17

parquet_go18_test,file_test: skip failing test

b1886e5

.github: add link to ticket

ac0d556

go.mod: update dependencies

873a7a7

Dependabot flagged an issue, and while we are at it update all of the dependencies. This will fail until we rewrite all of the references in this repo to match the new import scheme.

README.md: add Segment info

d703d04

README.md: add parquet-go logo

c5ba125

Generated by Craiyon: https://www.craiyon.com/

README.md: make the logo smaller

8c3b7a6

chore: correct use of sync.Pool

fa05253

copy testdata/issue368.parquet from segment/parquet-go

9da5d64

I noticed that checksum of testdata/issue368.parquet was different meaning we were working with the wrong file. There is no need for further investigation on why we end up with a different file.

support older go version

71cc595

use interface{} instead of any

Merge pull request segmentio#11 from gernest/fix-flacky-test-failure

9cb80a6

fix: wrong file in testdata

chore: don't skip TestIssue368

563e358

Merge pull request segmentio#13 from gernest/unskip-TestIssue368

3e09dd9

chore: don't skip TestIssue368

move test to parquet_go18_test.go

8c15cb7

Merge pull request segmentio#9 from gernest/fix-zero-valueof-pointer-…

3d3ee2a

…field fix: zero value of nested field pointer

remove 1.17.x from github workflow

603daa1

update README stating go 1.18 is required

90e2359

move reader_go18* to reader*

9baf4ca

remove value_go17

11c6d14

remove value_go18

8cab7a5

gernest and others added 28 commits July 15, 2023 22:38

remove column_buffer_go18

0254041

remove build //go:build go1.18 from non *go1.8 files

43fbff7

remove internal/unsafecast/unsafecast_go18*

aec4036

remove go1.18 build tag from rle_amd64_test

cfde29a

remove parquet_go18*

a1d19b8

Revert "remove unused array functions"

133c83f

This reverts commit 4c444d3.

fix error handling when reading from io.ReaderAt

5d62bf0

As far as we are concerned, when we receive the bytes we want, its none of our business if there was an error returned by the interface. The operation is deemed a success when requested bytes are served.

Merge pull request segmentio#18 from gernest/fix-reading-from-reader-at

4104850

fix error handling when reading from io.ReaderAt

Merge pull request segmentio#17 from gernest/optimize-generic-reader

02bcae9

Absolute performance improvement on GenericReader

Merge pull request segmentio#16 from gernest/drop-go1.7

bb12c19

Drop go1.17

add tests from issue 23

7719456

add preview changelog for v0.17.0

bf35002

only run tests on 1.20.x

0f34c63

we already state go 1.20 in go.mod

Merge pull request segmentio#33 from parquet-go/minimize-ci-time

ae2b986

only run tests on 1.20.x

Merge pull request segmentio#32 from parquet-go/v0.17.0-changelog

16ffc72

add preview changelog for `v0.17.0`

reformat changelog

34bfc33

move PR link after description

Merge pull request segmentio#34 from parquet-go/reformat-changelog

c0869a1

reformat changelog

Merge pull request segmentio#31 from parquet-go/merge-row-writer-to

63e5480

fix memory corruption in MergeRowGroups

remove preview from changelog

69c4161

Merge pull request segmentio#35 from parquet-go/remove-preview-from-c…

623ff76

…hangelog remove preview from changelog

Update README to state its community maintained

cc18ac0

I'm neither employed nor paid by twilio to maintain this.

Update README.md

c33fb94

Co-authored-by: Filip Petkovski <[email protected]>

Update README.md

4fade97

Co-authored-by: Kevin Burke <[email protected]>

Merge pull request segmentio#37 from parquet-go/update-readme

1e4f8bb

Update README to state its community maintained

schema handle pointer timestamp & date

b41156f

schema forEachTagOption redirect reflect.Ptr to it's Elem

e532780

DuanWeiFan force-pushed the main branch from 35a603d to e532780 Compare July 31, 2023 16:57

DuanWeiFan mentioned this pull request Jul 31, 2023

Enable schema support for optional *int32 as date & *int64 as timestamp. parquet-go/parquet-go#45

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable int64 to write as optional timestamp, int32 to write as optional date. #472

Enable int64 to write as optional timestamp, int32 to write as optional date. #472

DuanWeiFan commented Feb 6, 2023

DuanWeiFan commented Mar 6, 2023

DuanWeiFan commented Apr 27, 2023

achille-roussel commented Apr 27, 2023

Enable *int64 to write as optional timestamp, *int32 to write as optional date. #472

Are you sure you want to change the base?

Enable *int64 to write as optional timestamp, *int32 to write as optional date. #472

Conversation

DuanWeiFan commented Feb 6, 2023

DuanWeiFan commented Mar 6, 2023

DuanWeiFan commented Apr 27, 2023

achille-roussel commented Apr 27, 2023

Enable int64 to write as optional timestamp, int32 to write as optional date. #472

Enable int64 to write as optional timestamp, int32 to write as optional date. #472