Sse validation

[Nick-Nuon]

Here is the SSE Validation
routine.

For reference , here are the benchmarks on my machine:

Method	FileName	Mean	Error	StdDev	Speed (GB/s)
SIMDUtf8ValidationRealDataAvx2	data/Arabic-Lipsum.utf8.txt	5,096.3 ns	137.51 ns	7.54 ns	16.03
SIMDUtf8ValidationRealDataAvx2	data/Chinese-Lipsum.utf8.txt	4,593.7 ns	604.91 ns	33.16 ns	15.20
SIMDUtf8ValidationRealDataAvx2	data/Emoji-Lipsum.utf8.txt	4,137.7 ns	975.40 ns	53.47 ns	15.84
SIMDUtf8ValidationRealDataAvx2	data/Hebrew-Lipsum.utf8.txt	4,197.4 ns	1,212.04 ns	66.44 ns	15.84
SIMDUtf8ValidationRealDataAvx2	data/Hindi-Lipsum.utf8.txt	5,451.5 ns	702.00 ns	38.48 ns	16.14
SIMDUtf8ValidationRealDataAvx2	data/Japanese-Lipsum.utf8.txt	4,235.2 ns	1,945.06 ns	106.62 ns	16.01
SIMDUtf8ValidationRealDataAvx2	data/Korean-Lipsum.utf8.txt	4,216.3 ns	973.44 ns	53.36 ns	15.80
SIMDUtf8ValidationRealDataAvx2	data/Latin-Lipsum.utf8.txt	854.8 ns	61.09 ns	3.35 ns	101.70
SIMDUtf8ValidationRealDataAvx2	data/Russian-Lipsum.utf8.txt	6,516.1 ns	490.42 ns	26.88 ns	16.08
SIMDUtf8ValidationRealDataAvx2	data/arabic.utf8.txt	38,452.2 ns	3,060.36 ns	167.75 ns	13.88
SIMDUtf8ValidationRealDataAvx2	data/chinese.utf8.txt	9,299.1 ns	4,546.71 ns	249.22 ns	19.50
SIMDUtf8ValidationRealDataAvx2	data/czech.utf8.txt	7,412.6 ns	703.46 ns	38.56 ns	20.60
SIMDUtf8ValidationRealDataAvx2	data/english.utf8.txt	12,747.7 ns	1,535.81 ns	84.18 ns	30.62
SIMDUtf8ValidationRealDataAvx2	data/esperanto.utf8.txt	3,316.2 ns	1,893.26 ns	103.78 ns	26.22
SIMDUtf8ValidationRealDataAvx2	data/french.utf8.txt	33,861.0 ns	123,894.59 ns	6,791.08 ns	13.20
SIMDUtf8ValidationRealDataAvx2	data/german.utf8.txt	7,529.1 ns	1,205.32 ns	66.07 ns	27.33
SIMDUtf8ValidationRealDataAvx2	data/greek.utf8.txt	8,793.7 ns	471.61 ns	25.85 ns	20.62
SIMDUtf8ValidationRealDataAvx2	data/hebrew.utf8.txt	10,437.9 ns	6,899.80 ns	378.20 ns	18.21
SIMDUtf8ValidationRealDataAvx2	data/hindi.utf8.txt	24,947.7 ns	8,144.26 ns	446.41 ns	15.90
SIMDUtf8ValidationRealDataAvx2	data/japanese.utf8.txt	8,894.1 ns	2,398.24 ns	131.46 ns	18.48
SIMDUtf8ValidationRealDataAvx2	data/korean.utf8.txt	5,487.9 ns	2,901.44 ns	159.04 ns	17.83
SIMDUtf8ValidationRealDataAvx2	data/persan.utf8.txt	8,127.1 ns	1,715.07 ns	94.01 ns	19.22
SIMDUtf8ValidationRealDataAvx2	data/portuguese.utf8.txt	12,465.3 ns	6,391.80 ns	350.36 ns	22.52
SIMDUtf8ValidationRealDataAvx2	data/russian.utf8.txt	27,244.0 ns	24,884.59 ns	1,364.01 ns	14.94
SIMDUtf8ValidationRealDataAvx2	data/thai.utf8.txt	40,098.6 ns	14,855.59 ns	814.29 ns	14.80
SIMDUtf8ValidationRealDataAvx2	data/turkish.utf8.txt	9,368.9 ns	4,579.15 ns	251.00 ns	20.82
SIMDUtf8ValidationRealDataAvx2	data/vietnamese.utf8.txt	16,556.3 ns	1,273.36 ns	69.80 ns	19.27
DotnetRuntimeUtf8ValidationRealData	data/Arabic-Lipsum.utf8.txt	27,644.3 ns	14,009.84 ns	767.93 ns	2.95
DotnetRuntimeUtf8ValidationRealData	data/Chinese-Lipsum.utf8.txt	11,385.5 ns	4,033.61 ns	221.10 ns	6.13
DotnetRuntimeUtf8ValidationRealData	data/Emoji-Lipsum.utf8.txt	42,464.2 ns	4,367.18 ns	239.38 ns	1.54
DotnetRuntimeUtf8ValidationRealData	data/Hebrew-Lipsum.utf8.txt	22,648.9 ns	4,190.20 ns	229.68 ns	2.94
DotnetRuntimeUtf8ValidationRealData	data/Hindi-Lipsum.utf8.txt	23,489.4 ns	1,081.84 ns	59.30 ns	3.75
DotnetRuntimeUtf8ValidationRealData	data/Japanese-Lipsum.utf8.txt	11,693.7 ns	821.81 ns	45.05 ns	5.80
DotnetRuntimeUtf8ValidationRealData	data/Korean-Lipsum.utf8.txt	27,519.2 ns	2,653.69 ns	145.46 ns	2.42
DotnetRuntimeUtf8ValidationRealData	data/Latin-Lipsum.utf8.txt	1,247.3 ns	108.66 ns	5.96 ns	69.70
DotnetRuntimeUtf8ValidationRealData	data/Russian-Lipsum.utf8.txt	35,343.5 ns	3,513.31 ns	192.58 ns	2.96
DotnetRuntimeUtf8ValidationRealData	data/arabic.utf8.txt	201,605.4 ns	17,407.77 ns	954.18 ns	2.65
DotnetRuntimeUtf8ValidationRealData	data/chinese.utf8.txt	32,318.4 ns	23,394.31 ns	1,282.32 ns	5.61
DotnetRuntimeUtf8ValidationRealData	data/czech.utf8.txt	21,958.4 ns	2,657.32 ns	145.66 ns	6.96
DotnetRuntimeUtf8ValidationRealData	data/english.utf8.txt	15,884.2 ns	4,796.65 ns	262.92 ns	24.58
DotnetRuntimeUtf8ValidationRealData	data/esperanto.utf8.txt	6,381.1 ns	501.60 ns	27.49 ns	13.63
DotnetRuntimeUtf8ValidationRealData	data/french.utf8.txt	70,773.7 ns	18,595.34 ns	1,019.27 ns	6.31
DotnetRuntimeUtf8ValidationRealData	data/german.utf8.txt	14,378.6 ns	1,571.04 ns	86.11 ns	14.31
DotnetRuntimeUtf8ValidationRealData	data/greek.utf8.txt	27,925.3 ns	27,221.56 ns	1,492.11 ns	6.49
DotnetRuntimeUtf8ValidationRealData	data/hebrew.utf8.txt	73,071.6 ns	16,015.58 ns	877.87 ns	2.60
DotnetRuntimeUtf8ValidationRealData	data/hindi.utf8.txt	155,258.3 ns	49,876.95 ns	2,733.92 ns	2.55
DotnetRuntimeUtf8ValidationRealData	data/japanese.utf8.txt	28,894.3 ns	55,436.77 ns	3,038.68 ns	5.69
DotnetRuntimeUtf8ValidationRealData	data/korean.utf8.txt	22,859.5 ns	726.63 ns	39.83 ns	4.28
DotnetRuntimeUtf8ValidationRealData	data/persan.utf8.txt	29,108.4 ns	5,344.17 ns	292.93 ns	5.37
DotnetRuntimeUtf8ValidationRealData	data/portuguese.utf8.txt	22,632.9 ns	3,424.25 ns	187.69 ns	12.40
DotnetRuntimeUtf8ValidationRealData	data/russian.utf8.txt	160,882.4 ns	8,880.43 ns	486.77 ns	2.53
DotnetRuntimeUtf8ValidationRealData	data/thai.utf8.txt	124,392.2 ns	93,824.32 ns	5,142.83 ns	4.77
DotnetRuntimeUtf8ValidationRealData	data/turkish.utf8.txt	25,249.8 ns	7,958.18 ns	436.21 ns	7.73
DotnetRuntimeUtf8ValidationRealData	data/vietnamese.utf8.txt	163,197.5 ns	27,828.58 ns	1,525.38 ns	1.95
Utf8ValidationRealDataScalar	data/Arabic-Lipsum.utf8.txt	46,941.7 ns	18,721.08 ns	1,026.17 ns	1.74
Utf8ValidationRealDataScalar	data/Chinese-Lipsum.utf8.txt	39,227.8 ns	6,308.61 ns	345.80 ns	1.78
Utf8ValidationRealDataScalar	data/Emoji-Lipsum.utf8.txt	36,958.7 ns	7,953.54 ns	435.96 ns	1.77
Utf8ValidationRealDataScalar	data/Hebrew-Lipsum.utf8.txt	36,150.4 ns	8,475.39 ns	464.56 ns	1.84
Utf8ValidationRealDataScalar	data/Hindi-Lipsum.utf8.txt	66,283.2 ns	58,307.02 ns	3,196.00 ns	1.33
Utf8ValidationRealDataScalar	data/Japanese-Lipsum.utf8.txt	40,542.1 ns	5,894.26 ns	323.08 ns	1.67
Utf8ValidationRealDataScalar	data/Korean-Lipsum.utf8.txt	36,073.3 ns	27,843.95 ns	1,526.22 ns	1.85
Utf8ValidationRealDataScalar	data/Latin-Lipsum.utf8.txt	40,945.5 ns	19,935.82 ns	1,092.75 ns	2.12
Utf8ValidationRealDataScalar	data/Russian-Lipsum.utf8.txt	95,932.3 ns	12,185.80 ns	667.94 ns	1.09
Utf8ValidationRealDataScalar	data/arabic.utf8.txt	342,659.3 ns	50,429.46 ns	2,764.21 ns	1.56
Utf8ValidationRealDataScalar	data/chinese.utf8.txt	125,974.7 ns	4,668.50 ns	255.90 ns	1.44
Utf8ValidationRealDataScalar	data/czech.utf8.txt	102,313.7 ns	13,906.43 ns	762.26 ns	1.49
Utf8ValidationRealDataScalar	data/english.utf8.txt	184,610.7 ns	17,824.90 ns	977.04 ns	2.11
Utf8ValidationRealDataScalar	data/esperanto.utf8.txt	46,035.5 ns	13,354.04 ns	731.98 ns	1.89
Utf8ValidationRealDataScalar	data/french.utf8.txt	166,430.8 ns	53,408.04 ns	2,927.47 ns	2.69
Utf8ValidationRealDataScalar	data/german.utf8.txt	106,416.5 ns	10,209.27 ns	559.60 ns	1.93
Utf8ValidationRealDataScalar	data/greek.utf8.txt	135,734.6 ns	59,729.71 ns	3,273.99 ns	1.34
Utf8ValidationRealDataScalar	data/hebrew.utf8.txt	169,727.9 ns	52,055.17 ns	2,853.32 ns	1.12
Utf8ValidationRealDataScalar	data/hindi.utf8.txt	316,735.8 ns	181,489.16 ns	9,948.03 ns	1.25
Utf8ValidationRealDataScalar	data/japanese.utf8.txt	112,734.7 ns	29,780.08 ns	1,632.35 ns	1.46
Utf8ValidationRealDataScalar	data/korean.utf8.txt	78,674.7 ns	49,894.33 ns	2,734.88 ns	1.24
Utf8ValidationRealDataScalar	data/persan.utf8.txt	129,233.4 ns	89,750.68 ns	4,919.54 ns	1.21
Utf8ValidationRealDataScalar	data/portuguese.utf8.txt	159,794.9 ns	54,797.64 ns	3,003.64 ns	1.76
Utf8ValidationRealDataScalar	data/russian.utf8.txt	338,640.7 ns	28,655.81 ns	1,570.72 ns	1.20
Utf8ValidationRealDataScalar	data/thai.utf8.txt	373,161.4 ns	212,578.42 ns	11,652.14 ns	1.59
Utf8ValidationRealDataScalar	data/turkish.utf8.txt	92,388.2 ns	11,103.39 ns	608.61 ns	2.11
Utf8ValidationRealDataScalar	data/vietnamese.utf8.txt	247,809.9 ns	164,508.93 ns	9,017.29 ns	1.29
SIMDUtf8ValidationRealDataSse	data/Arabic-Lipsum.utf8.txt	10,466.6 ns	1,965.68 ns	107.75 ns	7.80
SIMDUtf8ValidationRealDataSse	data/Chinese-Lipsum.utf8.txt	8,645.4 ns	6,297.41 ns	345.18 ns	8.08
SIMDUtf8ValidationRealDataSse	data/Emoji-Lipsum.utf8.txt	7,957.2 ns	2,975.72 ns	163.11 ns	8.24
SIMDUtf8ValidationRealDataSse	data/Hebrew-Lipsum.utf8.txt	8,260.3 ns	3,144.62 ns	172.37 ns	8.05
SIMDUtf8ValidationRealDataSse	data/Hindi-Lipsum.utf8.txt	10,609.1 ns	2,513.38 ns	137.77 ns	8.29
SIMDUtf8ValidationRealDataSse	data/Japanese-Lipsum.utf8.txt	8,224.3 ns	616.58 ns	33.80 ns	8.24
SIMDUtf8ValidationRealDataSse	data/Korean-Lipsum.utf8.txt	7,823.2 ns	1,532.62 ns	84.01 ns	8.51
SIMDUtf8ValidationRealDataSse	data/Latin-Lipsum.utf8.txt	4,489.6 ns	289.04 ns	15.84 ns	19.36
SIMDUtf8ValidationRealDataSse	data/Russian-Lipsum.utf8.txt	12,495.5 ns	414.87 ns	22.74 ns	8.38
SIMDUtf8ValidationRealDataSse	data/arabic.utf8.txt	73,271.2 ns	47,313.21 ns	2,593.40 ns	7.29
SIMDUtf8ValidationRealDataSse	data/chinese.utf8.txt	17,254.5 ns	24,133.60 ns	1,322.84 ns	10.51
SIMDUtf8ValidationRealDataSse	data/czech.utf8.txt	12,254.6 ns	2,347.03 ns	128.65 ns	12.46
SIMDUtf8ValidationRealDataSse	data/english.utf8.txt	23,262.2 ns	2,751.43 ns	150.82 ns	16.78
SIMDUtf8ValidationRealDataSse	data/esperanto.utf8.txt	6,522.1 ns	516.63 ns	28.32 ns	13.33
SIMDUtf8ValidationRealDataSse	data/french.utf8.txt	80,789.4 ns	182,150.31 ns	9,984.27 ns	5.53
SIMDUtf8ValidationRealDataSse	data/german.utf8.txt	16,775.7 ns	15,780.16 ns	864.96 ns	12.27
SIMDUtf8ValidationRealDataSse	data/greek.utf8.txt	18,544.9 ns	16,458.34 ns	902.14 ns	9.78
SIMDUtf8ValidationRealDataSse	data/hebrew.utf8.txt	19,483.2 ns	2,912.47 ns	159.64 ns	9.76
SIMDUtf8ValidationRealDataSse	data/hindi.utf8.txt	48,527.2 ns	25,193.01 ns	1,380.91 ns	8.17
SIMDUtf8ValidationRealDataSse	data/japanese.utf8.txt	15,435.8 ns	17,267.97 ns	946.52 ns	10.65
SIMDUtf8ValidationRealDataSse	data/korean.utf8.txt	10,305.3 ns	31,123.07 ns	1,705.96 ns	9.50
SIMDUtf8ValidationRealDataSse	data/persan.utf8.txt	16,133.6 ns	18,280.18 ns	1,002.00 ns	9.68
SIMDUtf8ValidationRealDataSse	data/portuguese.utf8.txt	23,160.9 ns	5,125.52 ns	280.95 ns	12.12
SIMDUtf8ValidationRealDataSse	data/russian.utf8.txt	54,676.0 ns	7,207.48 ns	395.07 ns	7.45
SIMDUtf8ValidationRealDataSse	data/thai.utf8.txt	71,905.8 ns	8,161.58 ns	447.36 ns	8.26
SIMDUtf8ValidationRealDataSse	data/turkish.utf8.txt	16,562.7 ns	24,616.61 ns	1,349.32 ns	11.78
SIMDUtf8ValidationRealDataSse	data/vietnamese.utf8.txt	36,153.8 ns	111,988.30 ns	6,138.46 ns	8.82

[lemire]

Looks good to me. Merge at will. I will link the issue to it.

[Nick-Nuon]

Merging