Merge branch 'main' into kmprt17

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,57 @@
+name: Bug Report
+description: File a bug report.
+title: "[Bug]: "
+labels: ["bug"]
+assignees:
+  - 6fj
+  - qxzhou1010
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for taking the time to fill out this bug report!
+  - type: textarea
+    attributes:
+      label: Describe the bug
+      description: A clear and concise description of what the bug is.
+      placeholder: |
+        Tell us what you see!
+    validations:
+      required: true
+  - type: textarea
+    attributes:
+      label: Steps To Reproduce
+      description: Steps to reproduce the behavior.
+      placeholder: |
+        1. In this environment...
+        1. With this config...
+        1. Run '...'
+        1. See error...
+    validations:
+      required: true
+  - type: textarea
+    attributes:
+      label: Expected behavior
+      description: A clear and concise description of what you expected to happen.
+      placeholder: |
+        I expect the output to be...
+    validations:
+      required: true
+  - type: input
+    attributes:
+      label: Version
+      placeholder: eg. 0.4.0.dev240424
+    validations:
+      required: true
+  - type: input
+    attributes:
+      label: Operating system
+      placeholder: eg. CentOS 7 x64
+    validations:
+      required: true
+  - type: input
+    attributes:
+      label: Hardware Resources
+      placeholder: eg. 8c16g
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,44 @@
+name: Feature request
+description: Suggest an idea for this project
+title: "[Feature]: "
+labels: ["enhancement"]
+assignees:
+  - 6fj
+  - qxzhou1010
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for taking the time to fill out this fature request!
+  - type: textarea
+    attributes:
+      label: Related problem
+      description: Is your feature request related to a problem? Please describe.
+      placeholder: |
+        A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+    validations:
+      required: true
+  - type: textarea
+    attributes:
+      label: Solution
+      description: Describe the solution you'd like.
+      placeholder: |
+        A clear and concise description of what you want to happen.
+    validations:
+      required: true
+  - type: textarea
+    attributes:
+      label: Alternatives
+      description: Describe alternatives you've considered.
+      placeholder: |
+        A clear and concise description of any alternative solutions or features you've considered.
+    validations:
+      required: true
+  - type: textarea
+    attributes:
+      label: Additional context
+      description: Additional context you would like to provide.
+      placeholder: |
+        Add any other context or screenshots about the feature request here.
+    validations:
+      required: true

README.md

@@ -119,13 +119,20 @@ sender.config:
 In the first terminal, run the following command
 
 ```bash
-docker run -it  --rm  --network host --mount type=bind,source=/tmp/receiver,target=/root/receiver --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --cap-add=NET_ADMIN --privileged=true secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/psi-anolis8:latest bash -c "./main --config receiver/receiver.config"
+docker run -it  --rm  --network host --mount type=bind,source=/tmp/receiver,target=/root/receiver --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --cap-add=NET_ADMIN --privileged=true secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/psi-anolis8:latest --config receiver/receiver.config
 ```
 
 In the other terminal, run the following command simultaneously.
 
 ```bash
-docker run -it  --rm  --network host --mount type=bind,source=/tmp/sender,target=/root/sender  --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --cap-add=NET_ADMIN --privileged=true secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/psi-anolis8:latest bash -c "./main --config sender/sender.config"
+docker run -it  --rm  --network host --mount type=bind,source=/tmp/sender,target=/root/sender  --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --cap-add=NET_ADMIN --privileged=true secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/psi-anolis8:latest --config sender/sender.config
+```
+
+You could also pass a minified JSON config directly. A minified JSON is a compact one without white space and line breaks.
+
+e.g.
+```
+docker run -it  --rm  --network host --mount type=bind,source=/tmp/sender,target=/root/sender  --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --cap-add=NET_ADMIN --privileged=true secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/psi-anolis8:latest --json '{"psi_config":{"protocol_config":{"protocol":"PROTOCOL_KKRT","role":"ROLE_RECEIVER","broadcast_result":true},"input_config":{"type":"IO_TYPE_FILE_CSV","path":"/root/receiver/receiver_input.csv"},"output_config":{"type":"IO_TYPE_FILE_CSV","path":"/root/receiver/receiver_output.csv"},"keys":["id0","id1"],"debug_options":{"trace_path":"/root/receiver/receiver.trace"}},"self_link_party":"receiver","link_config":{"parties":[{"id":"receiver","host":"127.0.0.1:5300"},{"id":"sender","host":"127.0.0.1:5400"}]}}'
 ```
 
 ## Building SecretFlow PSI Library

RELEASE.md

@@ -5,6 +5,18 @@
 > - `[API]` prefix for API changes.
 > - `[Improvement]` prefix for implementation improvement.
 
+## v0.4.0.dev240521
+- [API] remove BC22 protocol
+
+## v0.4.0.dev240517
+- [Improvement] upgrade yacl to 0.4.5b0.
+
+## v0.4.0.dev240514
+- [API] add entrypoint for docker file.
+- [API] allow passing config JSON directly to main.
+- [Bugfix] fix ic mode.
+- [Bugfix] fix RR22, SealPIR and APSI.
+
 ## v0.4.0.dev240401
 - [Improvement] upgrade download uri of xz.
 

bazel/patches/boost.patch

@@ -39,4 +39,4 @@ index 8277dbb..afc9569 100644
 +        strip_prefix = "xz-5.4.6",
      )
 
-     maybe(
+     maybe(

bazel/patches/emp-tool.patch

@@ -161,3 +161,16 @@ index 23bbf42..5101d7e 100644
  		}
  	}
 
+diff --git a/emp-tool/utils/block.h b/emp-tool/utils/block.h
+index f7d3d34..3c25a73 100644
+--- a/emp-tool/utils/block.h
++++ b/emp-tool/utils/block.h
+@@ -19,6 +19,7 @@ inline __m128i _mm_aesdeclast_si128 (__m128i a, __m128i RoundKey)
+ #include <cstring>
+ #include <iostream>
+ #include <iomanip>
++#include <cstdint>
+
+ namespace emp {
+
+

bazel/repositories.bzl

@@ -51,10 +51,10 @@ def _yacl():
         http_archive,
         name = "yacl",
         urls = [
-            "https://github.com/secretflow/yacl/archive/refs/tags/0.4.4b3.tar.gz",
+            "https://github.com/secretflow/yacl/archive/refs/tags/0.4.5b1.tar.gz",
         ],
-        strip_prefix = "yacl-0.4.4b3",
-        sha256 = "c6b5f32e92d2e31c1c5d7176792965fcf332d1ae892ab8b049d2e66f6f47e4f2",
+        strip_prefix = "yacl-0.4.5b1",
+        sha256 = "28064053b9add0db8e1e8e648421a0579f1d3e7ee8a4bbd7bd5959cb59598088",
     )
 
 def _bazel_platform():

docker/Dockerfile

@@ -46,4 +46,4 @@ LABEL kuscia.secretflow.deploy-templates=$deploy_templates
 # run as root for now
 WORKDIR /root
 
-CMD ["/bin/bash"]
+ENTRYPOINT ["./main"]

docker/README.md

@@ -7,7 +7,7 @@ docker run -it  --rm   --mount type=bind,source="$(pwd)/../../psi",target=/home/
 # build psi dev docker
 
 ```bash
-sh build.sh -v <version> -u -l
+bash build.sh -v <version> -u -l
 ```
 - *-u* means upload docker to reg.
 - *-l* means tag docker as *latest* as well.

docs/development/psi_protocol_intro.rst

@@ -6,7 +6,6 @@ SecretFlow SPU implements the following PSI protocols,
 - Semi-honest ECDH-based two-party PSI protocol [HFH99]_
 - Semi-honest ECDH-based three-party PSI protocol
 - Semi-honest OT-based two-party PSI protocol [KKRT16]_
-- Semi-honest OT-based two-party PSI protocol (with improved communication efficiency) [BC22]_
 - Differentially Private (DP) PSI Protocol [DP-PSI]_
 - Unbalanced PSI Protocol
 - Semi-honest and Malicious VOLE-based two-party PSI protocol [RS21]_ [RR22]_
@@ -120,42 +119,6 @@ We use 3-way stash-less CuckooHash proposed in [PSZ18]_.
 6. Receiver compares two BaRK-OPRFs set and obtains the intersection.
 
 
-BC22 PCG-PSI
-------------
-
-Pseudorandom Correlation Generator (PCG), is a primitive introduced in the work of Boyle et
-al. [BCG+19b]_, [BCGI18]_, [SGRR19]_, [BCG+19a]_, [CIK+20]_. The goal of PCG is to compress long sources
-of correlated randomness without violating security.
-
-Boyle et al. have designed multiple concretely efficient PCGs
-for specific correlations, such as vector oblivious linear evaluation (VOLE) or batch oblivious linear
-evaluation (BOLE). These primitives are at the heart of modern secure computation protocols with low
-communication overhead.The VOLE functionality allows a receiver to learn a secret linear combination
-of two vectors held by a sender and constructed (with sublinear communication) under variants
-of the syndrome decoding assumption.
-
-[BC22]_ uses PCG to speed up private set intersection protocols, minimizing computation and communication.
-We implement semi-honest version psi in [BC22]_ and use PCG/VOLE from [WYKW21]_ . [BC22]_ PSI protocol
-requires only 30 seconds for the case of larger sets ( :math:`2^{24}` items each) of long strings (128 bits),
-and reduces 1/3 communication than [KKRT16]_.
-
-.. figure:: ../_static/pcg_psi.png
-
-1. Sender and Receiver agree on :math:`(3,2)`-Generalized CuckooHash :math:`h_1,h_2: {\{0,1\}}^{*} \rightarrow [m]`
-
-2. Receiver inserts each x into bin :math:`h_1(x)` or :math:`h_2(x)`
-
-3. Sender inserts each y into bin :math:`h_1(y)` and :math:`h_2(y)`
-
-4. Run PCG/VOLE from [WYKW21]_, :math:`w_i = \Delta * u_i + v_i`,  Receiver gets :math:`w_i` and :math:`\Delta`,
-   Sender gets :math:`u_i` and :math:`v_i`, for each :math:`bin_i`
-
-5. Receiver sends Masked Bin Polynomial Coefficients to Sender, and receives BaRK-OPRF values
-
-6. Sender sends all BaRK-OPRF values for each :math:`{\{y_i\}}_{i=1}^{n_2}` to Receiver
-
-7. Receiver compares two BaRK-OPRFs sets and gets intersection.
-
 Differentially Private PSI
 --------------------------
 
@@ -240,7 +203,7 @@ An Oblivious Pseudorandom Function (OPRF) is a two-party protocol between client
 output of a Pseudorandom Function (PRF). [draft-irtf-cfrg-voprf-10]_ specifies OPRF, VOPRF, and POPRF protocols
 built upon prime-order groups.
 
-.. figure:: ../_static/ecdh_oprf_psi.png
+.. figure:: ../_static/ecdh_oprf_psi.jpg
 
 - Offline Phase
 
@@ -409,8 +372,6 @@ Reference
    Efficient two-round OT extension and silent non-interactive secure computation. In ACM CCS 2019,
    pages 291–308. ACM Press, November 2019.
 
-.. [BC22] Private Set Intersection from Pseudorandom Correlation Generators
-
 .. [Ber06] Daniel J. Bernstein. Curve25519: new diffie-hellman speed records. In In Public
    Key Cryptography (PKC), Springer-Verlag LNCS 3958, page 2006, 2006. (Cited on page 4.)
 

docs/reference/psi_config.md

@@ -162,7 +162,6 @@ The algorithm type of psi.
 | INVALID_PSI_TYPE | 0 | none |
 | ECDH_PSI_2PC | 1 | DDH based PSI |
 | KKRT_PSI_2PC | 2 | Efficient Batched Oblivious PRF with Applications to Private Set Intersection https://eprint.iacr.org/2016/799.pdf |
-| BC22_PSI_2PC | 3 | PSI from Pseudorandom Correlation Generators https://eprint.iacr.org/2022/334 |
 | ECDH_PSI_3PC | 4 | Multi-party PSI based on ECDH (Say A, B, C (receiver)) notice: two-party intersection cardinarlity leak (|A intersect B|) |
 | ECDH_PSI_NPC | 5 | Iterative running 2-party ecdh psi to get n-party PSI. Notice: two-party intersection leak |
 | KKRT_PSI_NPC | 6 | Iterative running 2-party kkrt psi to get n-party PSI. Notice: two-party intersection leak |

docs/user_guide/pir.rst

@@ -198,7 +198,7 @@ Setup Phase
 
 .. code-block:: bash
 
-  docker run -it  --rm  --network host --mount type=bind,source=/tmp/server,target=/root/server --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --cap-add=NET_ADMIN --privileged=true secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/psi-anolis8:0.1.0beta bash -c "./main --config server/apsi_server_setup.json"
+  docker run -it  --rm  --network host --mount type=bind,source=/tmp/server,target=/root/server --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --cap-add=NET_ADMIN --privileged=true secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/psi-anolis8:0.1.0beta --config server/apsi_server_setup.json
 
 Online Phase
 >>>>>>>>>>>>
@@ -209,14 +209,14 @@ In the server's terminal.
 
 .. code-block:: bash
 
-  docker run -it  --rm  --network host --mount type=bind,source=/tmp/server,target=/root/server --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --cap-add=NET_ADMIN --privileged=true secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/psi-anolis8:0.1.0beta bash -c "./main --config server/apsi_server_online.json"
+  docker run -it  --rm  --network host --mount type=bind,source=/tmp/server,target=/root/server --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --cap-add=NET_ADMIN --privileged=true secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/psi-anolis8:0.1.0beta --config server/apsi_server_online.json
 
 
 In the client's terminal.
 
 .. code-block:: bash
 
-  docker run -it  --rm  --network host --mount type=bind,source=/tmp/client,target=/root/client --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --cap-add=NET_ADMIN --privileged=true secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/psi-anolis8:0.1.0beta bash -c "./main --config client/apsi_client.json"
+  docker run -it  --rm  --network host --mount type=bind,source=/tmp/client,target=/root/client --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --cap-add=NET_ADMIN --privileged=true secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/psi-anolis8:0.1.0beta --config client/apsi_client.json
 
 
 More examples

docs/user_guide/psi.rst

@@ -69,12 +69,12 @@ Run PSI
 
 In the first terminal, run the following command::
 
-    docker run -it  --rm  --network host --mount type=bind,source=/tmp/receiver,target=/root/receiver  --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --cap-add=NET_ADMIN --privileged=true secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/psi-anolis8:latest bash -c "./main --config receiver/receiver.config"
+    docker run -it  --rm  --network host --mount type=bind,source=/tmp/receiver,target=/root/receiver  --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --cap-add=NET_ADMIN --privileged=true secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/psi-anolis8:latest --config receiver/receiver.config
 
 
 In the other terminal, run the following command simultaneously::
 
-    docker run -it  --rm  --network host --mount type=bind,source=/tmp/sender,target=/root/sender  --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --cap-add=NET_ADMIN --privileged=true secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/psi-anolis8:latest bash -c "./main --config sender/sender.config"
+    docker run -it  --rm  --network host --mount type=bind,source=/tmp/sender,target=/root/sender  --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --cap-add=NET_ADMIN --privileged=true secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/psi-anolis8:latest --config sender/sender.config
 
 
 Building from source
@@ -157,26 +157,10 @@ Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz
 |           | online  | 25.434s | 100.68s | 415.94s | 1672.21s |
 +-----------+---------+---------+---------+---------+----------+
 
-bc22 pcg-psi Benchmark
->>>>>>>>>>>>>>>>>>>>>>
-
-Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz
-
-+-----------+---------+---------+---------+----------+---------+---------+
-| bandwidth |   2^18  |   2^20  |   2^21  |   2^22   |   2^23  |   2^24  |
-+===========+=========+=========+=========+==========+=========+=========+
-|    LAN    | 1.261s  | 2.191s  | 3.503s  | 6.51s    | 13.012s | 26.71s  |
-+-----------+---------+---------+---------+----------+---------+---------+
-|  100Mbps  | 2.417s  | 6.054s  | 11.314s | 21.864s  | 43.778s | 88.29s  |
-+-----------+---------+---------+---------+----------+---------+---------+
-|   10Mbps  | 18.826s | 50.038s | 96.516s | 186.097s | 369.84s | 737.71s |
-+-----------+---------+---------+---------+----------+---------+---------+
-
 
 Security Tips
 -------------
 
-Warning:  `KKRT16 <https://eprint.iacr.org/2016/799.pdf>`_ and
-`BC22 PCG <https://eprint.iacr.org/2022/334.pdf>`_ are semi-honest PSI protocols,
+Warning:  `KKRT16 <https://eprint.iacr.org/2016/799.pdf>`_ is semi-honest PSI protocols,
 and may be attacked in malicious model.
-We recommend using KKRT16 and BC22_PCG PSI protocol as one-way PSI, i.e., one party gets the final intersection result.
+We recommend using KKRT16 PSI protocol as one-way PSI, i.e., one party gets the final intersection result.