Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(webserver): Middleware with default middleware for cors, authc, curl-like logging #10186

Open
wants to merge 1 commit into
base: release/v3.1.x
Choose a base branch
from
Open

feat(webserver): Middleware with default middleware for cors, authc, curl-like logging #10186

wants to merge 1 commit into from
+897 −101

Conversation

mathieucarbou
Copy link
Contributor

@mathieucarbou mathieucarbou commented Aug 17, 2024 •
edited
Loading

This PR improves WebServer with the following additions:

  • Add Middleware support (aka Expressif) with some built-in middlewares like cors, authc and logging
  • Add ability to collect all incoming request headers
  • Added response code and info on server

As discussed in #10185, I tested the example thanks to a pio config.

Ideally it would be nice if @me-no-dev and @ayushsharma82 could review this PR, both were involved in these parts.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 17, 2024 •
edited
Loading

Warnings
⚠️ 
	The **target branch** for this Pull Request **must be the default branch** of the project (`master`).

	If you would like to add this feature to a different branch, please state this in the PR description and we will consider it.

👋 Hello mathieucarbou, we appreciate your contribution to this project!

Click to see more instructions ...


This automated output is generated by the PR linter DangerJS, which checks if your Pull Request meets the project's requirements and helps you fix potential issues.

DangerJS is triggered with each push event to a Pull Request and modify the contents of this comment.

Please consider the following:
- Danger mainly focuses on the PR structure and formatting and can't understand the meaning behind your code or changes.
- Danger is not a substitute for human code reviews; it's still important to request a code review from your colleagues.
- Resolve all warnings (⚠️ ) before requesting a review from human reviewers - they will appreciate it.
- To manually retry these Danger checks, please navigate to the Actions tab and re-run last Danger workflow.

Review and merge process you can expect ...


We do welcome contributions in the form of bug reports, feature requests and pull requests.

1. An internal issue has been created for the PR, we assign it to the relevant engineer.
2. They review the PR and either approve it or ask you for changes or clarifications.
3. Once the GitHub PR is approved we do the final review, collect approvals from core owners and make sure all the automated tests are passing.
- At this point we may do some adjustments to the proposed change, or extend it by adding tests or documentation.
4. If the change is approved and passes the tests it is merged into the default branch.

Generated by 🚫 dangerJS against 6b1e5d0

@mathieucarbou mathieucarbou changed the title Arduino WebServer improvements feat: Middleware support for WebServer Aug 17, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 17, 2024 •
edited
Loading

Test Results

 62 files   62 suites   16m 34s ⏱️
 24 tests  18 ✅ 0 💤 6 ❌
140 runs  134 ✅ 0 💤 6 ❌

For more details on these failures, see this check.

Results for commit 6b1e5d0.

♻️ This comment has been updated with latest results.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 17, 2024 •
edited
Loading

Memory usage test (comparing PR against master branch)

The table below shows the summary of memory usage change (decrease - increase) in bytes and percentage for each target.

MemoryFLASH [bytes]FLASH [%]RAM [bytes]RAM [%]
TargetDECINCDECINCDECINCDECINC
ESP32S30‼️ +23K0.00‼️ +2.610‼️ +3K0.00‼️ +6.93
ESP32S20‼️ +19K0.00‼️ +2.250‼️ +3K0.00‼️ +6.99
ESP32C30‼️ +23K0.00‼️ +2.420‼️ +2K0.00‼️ +6.67
ESP32C60‼️ +42K0.00‼️ +4.620‼️ +25K0.00‼️ +62.11
ESP320‼️ +23K0.00‼️ +2.430⚠️ +19600.00‼️ +3.99
Click to expand the detailed deltas report [usage change in BYTES]
TargetESP32S3ESP32S2ESP32C3ESP32C6ESP32
ExampleFLASHRAMFLASHRAMFLASHRAMFLASHRAMFLASHRAM
WebServer/examples/AdvancedWebServer‼️ +22K‼️ +3K‼️ +18K‼️ +2K‼️ +23K‼️ +2K‼️ +42K‼️ +25K‼️ +21K⚠️ +1800
WebServer/examples/FSBrowser‼️ +21K‼️ +3K‼️ +18K‼️ +3K‼️ +21K‼️ +2K‼️ +39K‼️ +24K‼️ +21K⚠️ +1816
WebServer/examples/Filters‼️ +22K‼️ +3K‼️ +18K‼️ +2K‼️ +23K‼️ +2K‼️ +42K‼️ +25K‼️ +21K⚠️ +1800
WebServer/examples/HelloServer‼️ +22K‼️ +3K‼️ +17K‼️ +2K‼️ +23K‼️ +2K‼️ +42K‼️ +25K‼️ +21K⚠️ +1800
WebServer/examples/HttpAdvancedAuth‼️ +22K‼️ +3K‼️ +18K‼️ +2K‼️ +22K‼️ +2K‼️ +42K‼️ +25K‼️ +21K⚠️ +1792
WebServer/examples/HttpAuthCallback‼️ +22K‼️ +3K‼️ +17K‼️ +2K‼️ +22K‼️ +2K‼️ +42K‼️ +25K‼️ +21K⚠️ +1792
WebServer/examples/HttpAuthCallbackInline‼️ +22K‼️ +3K‼️ +18K‼️ +2K‼️ +22K‼️ +2K‼️ +42K‼️ +25K‼️ +21K⚠️ +1792
WebServer/examples/HttpBasicAuth‼️ +22K‼️ +3K‼️ +18K‼️ +2K‼️ +22K‼️ +2K‼️ +42K‼️ +25K‼️ +21K⚠️ +1792
WebServer/examples/HttpBasicAuthSHA1‼️ +22K‼️ +3K‼️ +18K‼️ +2K‼️ +22K‼️ +2K‼️ +42K‼️ +25K‼️ +22K⚠️ +1792
WebServer/examples/HttpBasicAuthSHA1orBearerToken‼️ +22K‼️ +3K‼️ +18K‼️ +2K‼️ +22K‼️ +2K‼️ +41K‼️ +25K‼️ +22K⚠️ +1792
WebServer/examples/Middleware----------
WebServer/examples/MultiHomedServers‼️ +22K‼️ +3K‼️ +18K‼️ +2K‼️ +23K‼️ +2K‼️ +42K‼️ +25K‼️ +21K⚠️ +1792
WebServer/examples/PathArgServer‼️ +19K‼️ +3K‼️ +15K‼️ +3K‼️ +14K‼️ +2048‼️ +35K‼️ +24K‼️ +18K⚠️ +1960
WebServer/examples/SDWebServer‼️ +20K‼️ +3K‼️ +15K‼️ +2K‼️ +22K‼️ +2K‼️ +41K‼️ +24K‼️ +19K⚠️ +1776
WebServer/examples/SimpleAuthentification‼️ +23K‼️ +3K‼️ +19K‼️ +3K‼️ +22K⚠️ +2040‼️ +42K‼️ +24K‼️ +23K⚠️ +1808
WebServer/examples/UploadHugeFile‼️ +18K‼️ +3K‼️ +13K‼️ +2K‼️ +14K‼️ +2K‼️ +35K‼️ +25K‼️ +18K⚠️ +1960
WebServer/examples/WebServer‼️ +22K‼️ +3K‼️ +18K‼️ +3K‼️ +22K‼️ +2048‼️ +41K‼️ +24K‼️ +21K⚠️ +1784
WebServer/examples/WebUpdate‼️ +22K‼️ +3K‼️ +18K‼️ +2K‼️ +23K‼️ +2K‼️ +42K‼️ +25K‼️ +21K⚠️ +1792

@Jason2866
Copy link
Collaborator

@mathieucarbou Do you see the possibility to make this enhancement optional?
The size increase is not just a few bytes.

@mathieucarbou
Copy link
Contributor Author

mathieucarbou commented Aug 17, 2024 •
edited
Loading

@mathieucarbou Do you see the possibility to make this enhancement optional? The size increase is not just a few bytes.

I saw that and I was surprised by it... I don't feel to have written 2k of code...

@mathieucarbou mathieucarbou reopened this Aug 17, 2024
@mathieucarbou mathieucarbou force-pushed the middleware branch 2 times, most recently from bc2dc4b to 6607a91 Compare August 17, 2024 15:04
@VojtechBartoska VojtechBartoska added the Status: Review needed Issue or PR is awaiting review label Aug 21, 2024
@VojtechBartoska VojtechBartoska added the Area: Libraries Issue is related to Library support. label Aug 21, 2024
@mathieucarbou
Copy link
Contributor Author

Adding a ref to: #10221 (comment)

@mathieucarbou mathieucarbou mentioned this pull request Aug 22, 2024
Merged
@mathieucarbou mathieucarbou changed the title feat: Middleware support for WebServer feat(webserver) Middleware support for WebServer with default middlewares for cors, authc, curl-like logging Sep 6, 2024
@mathieucarbou mathieucarbou changed the title feat(webserver) Middleware support for WebServer with default middlewares for cors, authc, curl-like logging feat(webserver) Middleware with default middleware for cors, authc, curl-like logging Sep 6, 2024
@VojtechBartoska VojtechBartoska changed the title feat(webserver) Middleware with default middleware for cors, authc, curl-like logging feat(webserver): Middleware with default middleware for cors, authc, curl-like logging Sep 16, 2024
@mathieucarbou
Copy link
Contributor Author

@safocl : I would be glad to revisit this PR but only if there is a real motivation coming from the Espressif team to merge it.

Otherwise I won't spend my time updating a PR for nothing.

I have already implemented middleware support in the fork of https://github.com/mathieucarbou/ESPAsyncWebServer we maintain (https://github.com/mathieucarbou/ESPAsyncWebServer) and also in v2 branch of PsychicHttp (https://github.com/hoeken/PsychicHttp) which is the future version, with a lot of supported middlewares for CORS, Authc, authz, curl like logging, etc.

@me-no-dev
Copy link
Member

I would be glad to revisit this PR but only if there is a real motivation coming from the Espressif team to merge it.

There is interest. we are trying to finish up some big tasks and have a good look at this PR

@mathieucarbou mathieucarbou force-pushed the middleware branch 2 times, most recently from 521bbaa to 93c630f Compare September 25, 2024 12:38
@mathieucarbou
Copy link
Contributor Author

mathieucarbou commented Sep 25, 2024 •
edited
Loading

@safocl : thank your for your review

@me-no-dev : thank you for your confirmation

I have reworked the PR to clean it - as it was it was more a PoC waiting for some interest. So I correctly did it and cleaned up the things saw by @safocl and much more.

I saw several things that I really don't like in WebServer:

  1. Returning String instead of const String&: I saw a bunch of placed where a String copy would be done in the return instead of returning a ref. Sadly, fixing that would break backward compatibility

  2. Use of chained pointers: this just makes the code error prone and harder to read with all these boilerplate loops. I would rather use std::list instead, but since I didn't see any usage of that (and only a few about vector), I don't know if there's a reason why the Arduino core code would not use it ? Especially that in the case of WebServer, the number of elements in such list would not be high enough to trigger any downsides of using them.

  3. Last point is more about organisation: the file structure inside src folder is quite fuzzy I find... Also WebServer.h would need a bit of re-ordering / api doc : there are so many public methods, and a mix of them for request and response.

@mathieucarbou mathieucarbou force-pushed the middleware branch 6 times, most recently from 7f62ca4 to cf0f79a Compare September 25, 2024 13:08
@mathieucarbou mathieucarbou force-pushed the middleware branch 2 times, most recently from f8d4eea to 4668e78 Compare September 28, 2024 09:36
lucasssvaz
lucasssvaz approved these changes Oct 8, 2024
View reviewed changes
Copy link
Collaborator

@lucasssvaz lucasssvaz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Tested and it works as expected. Just a couple nitpicks.

libraries/WebServer/examples/Middleware/ci.json Outdated Show resolved Hide resolved
libraries/WebServer/examples/Middleware/Middleware.ino Show resolved Hide resolved
@mathieucarbou
Copy link
Contributor Author

@lucasssvaz : thanks! I will update the PR soon.

@lucasssvaz
Copy link
Collaborator

lucasssvaz commented Oct 9, 2024 •
edited
Loading

@mathieucarbou Also, as there are some changes to the WebServer. Maybe it would be better to target branch release/v3.1.x to make sure we don't introduce any unwanted breaking change by mistake. Probably you just need to rebase this PR and the commits on top of it.

@mathieucarbou
Copy link
Contributor Author

@mathieucarbou Also, as there are some changes to the WebServer. Maybe it would be better to target branch release/v3.1.x to make sure we don't introduce any unwanted breaking change by mistake. Probably you just need to rebase this PR and the commits on top of it.

Yes that would be better.

What do you think about the 3 bullet points above ? Could I also switch to std::list ?

@mathieucarbou mathieucarbou changed the base branch from master to release/v3.1.x October 9, 2024 12:35
@lucasssvaz
Copy link
Collaborator

lucasssvaz commented Oct 9, 2024 •
edited
Loading

@mathieucarbou Also, as there are some changes to the WebServer. Maybe it would be better to target branch release/v3.1.x to make sure we don't introduce any unwanted breaking change by mistake. Probably you just need to rebase this PR and the commits on top of it.

Yes that would be better.

What do you think about the 3 bullet points above ? Could I also switch to std::list ?

I think it should be fine now that we are targeting v3.1.x as long as the flash usage increase is not very significant. There are too many changes from IDF recently that increased flash usage a lot and we are trying to optimize things for space.

What do you think @me-no-dev ?

@VojtechBartoska VojtechBartoska added this to the 3.1.1 milestone Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lucasssvaz lucasssvaz lucasssvaz approved these changes

@me-no-dev me-no-dev Awaiting requested review from me-no-dev

@safocl safocl Awaiting requested review from safocl

Assignees
No one assigned
Labels
Area: Libraries Issue is related to Library support. Status: Review needed Issue or PR is awaiting review
Projects
None yet
Milestone
3.1.1
Development

Successfully merging this pull request may close these issues.
6 participants
@mathieucarbou @Jason2866 @me-no-dev @lucasssvaz @safocl @VojtechBartoska