Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: use Frameworks API #2547

Draft
wants to merge 21 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
2729455
migrate blobs to frameworks api
pieh Jun 28, 2024
5a5fcb0
migrate serverless functions to frameworks api
pieh Jun 28, 2024
dda77b7
migrate edge functions to frameworks api
pieh Jul 3, 2024
173cefe
prep blobs handling to conditionally use frameworks api, but still be…
pieh Jul 3, 2024
96edce2
conditionally use frameworski api when generating serverless and edge…
pieh Jul 16, 2024
c9f0f41
memoize deployment config
pieh Jul 16, 2024
0c91ea6
try integration tests with recent build version
pieh Jul 16, 2024
869e448
integration tests handle frameworks api
pieh Jul 16, 2024
3f27a9a
test: add e2e test for pre-frameworks api
pieh Jul 16, 2024
9864c7c
don't introduce eslint disables
pieh Jul 16, 2024
1d49fff
test: add unit test for frameworks API dirs
pieh Jul 16, 2024
4817063
chore: update some code comments
pieh Jul 16, 2024
aec7ac2
fix lint
pieh Jul 16, 2024
f3b7306
add a helper for how edge function config is defined
pieh Jul 17, 2024
ab76a7d
Merge remote-tracking branch 'origin/main' into michalpiechowiak/frp-…
pieh Jul 22, 2024
7630ab4
add a helper for how serverless function config is defined
pieh Jul 23, 2024
6c7874c
Merge remote-tracking branch 'origin/main' into michalpiechowiak/frp-…
pieh Jul 23, 2024
4358a1c
Merge remote-tracking branch 'origin/main' into michalpiechowiak/frp-…
pieh Aug 21, 2024
ee47e8b
add next.deployStrategy otel attribute
pieh Aug 21, 2024
e3a806c
update new test assertion
pieh Aug 21, 2024
4424419
validate deployment setup to ensure we are testing what we think we a…
pieh Aug 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 2 additions & 5 deletions src/build/content/prerendered.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { existsSync } from 'node:fs'
import { mkdir, readFile, writeFile } from 'node:fs/promises'
import { mkdir, readFile } from 'node:fs/promises'
import { join } from 'node:path'

import { trace } from '@opentelemetry/api'
Expand All @@ -8,7 +8,6 @@ import { glob } from 'fast-glob'
import pLimit from 'p-limit'
import { satisfies } from 'semver'

import { encodeBlobKey } from '../../shared/blobkey.js'
import type {
CachedFetchValue,
NetlifyCachedAppPageValue,
Expand All @@ -31,13 +30,11 @@ const writeCacheEntry = async (
lastModified: number,
ctx: PluginContext,
): Promise<void> => {
const path = join(ctx.blobDir, await encodeBlobKey(route))
const entry = JSON.stringify({
lastModified,
value,
} satisfies NetlifyCacheHandlerValue)

await writeFile(path, entry, 'utf-8')
await ctx.setBlob(route, entry)
}

/**
Expand Down
5 changes: 2 additions & 3 deletions src/build/content/static.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@
import { existsSync } from 'node:fs'
import { cp, mkdir, readFile, rename, rm, writeFile } from 'node:fs/promises'
import { cp, mkdir, readFile, rename, rm } from 'node:fs/promises'
import { basename, join } from 'node:path'

import { trace } from '@opentelemetry/api'
import { wrapTracer } from '@opentelemetry/api/experimental'
import glob from 'fast-glob'

import { encodeBlobKey } from '../../shared/blobkey.js'
import { PluginContext } from '../plugin-context.js'
import { verifyNetlifyForms } from '../verification.js'

Expand All @@ -33,7 +32,7 @@ export const copyStaticContent = async (ctx: PluginContext): Promise<void> => {
.map(async (path): Promise<void> => {
const html = await readFile(join(srcDir, path), 'utf-8')
verifyNetlifyForms(ctx, html)
await writeFile(join(destDir, await encodeBlobKey(path)), html, 'utf-8')
await ctx.setBlob(path, html)
}),
)
} catch (error) {
Expand Down
36 changes: 24 additions & 12 deletions src/build/functions/edge.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { cp, mkdir, readFile, rm, writeFile } from 'node:fs/promises'
import { dirname, join } from 'node:path'

import type { Manifest, ManifestFunction } from '@netlify/edge-functions'
import type { IntegrationsConfig, Manifest, ManifestFunction } from '@netlify/edge-functions'
import { glob } from 'fast-glob'
import type { EdgeFunctionDefinition as NextDefinition } from 'next/dist/build/webpack/plugins/middleware-plugin.js'
import { pathToRegexp } from 'path-to-regexp'
Expand Down Expand Up @@ -53,7 +53,7 @@ const augmentMatchers = (
})
}

const writeHandlerFile = async (ctx: PluginContext, { matchers, name }: NextDefinition) => {
const writeHandlerFile = async (ctx: PluginContext, { matchers, name, page }: NextDefinition) => {
const nextConfig = ctx.buildConfig
const handlerName = getHandlerName({ name })
const handlerDirectory = join(ctx.edgeFunctionsDir, handlerName)
Expand All @@ -63,12 +63,11 @@ const writeHandlerFile = async (ctx: PluginContext, { matchers, name }: NextDefi
// Netlify Edge Functions and the Next.js edge runtime.
await copyRuntime(ctx, handlerDirectory)

const augmentedMatchers = augmentMatchers(matchers, ctx)

// Writing a file with the matchers that should trigger this function. We'll
// read this file from the function at runtime.
await writeFile(
join(handlerRuntimeDirectory, 'matchers.json'),
JSON.stringify(augmentMatchers(matchers, ctx)),
)
await writeFile(join(handlerRuntimeDirectory, 'matchers.json'), JSON.stringify(augmentedMatchers))

// The config is needed by the edge function to match and normalize URLs. To
// avoid shipping and parsing a large file at runtime, let's strip it down to
Expand All @@ -85,6 +84,17 @@ const writeHandlerFile = async (ctx: PluginContext, { matchers, name }: NextDefi
JSON.stringify(minimalNextConfig),
)

const isc = ctx.useFrameworksAPI
? `export const config = ${JSON.stringify({
name: name.endsWith('middleware')
? 'Next.js Middleware Handler'
: `Next.js Edge Handler: ${page}`,
pattern: augmentedMatchers.map((matcher) => matcher.regexp),
cache: name.endsWith('middleware') ? undefined : 'manual',
generator: `${ctx.pluginName}@${ctx.pluginVersion}`,
} satisfies IntegrationsConfig)};`
: ``

// Writing the function entry file. It wraps the middleware code with the
// compatibility layer mentioned above.
await writeFile(
Expand All @@ -93,7 +103,7 @@ const writeHandlerFile = async (ctx: PluginContext, { matchers, name }: NextDefi
import {handleMiddleware} from './edge-runtime/middleware.ts';
import handler from './server/${name}.js';
export default (req, context) => handleMiddleware(req, context, handler);
`,
${isc}`,
)
}

Expand Down Expand Up @@ -174,10 +184,12 @@ export const createEdgeHandlers = async (ctx: PluginContext) => {
]
await Promise.all(nextDefinitions.map((def) => createEdgeHandler(ctx, def)))

const netlifyDefinitions = nextDefinitions.flatMap((def) => buildHandlerDefinition(ctx, def))
const netlifyManifest: Manifest = {
version: 1,
functions: netlifyDefinitions,
if (!ctx.useFrameworksAPI) {
const netlifyDefinitions = nextDefinitions.flatMap((def) => buildHandlerDefinition(ctx, def))
const netlifyManifest: Manifest = {
version: 1,
functions: netlifyDefinitions,
}
await writeEdgeManifest(ctx, netlifyManifest)
}
await writeEdgeManifest(ctx, netlifyManifest)
}
8 changes: 6 additions & 2 deletions src/build/functions/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,9 @@ const getHandlerFile = async (ctx: PluginContext): Promise<string> => {
const templatesDir = join(ctx.pluginDir, 'dist/build/templates')

const templateVariables: Record<string, string> = {
'{{useRegionalBlobs}}': ctx.useRegionalBlobs.toString(),
'{{useRegionalBlobs}}': (ctx.blobsStrategy !== 'legacy').toString(),
'{{generator}}': `${ctx.pluginName}@${ctx.pluginVersion}`,
'{{serverHandlerRootDir}}': ctx.serverHandlerRootDir,
}
// In this case it is a monorepo and we need to use a own template for it
// as we have to change the process working directory
Expand Down Expand Up @@ -141,7 +143,9 @@ export const createServerHandler = async (ctx: PluginContext) => {
await copyNextServerCode(ctx)
await copyNextDependencies(ctx)
await copyHandlerDependencies(ctx)
await writeHandlerManifest(ctx)
if (!ctx.useFrameworksAPI) {
await writeHandlerManifest(ctx)
}
await writePackageMetadata(ctx)
await writeHandlerFile(ctx)

Expand Down
79 changes: 68 additions & 11 deletions src/build/plugin-context.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { existsSync, readFileSync } from 'node:fs'
import { readFile } from 'node:fs/promises'
import { mkdir, readFile, writeFile } from 'node:fs/promises'
import { createRequire } from 'node:module'
import { join, relative, resolve } from 'node:path'
import { dirname, join, relative, resolve } from 'node:path'
import { join as posixJoin } from 'node:path/posix'
import { fileURLToPath } from 'node:url'

Expand All @@ -15,6 +15,8 @@ import type { MiddlewareManifest } from 'next/dist/build/webpack/plugins/middlew
import type { NextConfigComplete } from 'next/dist/server/config-shared.js'
import { satisfies } from 'semver'

import { encodeBlobKey } from '../shared/blobkey.js'

const MODULE_DIR = fileURLToPath(new URL('.', import.meta.url))
const PLUGIN_DIR = join(MODULE_DIR, '../..')
const DEFAULT_PUBLISH_DIR = '.next'
Expand Down Expand Up @@ -137,36 +139,87 @@ export class PluginContext {

/**
* Absolute path of the directory that will be deployed to the blob store
* frameworks api: `.netlify/v1/blobs/deploy`
* region aware: `.netlify/deploy/v1/blobs/deploy`
* default: `.netlify/blobs/deploy`
*/
get blobDir(): string {
if (this.useRegionalBlobs) {
return this.resolveFromPackagePath('.netlify/deploy/v1/blobs/deploy')
switch (this.blobsStrategy) {
case 'frameworks-api':
return this.resolveFromPackagePath('.netlify/v1/blobs/deploy')
case 'regional':
return this.resolveFromPackagePath('.netlify/deploy/v1/blobs/deploy')
case 'legacy':
default:
return this.resolveFromPackagePath('.netlify/blobs/deploy')
}
}

return this.resolveFromPackagePath('.netlify/blobs/deploy')
async setBlob(key: string, value: string) {
switch (this.blobsStrategy) {
case 'frameworks-api': {
const path = join(this.blobDir, await encodeBlobKey(key), 'blob')
await mkdir(dirname(path), { recursive: true })
await writeFile(path, value, 'utf-8')
return
}
case 'regional':
case 'legacy':
default: {
const path = join(this.blobDir, await encodeBlobKey(key))
await writeFile(path, value, 'utf-8')
}
}
}

get buildVersion(): string {
return this.constants.NETLIFY_BUILD_VERSION || 'v0.0.0'
}

get useRegionalBlobs(): boolean {
if (!(this.featureFlags || {})['next-runtime-regional-blobs']) {
return false
// eslint-disable-next-line no-invalid-this
#useFrameworksAPI: typeof this.useFrameworksAPI | null = null
get useFrameworksAPI(): boolean {
if (this.#useFrameworksAPI === null) {
// Defining RegExp pattern in edge function inline config is only supported since 29.50.5
const REQUIRED_BUILD_VERSION = '>=29.50.5'
this.#useFrameworksAPI = satisfies(this.buildVersion, REQUIRED_BUILD_VERSION, {
includePrerelease: true,
})
}

return this.#useFrameworksAPI
}

// eslint-disable-next-line no-invalid-this
#blobsStrategy: typeof this.blobsStrategy | null = null
get blobsStrategy(): 'legacy' | 'regional' | 'frameworks-api' {
if (this.#blobsStrategy === null) {
if (this.useFrameworksAPI) {
this.#blobsStrategy = 'frameworks-api'
} else {
// Region-aware blobs are only available as of CLI v17.23.5 (i.e. Build v29.41.5)
const REQUIRED_BUILD_VERSION = '>=29.41.5'
this.#blobsStrategy = satisfies(this.buildVersion, REQUIRED_BUILD_VERSION, {
includePrerelease: true,
})
? 'regional'
: 'legacy'
}
//
}

// Region-aware blobs are only available as of CLI v17.23.5 (i.e. Build v29.41.5)
const REQUIRED_BUILD_VERSION = '>=29.41.5'
return satisfies(this.buildVersion, REQUIRED_BUILD_VERSION, { includePrerelease: true })
return this.#blobsStrategy
}

/**
* Absolute path of the directory containing the files for the serverless lambda function
* `.netlify/functions-internal`
*/
get serverFunctionsDir(): string {
if (this.useFrameworksAPI) {
return this.resolveFromPackagePath('.netlify/v1/functions')
}

return this.resolveFromPackagePath('.netlify/functions-internal')
}

Expand Down Expand Up @@ -194,6 +247,10 @@ export class PluginContext {
* `.netlify/edge-functions`
*/
get edgeFunctionsDir(): string {
if (this.useFrameworksAPI) {
return this.resolveFromPackagePath('.netlify/v1/edge-functions')
}

return this.resolveFromPackagePath('.netlify/edge-functions')
}

Expand Down
5 changes: 5 additions & 0 deletions src/build/templates/handler-monorepo.tmpl.js
Original file line number Diff line number Diff line change
Expand Up @@ -53,4 +53,9 @@ export default async function (req, context) {
export const config = {
path: '/*',
preferStatic: true,
name: 'Next.js Server Handler',
generator: '{{generator}}',
nodeBundler: 'none',
includedFiles: ['**'],
includedFilesBasePath: '{{serverHandlerRootDir}}',
}
5 changes: 5 additions & 0 deletions src/build/templates/handler.tmpl.js
Original file line number Diff line number Diff line change
Expand Up @@ -46,4 +46,9 @@ export default async function handler(req, context) {
export const config = {
path: '/*',
preferStatic: true,
name: 'Next.js Server Handler',
generator: '{{generator}}',
nodeBundler: 'none',
includedFiles: ['**'],
includedFilesBasePath: '{{serverHandlerRootDir}}',
}
62 changes: 44 additions & 18 deletions tests/utils/fixture.ts
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ export async function runPluginStep(
// EDGE_FUNCTIONS_DIST: '.netlify/edge-functions-dist/',
// CACHE_DIR: '.netlify/cache',
// IS_LOCAL: true,
// NETLIFY_BUILD_VERSION: '29.23.4',
NETLIFY_BUILD_VERSION: '29.50.5',
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes integration tests use Frameworks API code paths - because integration test setup rely on PluginContext already when looking for serverless and edge functions to bundle and invoke them, this almost doesn't need any additional changes other then adjustment to simulating file based blob uploads.

Is there value in trying to run integration tests vs multiple netlify/build version? I don't think it does given that we implement lot of integration/test setup ourselves, but doesn't hurt to ask for other opinions here.

I did add one ~kitchen-sink e2e test to run on ~older CLI implicitly testing blobs, serverless and edge functions as a way to at least have some ~smoke test to have some defense against regressions there in future, but it's pretty minimal

// INTERNAL_FUNCTIONS_SRC: '.netlify/functions-internal',
// INTERNAL_EDGE_FUNCTIONS_SRC: '.netlify/edge-functions',
},
Expand Down Expand Up @@ -298,28 +298,54 @@ export async function runPlugin(
)
}

await Promise.all([bundleEdgeFunctions(), bundleFunctions(), uploadBlobs(ctx, base.blobDir)])
await Promise.all([bundleEdgeFunctions(), bundleFunctions(), uploadBlobs(ctx, base)])

return options
}

export async function uploadBlobs(ctx: FixtureTestContext, blobsDir: string) {
const files = await glob('**/*', {
dot: true,
cwd: blobsDir,
})
export async function uploadBlobs(ctx: FixtureTestContext, pluginContext: PluginContext) {
if (pluginContext.blobsStrategy === 'frameworks-api') {
const files = await glob('**/blob', {
dot: true,
cwd: pluginContext.blobDir,
})

const keys = files.filter((file) => !basename(file).startsWith('$'))
await Promise.all(
keys.map(async (key) => {
const { dir, base } = parse(key)
const metaFile = join(blobsDir, dir, `$${base}.json`)
const metadata = await readFile(metaFile, 'utf-8')
.then((meta) => JSON.parse(meta))
.catch(() => ({}))
await ctx.blobStore.set(key, await readFile(join(blobsDir, key), 'utf-8'), { metadata })
}),
)
await Promise.all(
files.map(async (blobFilePath) => {
const { dir: key } = parse(blobFilePath)
const metaFile = join(pluginContext.blobDir, key, `blob.meta.json`)
const metadata = await readFile(metaFile, 'utf-8')
.then((meta) => JSON.parse(meta))
.catch(() => ({}))
await ctx.blobStore.set(
key,
await readFile(join(pluginContext.blobDir, blobFilePath), 'utf-8'),
{
metadata,
},
)
}),
)
} else {
const files = await glob('**/*', {
dot: true,
cwd: pluginContext.blobDir,
})

const keys = files.filter((file) => !basename(file).startsWith('$'))
await Promise.all(
keys.map(async (key) => {
const { dir, base } = parse(key)
const metaFile = join(pluginContext.blobDir, dir, `$${base}.json`)
const metadata = await readFile(metaFile, 'utf-8')
.then((meta) => JSON.parse(meta))
.catch(() => ({}))
await ctx.blobStore.set(key, await readFile(join(pluginContext.blobDir, key), 'utf-8'), {
metadata,
})
}),
)
}
}
Comment on lines +320 to 363
Copy link
Contributor Author

@pieh pieh Jul 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This entire function doesn't seem like should be implemented here (as in in next-runtime repo) as it does reimplement file-based blobs uploads (something which ideally could be imported from ~netlify/build and just used) - now it "has to" support different implementations (when moving to regional blobs it was only directory change, but frameworks api also has updated directory structure).

This is not the biggest deal because it's quite straight forward - but maybe something to keep in mind in case ~integration kind of tests would be introduced in other repos to avoid duplicating something like that


const DEFAULT_FLAGS = {
Expand Down
Loading