Due to the sandboxed nature of WebAssembly, by default WASM modules
don't have any access to the underlying filesystem.
There is however a capabilities based mechanism[0] for allowing such
access.
This adds a config option to the 'wasm' application type;
'access.filesystem' which takes an array of directory paths that are
then made available to the WASM module. This access works recursively,
i.e everything under a specific path is allowed access to.
Example config might look like
"access" {
"filesystem": [
"/tmp",
"/var/tmp"
]
}
The actual mechanism used allows directories to be mapped differently in
the guest. But at the moment we don't support that and just map say /tmp
to /tmp. This can be revisited if it's something users clamour for.
Network sockets are another resource that may be controlled in this
manner, for example there is a wasi_config_preopen_socket() function,
however this requires the runtime to open the network socket then
effectively pass this through to the guest.
This is something that can be revisited in the future if users desire
it.
[0]:
<https://github.com/bytecodealliance/wasmtime/blob/main/docs/WASI-capabilities.md>
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
This exposes various WebAssembly language module specific options.
The application type is "wasm".
There is a "module" option that is required, this specifies the full
path to the WebAssembly module to be run. This module should be in
binary format, i.e a .wasm file.
There are also currently eight function handlers that can be specified.
Three of them are _required_
1) request_handler
The main driving function. This may be called multiple times for a
single HTTP request if the request is larger than the shared memory.
2) malloc_handler
Used to allocate a chunk of memory at language module startup. This
memory is allocated from the WASM modules address space and is what is
sued for communicating between the WASM module (the guest) and Unit (the
host).
3) free_handler
Used to free the memory from above at language module shutdown.
Then there are the following five _optional_ handlers
1) module_init_handler
If set, called at language module startup.
2) module_end_handler
If set, called at language module shutdown.
3) request_init_handler
If set, called at the start of request. Called only once per HTTP
request.
4) request_end_handler
If set, called once all of a request has been sent to the WASM module.
5) response_end_handler
If set, called at the end of a request, once the WASM module has sent
all its headers and data.
Example config
"applications": {
"luw-echo-request": {
"type": "wasm",
"module": "/path/to/unit-wasm/examples/c/luw-echo-request.wasm",
"request_handler": "luw_request_handler",
"malloc_handler": "luw_malloc_handler",
"free_handler": "luw_free_handler",
"module_init_handler": "luw_module_init_handler",
"module_end_handler": "luw_module_end_handler",
}
}
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
This allows to configure the Wasm module, e.g
./configure wasm --include-path=/path/to/wasmtime-v11.0.0-x86_64-linux-c-api/include --lib-path=/path/to/wasmtime-v11.0.0-x86_64-linux-c-api/lib --rpath
--rpath as above says to set the rpath to the value of --lib-path. You
can alternatively specify a directory to use as the rpath. Or simply
omit the option to not have an rpath set.
This is mostly useful for during development where you may not have the
Wasmtime stuff installed to system directories or you want to test with
newer/different versions.
See ./configure wasm --help for a full list of options.
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
This adds the core of runtime WebAssembly[0] support. Future commits
will enable this in the Unit core and expose the configuration.
This introduces a new src/wasm directory for storing this source.
We are initially using Wasmtime[0] as the WebAssembly runtime, however
this has been designed with the ability to use different runtimes in
mind.
src/wasm/nxt_wasm.[ch] is the main interface to Unit.
src/wasm/nxt_rt_wasmtime.c is the Wasmtime runtime support. This is
nicely insulated from any knowledge of internal Unit workings.
Wasmtime is what loads and runs the Wasm modules.
The Wasm modules can export functions Wasmtime can call and Wasmtime can
export functions that the module can call.
We make use of both. The terminology used is that function exports are
what the Wasm module exports and function imports are what the Wasm
runtime exports to the module.
We currently have four function imports (functions exported by the
runtime to be called by the Wasm module).
1) nxt_wasm_get_init_mem_size
This allows Wasm modules to get the size of the initially allocated
shared memory. This is the size allocated at Unit startup and what the
Wasm modules can assume they have access to (in reality this shared
memory will likely be larger).
The amount of memory allocated at startup is NXT_WASM_MEM_SIZE which as
of this commit is 32MiB.
We do actually allocate NXT_WASM_MEM_SIZE + NXT_WASM_PAGE_SIZE at
startup which is an extra 64KiB (the smallest allocation unit), this is
to allow room for the response structure and so module developers can
just assume they have the full 32MiB for their actual response.
2) nxt_wasm_send_headers
This allows WASM modules to send their headers.
3) nxt_wasm_send_response
This allows WASM modules to send their response.
4) nxt_wasm_response_end
This allows WASM modules to inform Unit they have finished sending their
response. This calls nxt_unit_request_done()
Then there are currently up to eight functions that a module can export.
Three of which are required. These function can be named anything. I'll
use the Unit configuration names to refer to them
1) request_handler
The main driving function. This may be called multiple times for a
single HTTP request if the request is larger than the shared memory.
2) malloc_handler
Used to allocate a chunk of memory at language module startup. This
memory is allocated from the WASM modules address space and is what is
sued for communicating between the WASM module (the guest) and Unit (the
host).
3) free_handler
Used to free the memory from above at language module shutdown.
Then there are the following optional handlers
1) module_init_handler
If set, called at language module startup.
2) module_end_handler
If set, called at language module shutdown.
3) request_init_handler
If set, called at the start of request. Called only once per HTTP
request.
4) request_end_handler
If set, called once all of a request has been sent to the WASM module.
5) response_end_handler
If set, called at the end of a request, once the WASM module has sent
all its headers and data.
32bits
We currently support 32bit WASM modules, I.e wasm32-wasi. Newer version
of clang, 13+[2], do seem to have support for wasm64 as a target (which
uses a LP64 model). However it's not entirely clear if the WASI SDK
fully supports[3] this and by extension WASI libc/wasi-sysroot.
64bit support is something than can be explored more thoroughly in the
future.
As such in structures that are used to communicate between the host and
guest we use 32bit ints. Even when a single byte might be enough. This
is to avoid issues with structure layout differences between a 64bit
host and 32bit guest (I.e WASM module) and the need for various bits of
structure padding depending on host architecture. Instead everything is
4-byte aligned.
[0]: <https://webassembly.org/>
[1]: <https://wasmtime.dev/>
[2]: <https://reviews.llvm.org/rG670944fb20b226fc22fa993ab521125f9adbd30a>
[3]: <https://github.com/WebAssembly/wasi-sdk/issues/185>
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
This is required to actually _build_ the Wasm language module.
The nxt_wasm_app_conf_t structure consists of the modules name, e.g
wasm, then the three required function handlers followed by the five
optional function handlers.
See the next commit for details of these function handlers.
We also need to include the u.wasm union entry that provides access to
the above structure.
The bulk of the configuration infrastructure will be added in a
subsequent commit.
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
This is the first patch in adding WebAssembly language module support.
This just adds a new NXT_APP_WASM type, required by subsequent commits.
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
This makes it much more clear what's what.
This is in preparation for adding WebAssembly language module support.
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
This commit is to reimplement the variables with an unknown field
such as $header_{name} to make the parsing more generic,
it's a preparation for supporting response header variables.
Almost equivalent to b42f6b1d ("Tools: unitc edit mode for interactive
configuration."), implemented by Liam in tools/unitc.
I chose to give preference to vi(1) over vim(1) because Debian has vi(1)
as part of update-alternatives(1), so that sysadmins can configure it to
be a symlink to their favourite vi(1) implementation or variant.
We're ignoring the errors of the commands due to having the SSH tunnel
open. I should fix the script to use traps to close the tunnel on any
error, so we don't leak tunnels. Then, we'll be able to not ignore
curl(1) or editor errors. That will also probably allow moving the
tunneling code to the ctl command, thus deduplicating code.
Cc: Liam Crilly <liam@nginx.com>
Cc: Andrew Clayton <a.clayton@nginx.com>
Signed-off-by: Alejandro Colomar <alx@nginx.com>
It restarts all running unitd instances. This is useful when
recompiling unitd often, so that the latest build is running, without
having to manually kill unitd instances and re-run with the same exact
command line every time.
Reviewed-by: Andrew Clayton <a.clayton@nginx.com>
Signed-off-by: Alejandro Colomar <alx@nginx.com>
Some ps(1) implementations trim lines to 80 columns, even if the output
is being piped. Let's force ps(1) to output full lines with `ww`, which
is not in POSIX, but seems to be portable enough.
On 2023-06-08 13:19, Andrew Clayton wrote:
> Just for posterity...
>
> BusyBox ps(1) knows about a grand total of 2 options!
>
> BusyBox v1.35.0 (2022-11-19 10:13:10 UTC) multi-call binary.
>
> Usage: ps [-o COL1,COL2=HEADER] [-T]
>
> Show list of processes
>
> -o COL1,COL2=HEADER Select columns for display
> -T Show threads
>
> But at least it doesn't make it worse. In fact all of these three do
> exactly the same thing
>
> ps
> ps ax
> ps axww
>
> I.e it ignores any non option argument...
>
> It does however help on OpenIndiana...
Link: <https://github.com/nginx/unit/issues/875>
Link: <https://github.com/nginx/unit/issues/886>
Link: <https://github.com/nginx/unit/pull/885>
Cc: <https://github.com/mattxtaz>
Cc: Liam Crilly <liam@nginx.com>
Reviewed-by: Andrew Clayton <a.clayton@nginx.com>
Signed-off-by: Alejandro Colomar <alx@nginx.com>
Class usage came from the unittest framework and it was always redundant
after migration to the pytest. This commit removes classes from files
containing tests to make them more readable and understandable.
When a variable is accessed in the Unit configuration, the value is cached.
This was useful prior to the URI rewrite feature, but now that the URI (more
precisely, the request target) can be rewritten, the contents of the variable
$uri (which contains the path part of the request target, and is decoded)
should not be cached anymore, or at least the cached value should be invalidated
after a URI rewrite.
Example:
{
"rewrite": "/prefix$uri",
"share": "$uri"
}
For a request line like GET /foo?bar=baz HTTP/1.1\r\n, the expected file
served in the response would be /prefix/foo, but due to the caching issue,
Unit currently serves /foo.
Prerequisites check moved to the module level to simplify class structure.
Discovery and prerequisites checks functions moved to the separate files.
Introduced "require" fixture to provide per-test requirements check.
Improved cross-platform support by trying multiple ps(1) invocations to
obtain the unitd command line parameters. Additional error checking
detects when this process fails.
The first attempt uses `ps -wwo args=COMMAND -p` which has very broad
support and has the additional benefit of simplifying the output for
more reliable parsing of the process info. If that fails then we fall
back to simply `ps`.
The parsing of the process info has also changed. Instead of converting
'[]' into spaces we now convert them into explicit delimiters (using '^').
This is more reliable as it marks the beginning and the end of the info
we care about. Any trailing process information is now ignored (FreeBSD).
Additional error handling improves the robustness when starting unitd with
a different filename or from a relative path. In this case the control
socket and log file detection will fail when running `unitd --help`.
Additional error checking and messages are displayed when the control socket
cannot be determined. A single warning is shown when the log file cannot be
determined.
Lifespan state is a special dict in asgi lifespan scope, which allow
applications to persist data from the lifespan cycle to request/response
handling. The scope["state"] namespace provides a place to store these
sorts of things. The server will ensure that a shallow copy of the
namespace is passed into each subsequent request/response call into the
application.
Some frameworks are already taking advantage of this feature, for
example, starlette, and without this feature they wouldn't work
properly.
Signed-off-by: synodriver <diguohuangjiajinweijun@gmail.com>
Reviewed-by: Andrew Clayton <a.clayton@nginx.com>
[ Minor code tweaks to avoid lines > 80 chars, static a function and
re-work the PyMemberDef structure initialisation for Python <3.7
and -Wwrite-strings compatibility - Andrew ]
Tested-by: <https://github.com/synodriver>
Tested-by: <https://github.com/hawiliali>
Closes: <https://github.com/nginx/unit/issues/864>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
Common methods from applications/proto.py converted to the fixtures.
sysctl check moved to the specific file where it is using.
Some options moved to the constructor to have early access.
To make fixtures accessible inside of setup methods in tests all these methods
are renamed to the "setup_method_fixture" and decorated by autouse flag.
Also all setup methods moved to the top of the files.
If we don't update the pointer before copying the request body, then we
get the behavior shown below. After this patch, "foo\n" is rightly
appended at the end of the response body.
Request:
"GET / HTTP/1.1\r\nHost: _\nContent-Length: 4\n\nfoo\n"
Response body:
"""
Hello world!
foo
est data:
Method: GET
Protocol: HTTP/1.1
Remote addr: 127.0.0.1
Local addr: 127.0.0.1
Target: /
Path: /
Fields:
Host: _
Content-Length: 4
Body:
"""
Fixes: 1bb22d1e92 ("Unit application library.")
Reviewed-by: Andrew Clayton <a.clayton@nginx.com>
Signed-off-by: Alejandro Colomar <alx@nginx.com>
In this case, Andrew always committed under the same name, so it's not
strictly necessary.
This is for consistency, so that it's clearer whose is an email address.
It also makes it easier to sort the names alphabetically.
Reviewed-by: Andrew Clayton <a.clayton@nginx.com>
Signed-off-by: Alejandro Colomar <alx@nginx.com>
We renamed the options recently, with the intention of keeping the old
names as supported but deprecated for some time, before removal. This
was done with the configure script options, but in the unitd binary, we
accidentally removed the old names, causing some unintended breakage.
Keep support for the old names, albeit with a deprecation message to
stderr, for some time, until we decide to remove them.
Fixes: 5a37171f73 ("Added default values for pathnames.")
Closes: <https://github.com/nginx/unit/issues/876>
Reported-by: El RIDO <elrido@gmx.net>
Acked-by: Liam Crilly <liam@nginx.com>
Acked-by: Artem Konev <a.konev@f5.com>
Acked-by: Timo Stark <t.stark@nginx.com>
Reviewed-by: Andrew Clayton <a.clayton@nginx.com>
Cc: Andrei Zeliankou <zelenkov@nginx.com>
Signed-off-by: Alejandro Colomar <alx@nginx.com>
