admin: storage: updates and improve S3
The original documentation was written with a focus on local storage
and missed a few important aspects of how storage types can be mixed
together in some subsystems.
* unify the susbsystem paths to be "directory" instead of using "base
path". It is not technically a directory in S3 but looks like one
and the approximation is unlikely to cause confusion.
* use S3 instead of minio where possible to emphasize the storage type
is not MinIO specific and add a section for the compatibility with
garage
* change the document structure to have two separate parts:
* [storage] the global sections
* [xxxx] the subsystem specific section
and within each of them local and S3 are documented separately
because they share nothing
* [storage] is documented to be a fallback if [xxxx] does not exist
for a given subsystem. There is no notion of inheritance because
it behaves in ways that are not tested and for which consistency is
not guaranteed
* backward compatibility with Gitea is documented to be the reason why
there are no safeguards against undocumented features
(cherry picked from commit 852b447f91)
This commit is contained in:
github-actions[bot]
parent
654746f4a3
commit
f9fe00fdaa
1 changed files with 184 additions and 74 deletions
|
|
@ -3,11 +3,13 @@ title: 'Storage settings'
|
|||
license: 'CC-BY-SA-4.0'
|
||||
---
|
||||
|
||||
The storage for each subsystem is defined in `app.ini`. It can either be on disk
|
||||
(`local`) or using a MinIO server (`minio`). The default is `local`
|
||||
storage, using the following hierarchy under the `APP_DATA_PATH` directory:
|
||||
The storage for each subsystem is defined in `app.ini`. It can either
|
||||
be on disk (`local` which is the default) or using a S3 compatible
|
||||
server (`minio`). In both cases each subsystem stores all files (or
|
||||
objects in the S3 parlance) in a dedicated directory as shown in the
|
||||
table below:
|
||||
|
||||
| subsystem | default base path | app.ini sections |
|
||||
| subsystem | directory | app.ini sections |
|
||||
| ------------------- | ------------------ | --------------------- |
|
||||
| Attachments | attachments/ | [attachment] |
|
||||
| LFS | lfs/ | [lfs] |
|
||||
|
|
@ -18,55 +20,37 @@ storage, using the following hierarchy under the `APP_DATA_PATH` directory:
|
|||
| Actions logs | actions_log/ | [storage.actions_log] |
|
||||
| Actions Artifacts | actions_artifacts/ | [actions.artifacts] |
|
||||
|
||||
For instance if `APP_DATA_PATH` was `/appdata`, the default directory to
|
||||
store attachments will be `/appdata/attachments`.
|
||||
For instance:
|
||||
|
||||
## Overriding the defaults
|
||||
- if the `STORAGE_TYPE` is `local` and `APP_DATA_PATH` is
|
||||
`/appdata`, the default directory to store attachments is
|
||||
`/appdata/attachments`
|
||||
- if the `STORAGE_TYPE` is `minio`, the
|
||||
default directory to store attachments within the `MINIO_BUCKET`
|
||||
bucket will be `attachments/`.
|
||||
|
||||
These defaults can be modified for all subsystems in the `[storage]`
|
||||
section. For instance setting:
|
||||
## Changing the storage for all subsystems
|
||||
|
||||
The `[storage]` section can be used to modify the storage of all
|
||||
subsystems. The default is to use `local` storage under
|
||||
`APP_DATA_PATH` and is equivalent to writing the following in `app.ini`:
|
||||
|
||||
```
|
||||
[server]
|
||||
APP_DATA_PATH = /forgejo/data
|
||||
|
||||
[storage]
|
||||
PATH = /mystorage
|
||||
STORAGE_TYPE = local
|
||||
PATH = /forgejo/data
|
||||
```
|
||||
|
||||
will change the default for storing attachments to
|
||||
`/mystorage/attachments`. It is also possible to change these settings
|
||||
for each subsystem in their dedicated section. For instance:
|
||||
### Using local
|
||||
|
||||
```
|
||||
[storage]
|
||||
PATH = /mystorage
|
||||
For `local` storage, the `[storage]` section can only be used to
|
||||
change the path under which all subsystems directories will be
|
||||
created, using the `PATH` settings with an absolute pathname.
|
||||
|
||||
[attachment]
|
||||
PATH = /otherstorage/attachments
|
||||
```
|
||||
|
||||
will store attachments in `/otherstorage/attachments` while `lfs`
|
||||
files will be stored in `/mystorage/lfs`.
|
||||
|
||||
## Storage type
|
||||
|
||||
The value of `STORAGE_TYPE` can be `local` (the default) or `minio`. For instance:
|
||||
|
||||
```
|
||||
[storage]
|
||||
STORAGE_TYPE = minio
|
||||
```
|
||||
|
||||
will use `minio` for all subsystems (Attachments, LFS, etc.)
|
||||
instead of storing them on disk. Each storage type has its own
|
||||
settings, as explained below.
|
||||
|
||||
## `local` storage
|
||||
|
||||
There is just one setting when the `STORAGE_TYPE` is set to `local`,
|
||||
`PATH`. It must be an absolute path and is interpreted as follows.
|
||||
|
||||
In the `[storage]` section, `PATH` is the directory under which the default
|
||||
base path of each subsystem will be created instead of
|
||||
`APP_DATA_PATH`. For instance, if `APP_DATA_PATH` equals `/appdata`:
|
||||
For instance:
|
||||
|
||||
```
|
||||
[storage]
|
||||
|
|
@ -74,11 +58,82 @@ STORAGE_TYPE = local
|
|||
PATH = /mystorage
|
||||
```
|
||||
|
||||
Will create attachments in `/mystorage/attachments` instead of
|
||||
will change the default for storing attachments to
|
||||
`/mystorage/attachments`, LFS to `/mystorage/lfs` etc.
|
||||
|
||||
### Using minio
|
||||
|
||||
The `[storage]` section can be used to change the default storage type
|
||||
used by all subsystems to `minio`.
|
||||
|
||||
For instance:
|
||||
|
||||
```
|
||||
[storage]
|
||||
STORAGE_TYPE = minio
|
||||
|
||||
MINIO_ENDPOINT = 127.0.0.1:9000
|
||||
MINIO_ACCESS_KEY_ID = [redacted]
|
||||
MINIO_SECRET_ACCESS_KEY = [redacted]
|
||||
MINIO_BUCKET = forgejo
|
||||
MINIO_LOCATION = us-east-1
|
||||
```
|
||||
|
||||
will change the default for storing attachments to `attachments/` in
|
||||
the `forgejo` bucket, LFS to `lfs/` in the `forgejo` bucket etc.
|
||||
|
||||
> **NOTE:** `MINIO_BASE_PATH` must not be set in the `[storage]` section.
|
||||
|
||||
## Storage settings for a single subsystem
|
||||
|
||||
It is possible to configure some subsystems to use S3 storage and others to use local
|
||||
storage by adding settings to their respective sections. For instance:
|
||||
|
||||
```
|
||||
[attachment]
|
||||
PATH = /otherstorage/attachments
|
||||
|
||||
[lfs]
|
||||
STORAGE_TYPE = minio
|
||||
MINIO_BASE_PATH = lfs/
|
||||
|
||||
MINIO_ENDPOINT = 127.0.0.1:9000
|
||||
MINIO_ACCESS_KEY_ID = [redacted]
|
||||
MINIO_SECRET_ACCESS_KEY = [redacted]
|
||||
MINIO_BUCKET = forgejo
|
||||
MINIO_LOCATION = us-east-1
|
||||
```
|
||||
|
||||
will store attachments in the local directory
|
||||
`/otherstorage/attachments` while `lfs` files will be stored in the S3
|
||||
server within the `lfs/` directory of the `forgejo` bucket.
|
||||
|
||||
## Storage settings
|
||||
|
||||
The value of `STORAGE_TYPE` can be `local` (the default) for file
|
||||
system directories or `minio` for S3 servers. Each storage type has
|
||||
its own settings, as explained below.
|
||||
|
||||
### Using local
|
||||
|
||||
There is just one setting when the `STORAGE_TYPE` is set to `local`:
|
||||
`PATH`. It must be an absolute path and is interpreted as follows.
|
||||
|
||||
In the `[storage]` section, `PATH` is the path under which the
|
||||
directories of each subsystem will be created instead of
|
||||
`APP_DATA_PATH`. For instance, if `APP_DATA_PATH` equals `/appdata`
|
||||
|
||||
```
|
||||
[storage]
|
||||
STORAGE_TYPE = local
|
||||
PATH = /mystorage
|
||||
```
|
||||
|
||||
will create attachments in `/mystorage/attachments` instead of
|
||||
`/appdata/attachments`, LFS files in `/mystorage/lfs` instead of
|
||||
`/appdata/lfs`, etc.
|
||||
|
||||
In the section dedicated to a subsystem (see the table above), `PATH`
|
||||
In the section dedicated to a subsystem (see the table in the introduction), `PATH`
|
||||
is the base path under which all files will be stored. For instance:
|
||||
|
||||
```
|
||||
|
|
@ -94,44 +149,73 @@ PATH = /otherstorage/attachments
|
|||
will store attachments in `/otherstorage/attachments` while `lfs`
|
||||
files will be stored in `/mystorage/lfs`.
|
||||
|
||||
## `minio` storage
|
||||
### Using minio
|
||||
|
||||
When the `STORAGE_TYPE` is set to `minio`, the settings available in
|
||||
all sections (`[storage]` and `[XXXX]`) are:
|
||||
When the `STORAGE_TYPE` is set to `minio`, the settings are used to to
|
||||
connect to a S3 compatible server:
|
||||
|
||||
- `SERVE_DIRECT`: **false**: Allows the storage driver to redirect to authenticated URLs to serve files directly. Only supported via signed URLs.
|
||||
- `MINIO_ENDPOINT`: **localhost:9000**: Minio endpoint to connect.
|
||||
- `MINIO_ACCESS_KEY_ID`: Minio accessKeyID to connect.
|
||||
- `MINIO_SECRET_ACCESS_KEY`: Minio secretAccessKey to connect.
|
||||
- `MINIO_BUCKET`: **gitea**: Minio bucket to store the data.
|
||||
- `MINIO_LOCATION`: **us-east-1**: Minio location to create bucket.
|
||||
- `MINIO_USE_SSL`: **false**: Minio enabled ssl.
|
||||
- `MINIO_INSECURE_SKIP_VERIFY`: **false**: Minio skip SSL verification.
|
||||
- `MINIO_ENDPOINT`: **localhost:9000**: S3 endpoint to connect.
|
||||
- `MINIO_ACCESS_KEY_ID`: S3 accessKeyID to connect.
|
||||
- `MINIO_SECRET_ACCESS_KEY`: S3 secretAccessKey to connect.
|
||||
- `MINIO_BUCKET`: **forgejo**: S3 bucket to store the data.
|
||||
- `MINIO_LOCATION`: **us-east-1**: S3 location to create bucket.
|
||||
- `MINIO_USE_SSL`: **false**: S3 enabled ssl.
|
||||
- `MINIO_INSECURE_SKIP_VERIFY`: **false**: S3 skip SSL verification.
|
||||
|
||||
One setting is only available in the `[XXXX]` sections:
|
||||
|
||||
- `MINIO_BASE_PATH`: defaults to the `default base path` of the `XXXX`
|
||||
subsystem (see the table above) and is a relative path within the
|
||||
MinIO bucket defined by `MINIO_BUCKET`.
|
||||
|
||||
## Sections precedence
|
||||
|
||||
The sections in which a setting is found have the following priority:
|
||||
|
||||
- [XXXX] has precedence
|
||||
- [storage] is the default
|
||||
|
||||
For instance:
|
||||
When used in the `[storage]` section they apply to all
|
||||
subsystems. When used in the section specific to a subsystem (see the table in the introduction), they
|
||||
are only used for objects that belong to this subsystem. Here is a example:
|
||||
|
||||
```
|
||||
[storage]
|
||||
PATH = /default
|
||||
STORAGE_TYPE = minio
|
||||
|
||||
[attachment]
|
||||
PATH = /first
|
||||
SERVE_DIRECT = false
|
||||
MINIO_ENDPOINT = garage:9000
|
||||
MINIO_ACCESS_KEY_ID = [redacted]
|
||||
MINIO_SECRET_ACCESS_KEY = [redacted]
|
||||
MINIO_BUCKET = forgejo
|
||||
MINIO_LOCATION = us-east-1
|
||||
MINIO_USE_SSL = false
|
||||
MINIO_INSECURE_SKIP_VERIFY = false
|
||||
|
||||
[lfs]
|
||||
STORAGE_TYPE = minio
|
||||
MINIO_BASE_PATH = nonstandardlfs/
|
||||
|
||||
SERVE_DIRECT = false
|
||||
MINIO_ENDPOINT = minio:9000
|
||||
MINIO_ACCESS_KEY_ID = [redacted]
|
||||
MINIO_SECRET_ACCESS_KEY = [redacted]
|
||||
MINIO_BUCKET = forgejo
|
||||
MINIO_LOCATION = us-east-1
|
||||
MINIO_USE_SSL = false
|
||||
MINIO_INSECURE_SKIP_VERIFY = false
|
||||
```
|
||||
|
||||
Will set the value of `PATH` for attachments to `/first`.
|
||||
- `MINIO_BASE_PATH`: **only valid in the specific subsystem section (see the table in the introduction)**
|
||||
overrides the default directory in which objects are stored in the `MINIO_BUCKET` bucket.
|
||||
|
||||
For all subsystems that use the `minio` storage type found in the
|
||||
`[storage]` section, the directory in which the objects are stored is
|
||||
determined using the table in the introduction. For instance LFS files will be
|
||||
stored in the `lfs/` directory within the `forgejo` bucket.
|
||||
|
||||
When the `minio` storage is set in a section specific to a subsystem,
|
||||
the `MINIO_BASE_PATH` setting can be used to override the default
|
||||
directory. In the example above, `MINIO_BASE_PATH = nonstandardlfs/`
|
||||
means LFS objects will be stored in the `nonstandardlfs/` directory
|
||||
within the `forgejo` bucket instead of the `lfs/` directory
|
||||
|
||||
## S3 servers compatibility
|
||||
|
||||
Although the S3 storage type is named `minio` it does not rely on any
|
||||
[MinIO](https://min.io/) specific features. The S3 storage type is
|
||||
[tested](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/.forgejo/upgrades/test-upgrade.sh) to be compatible with:
|
||||
|
||||
- [MinIO](https://min.io/) 2021.3.17 and 2023-08-23
|
||||
- [garage](https://garagehq.deuxfleurs.fr/) v0.8.2
|
||||
|
||||
## Undocumented features
|
||||
|
||||
|
|
@ -140,3 +224,29 @@ such as `[storage.attachments]` as an alternative to `[attachment]`
|
|||
for instance (the plural is not a typo, it is a unification problem) -
|
||||
because their behavior is not thoroughly tested and may lead to
|
||||
unexpected results.
|
||||
|
||||
There are no safeguards preventing the use of undocumented features
|
||||
because it may not be backward compatible when upgrading from Gitea to
|
||||
Forgejo.
|
||||
|
||||
## Legacy settings
|
||||
|
||||
Some settings are deprecated but still supported in the interest of
|
||||
backward compatibility. They should be replaced as follows:
|
||||
|
||||
- `[server].LFS_CONTENT_PATH` is replaced with `[lfs].PATH`
|
||||
- `[picture].AVATAR_UPLOAD_PATH` is replaced with `[avatar].PATH`
|
||||
- `[picture].REPOSITORY_AVATAR_UPLOAD_PATH` is replaced with `[repo-avatar].PATH`
|
||||
|
||||
Legacy settings have a lower priority and will be overridden by their
|
||||
replacement if both are present. For instance:
|
||||
|
||||
```
|
||||
[picture]
|
||||
AVATAR_UPLOAD_PATH = /legacy_path
|
||||
|
||||
[avatar]
|
||||
PATH = /avatar_path
|
||||
```
|
||||
|
||||
will store avatar files in `/avatar_path`.
|
||||
|
|
|
|||
Loading…
Reference in a new issue