Is it possible to setup conduit with a Postgres database? #307

New issue

Open

opened 2022-10-06 05:44:14 +00:00 by BobWs · 4 comments

BobWs commented

2022-10-06 05:44:14 +00:00

(Migrated from gitlab.com)

Headline

Description

Hi,
Is it possible to setup conduit with a Postgres database? If so how do I do that?
If not would you please consider adding the possibility!

TIA

## Headline ### Description Hi, Is it possible to setup conduit with a Postgres database? If so how do I do that? If not would you please consider adding the possibility! TIA

Nyaaori commented

2022-10-07 01:50:57 +00:00

(Migrated from gitlab.com)

To clarify: Conduit is built using Key-Value databases [a form of NoSQL database] and they are significantly different from SQL databases in design, and while one could map a Key-Value database into an SQL database, it would not have good performance. That poor performance caused by such direct mapping contributes largely to why SQLite is not set by default anymore.

If you happen to be running a Conduit instance on an SQLite database already, consider migrating it to RocksDB. conduit_toolbox from https://github.com/ShadowJonathan/conduit_toolbox will allow you to convert existing Conduit databases.

In short: PostgreSQL support could technically be added, but it would likely be detrimental to performance without significant changes to how Conduit uses its database, and you should consider using RocksDB instead.

Conduit makes use of database technology which happens to be significantly different from SQL and because of that, making use of an SQL database would likely result in a significant reduction in performance since it would not be using any features which makes SQL fast. To clarify: Conduit is built using Key-Value databases [a form of NoSQL database] and they are significantly different from SQL databases in design, and while one could map a Key-Value database into an SQL database, it would not have good performance. That poor performance caused by such direct mapping contributes largely to why SQLite is not set by default anymore. If you happen to be running a Conduit instance on an SQLite database already, consider migrating it to RocksDB. conduit_toolbox from https://github.com/ShadowJonathan/conduit_toolbox will allow you to convert existing Conduit databases. In short: PostgreSQL support could technically be added, but it would likely be detrimental to performance without significant changes to how Conduit uses its database, and you should consider using RocksDB instead.

BobWs commented

2022-10-07 06:29:16 +00:00

(Migrated from gitlab.com)

Thanks for explaining it is clear to me why the choice of RocksDB and not for SQL based databases.

I’m already running conduit with RocksDB, but I was wondering if a SQL based database would be easier to maintain, because I ran into some problems when I tried to migrate my conduit server to another host. I used Portainer to migrate my conduit container to another host and at first everything went fine and was working as expected, but the next day conduit stopt working and was complaining about the database was missing. So as I don’t know much about RocksDB and couldn’t find a solution, I decided to delete the entire mounted db folder and started with a fresh install.

So eventually everything is working again, but it wasn’t how it was supposed to go.

Thanks for explaining it is clear to me why the choice of RocksDB and not for SQL based databases. I’m already running conduit with RocksDB, but I was wondering if a SQL based database would be easier to maintain, because I ran into some problems when I tried to migrate my conduit server to another host. I used Portainer to migrate my conduit container to another host and at first everything went fine and was working as expected, but the next day conduit stopt working and was complaining about the database was missing. So as I don’t know much about RocksDB and couldn’t find a solution, I decided to delete the entire mounted db folder and started with a fresh install. So eventually everything is working again, but it wasn’t how it was supposed to go.

timokoesters commented

2022-10-07 07:18:35 +00:00

(Migrated from gitlab.com)

@Nyaaori

without significant changes to how Conduit uses its database,

I've been working on exactly these significant changes for a while now on the 'refactor' branch. I'm working towards adding postgres support and also improving our reliability guarantees.

@Nyaaori > without significant changes to how Conduit uses its database, I've been working on exactly these significant changes for a while now on the 'refactor' branch. I'm working towards adding postgres support and also improving our reliability guarantees.

erlend-sh commented

2024-01-25 08:47:34 +00:00

(Migrated from gitlab.com)

Just to add another data point in here, Postgres support is also very useful for any app (like Commune) that wants to support multiple pluggable homeserver backends, e.g. Synapse & Conduit. Such a dual setup is more difficult to maintain if the homeservers are using completely different types of databases.

p.s. it appears the refactor branch has since been merged in !365. How far did that get us?

p.p.s. from Timo:

conduit uses traits for the storage backend and it's possible to add postgres with some dedication: https://gitlab.com/famedly/conduit/-/blob/next/src/service/rooms/timeline/data.rs

Just to add another data point in here, Postgres support is also very useful for any app (like [Commune](https://github.com/commune-os/commune-rs)) that wants to support multiple pluggable homeserver backends, e.g. Synapse & Conduit. Such a dual setup is more difficult to maintain if the homeservers are using completely different types of databases. p.s. it appears the refactor branch has since been merged in !365. How far did that get us? p.p.s. from Timo: > conduit uses traits for the storage backend and it's possible to add postgres with some dedication: https://gitlab.com/famedly/conduit/-/blob/next/src/service/rooms/timeline/data.rs