Draft: Work in progres for backup and restore data from running conduit instance #971

Open

tglman wants to merge 12 commits from WIP_backup into next

tglman commented

2023-08-27 18:10:42 +00:00

(Migrated from gitlab.com)

Hi,

In the context of #16

This add some API in the KVEngine for export all the data, I used in all implementation I could the snapshot mechanism, to have a point in time read that do not include data that refer not existing data, the implementations that do not support snapshots are sqlite,sled and heed, sqlite just because rusqlite do not support it by now, but is available in sqlite itself so in long term should be possible to support also that.

An additional limitation is that as today an export of data may include partial "executions" because operations are not executed as a single transaction in the underlying storage as today,
to fix this is needed that the high level operations are executed in transaction nothing can be done at export level.

As far this include only the low level implementation, more work is needed to define a simple export format to a file and a way to schedule or trigger the export, together with some import/restore functionalities.

I agree to release my code and all other changes of this MR under the Apache-2.0 license

Hi, In the context of #16 This add some API in the KVEngine for export all the data, I used in all implementation I could the snapshot mechanism, to have a point in time read that do not include data that refer not existing data, the implementations that do not support snapshots are sqlite,sled and heed, sqlite just because rusqlite do not support it by now, but is available in sqlite itself so in long term should be possible to support also that. An additional limitation is that as today an export of data may include partial "executions" because operations are not executed as a single transaction in the underlying storage as today, to fix this is needed that the high level operations are executed in transaction nothing can be done at export level. As far this include only the low level implementation, more work is needed to define a simple export format to a file and a way to schedule or trigger the export, together with some import/restore functionalities. ----------------------------------------------------------------------------- - [x] I agree to release my code and all other changes of this MR under the Apache-2.0 license

tglman commented

2023-08-30 21:36:07 +00:00

(Migrated from gitlab.com)

added 1 commit

c6ab352e - feat: add implementation to export and import low level data in json

Compare with previous version

added 1 commit <ul><li>c6ab352e - feat: add implementation to export and import low level data in json</li></ul> [Compare with previous version](/famedly/conduit/-/merge_requests/546/diffs?diff_id=775204193&start_sha=1755eded99c44ec1ef0b0f609b00acf5321739d4)

tglman commented

2024-01-14 13:29:45 +00:00

(Migrated from gitlab.com)

added 25 commits

c6ab352e...ca621972 - 13 commits from branch famedly:next
ca621972...aad718d3 - 2 earlier commits
5d764947 - refactor: changed tree specific impls to be one liner
5498863d - test: add tests for rocksdb and persy impl
eaec8286 - test:add entry for seamless remove of not existing key from database
1c198f79 - fix:update implementation to recent changes on next
418aeee0 - chore: fix tests after refactor
db60a7c0 - fix: make heed implementation compile
089c7284 - feat: first implementation of KeyValue data export
5ecc1e35 - feat: additional implementation of export logic
08cde527 - test: add test for exporting and importing data from tree store
ba52bcef - feat: add implementation to export and import low level data in json

Compare with previous version

added 25 commits <ul><li>c6ab352e...ca621972 - 13 commits from branch <code>famedly:next</code></li><li>ca621972...aad718d3 - 2 earlier commits</li><li>5d764947 - refactor: changed tree specific impls to be one liner</li><li>5498863d - test: add tests for rocksdb and persy impl</li><li>eaec8286 - test:add entry for seamless remove of not existing key from database</li><li>1c198f79 - fix:update implementation to recent changes on next</li><li>418aeee0 - chore: fix tests after refactor</li><li>db60a7c0 - fix: make heed implementation compile</li><li>089c7284 - feat: first implementation of KeyValue data export</li><li>5ecc1e35 - feat: additional implementation of export logic</li><li>08cde527 - test: add test for exporting and importing data from tree store</li><li>ba52bcef - feat: add implementation to export and import low level data in json</li></ul> [Compare with previous version](/famedly/conduit/-/merge_requests/546/diffs?diff_id=893977927&start_sha=c6ab352e133539ff83dc9ef65cbfe50d9bd66c11)

This pull request is broken due to missing fork information.

View command line instructions