Backfilling data

If you’re working on an established project, you’ll likely have lots of data already. Fortunately, it’s easy to backfill it into TwiceDB.

Locking

To prevent reading partial data, you must lock the class(es) to be backfilled:

func main() {
  tt, _ := client.LockFqn(
    client.Context(),
    (&MyObject{}).Fqn())
  defer client.UnlockFqn(
    client.Context(),
    (&MyObject{}).Fqn())
  ...
}

Java

public static void main(String[] args) {
  try {
    Tt tt = client.lockFqn(
      new MyObject().getFqn(new MyObject()));
    ...
  } finally {
    Tt tt = client.unlockFqn(
      new MyObject().getFqn(new MyObject()));
  }
}

While a lock is active, the class may not be read, nor may the class be written without a backfill tt. Backfilling must be done in ascending backfillTt order per class.

Setting the Tt

The primary difference between normal use and backfilling is that in normal use, TwiceDB assigns the tt to writes. In backfilling, the user assigns the tt by setting the backfillTt:

o := NewMyObject(...)
o.SetBackfillTt(shared.TtOf(
  time.Date(2009, time.January, 2, 0, 0, 0, 0, time.UTC)))
w, _ := o.PutObject(
  client,
  shared.VtOf(time.Date(2009, time.January, 1, 0, 0, 0, time.UTC)))

Java

MyObject o = MyObjectFactory.getNew(...);
o.setBackfillTt(
  Tt.of(LocalDate.of(2009, Month.JANUARY, 2)));
WrittenObject w = IndexableI.putObject(
  client,
  o,
  Vt.of(2009, Month.JANUARY, 1));

The sync and async put and delete methods are supported.

Transactions are also supported as long as:

only locked classes participate in the transaction
all objects within the transaction have the same backfillTt

o0 := NewMyObject(...)
o1 := NewMyObject(...)
backfillTt := shared.TtOf(
  time.Date(2009, time.January, 2, 0, 0, 0, 0, time.UTC))
o0.SetBackfillTt(backfillTt)
o1.SetBackfillTt(backfillTt)
tx := txn.NewTxn()
vt := shared.VtOf(
  time.Date(2009, time.January, 1, 0, 0, 0, time.UTC))
tx.PutObject(o0, vt)
tx.PutObject(o1, vt)
w, _ := client.CommitTxn(tx)

Java

MyObject o0 = MyObjectFactory.getNew(...);
MyObject o1 = MyObjectFactory.getNew(...);
Tt backfillTt = Tt.of(
  LocalDate.of(2009, Month.JANUARY, 2));
o0.setBackfillTt(backfillTt)
o1.setBackfillTt(backfillTt)
Vt vt = Vt.of(2009, Month.JANUARY, 1);
Txn tx = new Txn();
tx.putObject(o0, vt);
tx.putObject(o1, vt);
WrittenTransaction w = client.commitTxn(tx);