Schema changes

Your data model will evolve over time, so TwiceDB supports adding and removing fields, indexes, and projections.

Suppose that your ticketing system uses a model like this:

type Ticket struct {
  Team *string
  Description *string
  Done *bool
}

Java

@TwiceDBObject
public class Ticket extends Indexable<Ticket> {
  public String team;
  public String description;
  public Boolean done;
}

After some time, you may want to begin tracking how long each Ticket takes so that you can better plan future work. Let’s add a new field to track that. Note that in addition to adding the field to the model itself, we also have to register the new field so that you can continue to read data written with the old schema:

type Ticket struct {
  Team *string
  Description *string
  Done *bool
  HoursWorked *int
}

func init() {
  shape_registry.RegisterAddableField(
    (&Ticket{}).Fqn(),
    cshared.NewIntAddableField(
      cshared.FieldName("HoursWorked"),
      0))
}

Java

@TwiceDBObject
public class Ticket extends Indexable<Ticket> {
  static {
    ShapeRegistry.registerAddableField(
      TicketFactory.getNewNull().getFqn(),
      new AddableField(
        new FieldName("hoursWorked"),
        0));
  }
  public String team;
  public String description;
  public Boolean done;
  public Integer hoursWorked;
}

With enough tickets, you’ll probably want to be able to search quickly and to do some server-side analytics. Let’s update the model to support queries by team and done and aggregations on hoursWorked:

type Ticket struct {
  Team *string `twicedb:"index"`
  Description *string
  Done *bool `twicedb:"index"`
  HoursWorked *int `twicedb:"project"`
}

func init() {
  shape_registry.RegisterAddableField(
    (&Ticket{}).Fqn(),
    cshared.NewIntAddableField(
      cshared.FieldName("HoursWorked"),
      0))
}

Java

@TwiceDBObject
public class Ticket extends Indexable<Ticket> {
  static {
    ShapeRegistry.RegisterAddableField(
      TicketFactory.getNewNull().getFqn(),
      new AddableField(
        new FieldName("hoursWorked"),
        0));
  }
  @TwiceDBIndex public String team;
  public String description;
  @TwiceDBIndex public Boolean done;
  @TwiceDBProject public Integer hoursWorked;
}

After making changes to indexes or projections, you must update the persisted data in TwiceDB itself. This is not necessary to add or remove fields that are neither indexed nor projected.

it := NewTicketIterator(
  client,
  shared.BatchSize(100),
  client.Lsqt().Tt())
for {
	hn, _ := it.HasNext()
	if !hn {
		break
	}
	ticket, _ := it.Next()
	_, _ = ticket.ReindexObject(client)
}

Java

Iterator<Ticket> it = TicketIterator.getNew(
	client,
	client.getLsqt().toTt(),
	new BatchSize(cnt));
while (it.hasNext()) {
	Ticket ticket = it.next();
	WrittenObject _ = client.ReindexObject(ticket);
}

It’s not necessary to lock Ticket while reindexing, but queries on indexes and projections will return incomplete results until all Tickets originally written with the old schema have been reindexed. Tickets written for the first time with the new indexes and projections do not need to be reindexed. However, since reindexing is idempotent, the data will remain consistent even if you do reindex them.