Panoptic segmentation

Panoptic segmentation models map each pixel of the input image to a category and an instance id.
It could be seen as performing both semantic segmentation and instance segmentation at the same time (with the constraint that instance masks cannot overlap, since each pixel is assigned a single instance id).

Dataset format

Datasets follow this structure:

endpoint_url/bucket
├── prefix/images/
├── prefix/panoptic_maps/
└── prefix/metadata.yaml

Dataset images are placed directly inside images/ (subdirectories are ignored).
The metadata file looks something like this:

task: panoptic segmentation
annotations: panoptic_maps/
categories: [cat1, cat2, cat3, cat4]
colors:
  cat1: [255, 0, 0]  # red
  cat2: [0, 0, 255]  # blue
  cat3: [255, 255, 0]  # yellow

The annotations field specifies the name of the folder containing the ground truth annotations, which share the file name with the image they are associated with (e.g. panoptic_maps/000.tif annotates images/000.jpg).
An annotation is a 2-page TIFF file; the first page is an RGB image representation of the category map, while the second is a grayscale instance id map.
Category colors can be any RGB triplet except for black ([0, 0, 0]), which is reserved for pixels to be ignored in training and validation.
Instance ids can be any value, as long as each instance is assigned a unique one. For example, they could be consecutive (e.g. 0, 1, 2 etc...) or spread out (e.g. 0, 128, 255).
All pages in the same TIFF must have the same shape, but that shape can be scaled down compared to the associated input image (the aspect ratio must be equal).

Here's a toy example showing how such an annotation file could be created in Python:

import numpy as np
import tifffile
category_map = np.random.randint(0, 256, size=(3, 100, 100), dtype=np.uint8)  # channels-first!
id_map = np.random.randint(0, 256, size=(100, 100), dtype=np.uint8)
with tifffile.TiffWriter("tmp.tiff") as tiff_writer:
    tiff_writer.write(category_map, photometric="rgb")
    tiff_writer.write(id_map, photometric="minisblack")

with tifffile.TiffFile("tmp.tiff") as tiff_file: 
    images = [page.asarray() for page in tiff_file.pages]
assert images[0].shape == (3, 100, 100)
assert images[1].shape == (100, 100)