CompactSize

A transaction is full of lists: a list of inputs, a list of outputs, scripts of different lengths. A reader needs to know how long each one is before reading it.

So each list or script starts with a small number saying how many items or bytes follow. That number is a CompactSize, sometimes called a varint.

The trick is that it’s compact. Most counts are tiny, so most CompactSizes are one byte. Only a big number pays for more bytes.

Try it

Type a number to see its CompactSize bytes, or type bytes to decode them.

How does it work?

The first byte decides everything. If it’s below 0xfd (253), it is the number. If it’s one of the three marker bytes, it tells you how many bytes to read next.

Size

1, 3, 5 or 9 bytes

Type

Unsigned integer

Format

Marker byte, then little-endian

Range

0 to 2⁶⁴ − 1

A few worked examples, all computed with the codex’s own encoder:

The bytes after the marker are little-endian, smallest byte first, like every other integer in Pearl.

Where are the CompactSizes in a real transaction?

Here is a real Pearl payment from block 130,825: one input, two outputs.

It contains seven CompactSizes, and every one of them fits in a single byte:

The 34-byte scripts are Taproot locks (51 20 followed by a 32-byte key), so 22 appears in almost every Pearl output.

Why not just use a fixed size?

Because most counts are tiny, and every byte is stored by every full node and paid for in fees.

If every count and length in the payment above were a fixed 4-byte number, its seven CompactSizes would take 28 bytes instead of 7, and the 205-byte transaction would grow to 226 bytes, about 10% bigger, to carry no extra information. CompactSize spends the extra bytes only on the rare big number.

Where else does Pearl use them?

Anywhere something has a variable length:

• The transaction count in a block. After the certificate and the 108-byte header, block 130,825 has the byte 73: 115 transactions.
• Witness data. Each input’s witness starts with an item count, and each item with its length (BIP-144, same as Bitcoin).
• Taproot and P2MR script trees. A leaf hash covers the leaf version, then a CompactSize of the script length, then the script. A 34-byte leaf script gets the prefix 22.

What does “non-canonical” mean?

Nothing stops you writing 5 as fd 05 00. It decodes to 5. But it’s wasteful, and worse, it means the same data could be written two ways, giving two different transaction IDs.

So Pearl’s decoder, like Bitcoin’s, rejects any CompactSize that could have been shorter. Each number has exactly one valid encoding:

• after fd, the value must be at least 253;
• after fe, at least 65,536;
• after ff, at least 4,294,967,296.

switch {
case val < 0xfd:
	buf[0] = uint8(val)
	_, err := w.Write(buf[:1])
	return err

case val <= math.MaxUint16:
	buf[0] = 0xfd
	littleEndian.PutUint16(buf[1:3], uint16(val))
	_, err := w.Write(buf[:3])
	return err

case val <= math.MaxUint32:
	buf[0] = 0xfe
	littleEndian.PutUint32(buf[1:5], uint32(val))
	_, err := w.Write(buf[:5])
	return err
// default: 0xff, then 8 bytes
}

function compactSize(n) {
  if (n < 0xfd) return [n];
  if (n <= 0xffff) return [0xfd, n & 0xff, n >> 8];
  if (n <= 0xffffffff) return [0xfe, n & 0xff, (n >> 8) & 0xff, (n >> 16) & 0xff, n >>> 24];
  throw new Error('use BigInt for 9-byte values');
}
compactSize(1000); // [253, 232, 3] = fd e8 03