MuseGlimmerVisionEncoder model

[source]

MuseGlimmerVisionEncoder class

keras_hub.models.MuseGlimmerVisionEncoder(
    num_layers,
    hidden_size,
    num_heads,
    intermediate_size,
    patch_size=14,
    patch_temporal=2,
    merge_size=2,
    pos_emb_height=32,
    pos_emb_width=32,
    rope_theta=10000.0,
    layer_norm_eps=1e-05,
    layer_types=None,
    out_hidden_size=None,
    max_num_windows=None,
    max_num_frames=None,
    max_frame_size=None,
    dtype=None,
    **kwargs
)

MuseGlimmer's ViT-style perception encoder.

The encoder takes pre-extracted, pre-flattened image patches. It applies patch embedding and bilinear position embeddings. It alternates window and full attention. Window attention uses an index permutation, not a windowed attention mask. The encoder also applies axial 2D RoPE and pixel-shuffle patch merging.

Attention runs per window and per frame. With eager inputs, the encoder batches the windows and frames of the same length with no padding. This matches the HF eager attention exactly. A set cap (max_num_windows, max_num_frames, max_frame_size) must fit the input, or the encoder raises a ValueError.

Traced inputs (for example under jax.jit) have no concrete grid values. For these inputs, the caps set the sizes of padded slot buffers. If a cap is None, the encoder uses one masked attention over the full sequence. This attention needs memory quadratic in the patch count. Set the caps to get static shapes under tracing. Each call then reserves memory for the full caps. Traced inputs skip the fit check. A cap that is too small gives incorrect output.

Arguments

  • num_layers: int. Number of transformer blocks.
  • hidden_size: int. ViT hidden dimension.
  • num_heads: int. Number of attention heads.
  • intermediate_size: int. MLP intermediate dimension.
  • patch_size: int. Spatial patch size in pixels.
  • patch_temporal: int. Temporal patch size in frames.
  • merge_size: int. Spatial merge (pixel-shuffle) factor.
  • pos_emb_height: int. Learned position-embedding grid height.
  • pos_emb_width: int. Learned position-embedding grid width.
  • rope_theta: float. Axial RoPE base wavelength. Defaults to 10000.0.
  • layer_norm_eps: float. Epsilon for ln_pre and ln_post. The block norms always use 1e-5, like Hugging Face. Defaults to 1e-5.
  • layer_types: list of str or None. Per-layer "window_attention" or "full_attention". Defaults to every 4th layer (and the last) being full attention.
  • out_hidden_size: int or None. Output dim after pixel-shuffle merge; defaults to hidden_size * merge_size ** 2.
  • max_num_windows: int or None. Optional opt-in cap for traced inputs. When set, window-attention layers put each window into a fixed (max_num_windows, window_patches ** 2) buffer. The value must be at least the total number of windows in one call. Defaults to None.
  • max_num_frames: int or None. Optional opt-in cap for traced inputs. Full-attention layers put each frame into a fixed (max_num_frames, max_frame_size) buffer. The value must be at least the total number of frames in one call. Set max_frame_size together with this argument. Defaults to None.
  • max_frame_size: int or None. The slot size for max_num_frames. The value must be at least the largest per-frame patch count (height * width) in one call. Defaults to None.
  • dtype: string or keras.mixed_precision.DTypePolicy. The dtype to use for model computations and weights. Defaults to None.