Related posts
Discover more insights and tips to boost your visual and physical AI workflows.
View all
minimax-m3-fiftyone plugin exposes M3 as both a FiftyOne Zoo model (foz.load_zoo_model("minimax/minimax-m3", task="detect")) and an App operator + chat panel, no training, no endpoint to stand up, just an HF_TOKEN.detect, keypoints, frame_detect, find_event, key_moments, caption_concise, caption_detailed, classify_single, classify_multi, and vqa, writing real fo.Detections, fo.Keypoints, fo.TemporalDetections, and fo.Classification(s) objects, not raw text that you have to parse yourself.[0, 1], which FiftyOne consumes with no rescaling.Ask MiniMax-M3 chat panel lives inside the sample modal: it streams multi-turn answers for each sample and includes a Convert to FiftyOne button that converts a JSON-shaped reply into a label field (m3_detections, m3_keypoints, m3_events, etc.) without leaving the modal.minimax_* attributes on the label object itself.quickstart dataset, M3’s raw response is:minimax_parser.to_fiftyone) turns directly into fo.Detection(label="chicken", bounding_box=[0.247, 0.0, 0.432, 0.87]), FiftyOne’s [x, y, w, h] convention, computed from M3’s top-left/bottom-right box. Ask for keypoints on the same image, and you get four labeled points back: chicken head, chicken body, chicken tail, chicken feet, each a normalized [x, y] pair.truck, car, two building instances, stop sign, street lamp, trash can, commercial sign, person, each with a normalized box. Ask it when an event happens over the whole sampled strip (“identify distinct events/activities and WHEN they occur”), and it responds with start/end seconds directly:fo.TemporalDetection with support=[start_frame, end_frame] computed from the video’s frame rate, plus the raw t_start_seconds / t_end_seconds kept as attributes.minimax_parser.py strips <think> blocks and code fences, then scans left-to-right for the first balanced JSON span rather than assuming the whole response is clean JSON, so stray prose before or after the payload doesn’t break parsing. It also tolerates alternate key names (box / bbox / bbox_2d / bounding_box; label / class / category), wrapper objects ({"detections": [...]}) instead of bare lists, and falls back gracefully: a keypoints request that gets a bounding box back just uses the box’s center instead of dropping the item.MiniMax-M3: run task operator works across a view. It has three modes, and the form reshapes itself depending on which one you pick and whether your dataset is images or video:detect, keypoints, per-frame frame_detect, key_moments, classify_single/multi, caption_concise/detailed, or vqa) and label the entire target view in a single run. You can type target classes as chips, or point it at an existing sample field (a string field, or a label field whose class names become the prompt) and skip samples with nothing to say.{"label": "yes"/"no", "confidence": <float>} and the view filters to matches above your threshold.TemporalDetection wherever it finds a match, then switches the App to a clips view, one row per event.prompt_tokens / completion_tokens) so cost isn’t a mystery after the fact.MiniMax-M3: the chat panel is a different shape entirely; it lives inside the sample modal for interrogating one sample conversationally. Answers stream token-by-token, conversations are multi-turn and saved per sample, and an Output selector lets you steer the reply toward Boxes / Keypoints / Temporal JSON instead of free text. When a reply is recognized as structured JSON, a Convert to FiftyOne bar appears under the message: confirm the field name (it defaults to something sensible per shape, like m3_detections or m3_keypoints) and the label writes to the sample, with its overlay appearing in the already-open modal, prev/next navigation intact.save_stream_as_label parses the stream file, writes the field, and (this took real engineering effort to get right) refreshes the open sample via FiftyOne’s own useRefreshSample hook rather than reloading the page, so the modal doesn’t tear down and lose its navigation context. The one caveat: writing into a brand-new field needs a one-time close-and-reopen of that sample so the App’s sidebar picks up the new schema path; every conversion after that first one is instant.thinking= argument:| Mode | Behavior |
|---|---|
| disabled | No reasoning. Fastest, cleanest JSON. The plugin’s default for grounding tasks (detect, keypoints, frame_detect, classification, captioning). |
| adaptive | M3 decides whether reasoning would help. Default for find_event, key_moments, and vqa. |
| enabled | Always reason first. |
thinking has to be sent as an object ({"type": "disabled"}), not a bare string, a detail easy to get wrong if you’re calling the HF router directly instead of going through the plugin or Zoo model.prompts.py that ask M3 for JSON in a fixed shape; minimax_parser.py converts whatever comes back into fo.Detections, fo.Keypoints, fo.TemporalDetections, or fo.Classification(s). There’s no training step.[0, 1], top-left/bottom-right for boxes ([x1, y1, x2, y2]) and [x, y] for points. FiftyOne consumes these directly; no per-sample rescaling against image dimensions is needed.find_event, key_moments, and shared tasks on video) or as N separate per-frame requests (frame_detect, for per-frame boxes with no cross-frame instance IDs; pair it with a downstream tracker if you need identity continuity).sample.metadata.frame_rate on every sample in the target view, since per-frame boxes need to be mapped back to a frame index. Run dataset.compute_metadata() once if you haven’t; the operator’s form blocks with an explicit error if metadata is missing.m3_detections, m3_keypoints, m3_class, m3_answer, m3_caption, m3_key_moments).minimax_prompt, minimax_raw_output, minimax_thinking, and the sampling parameters used to generate it, so you (or a reviewer) can always trace a label back to exactly how M3 produced it.detect, keypoints, frame_detect, find_event, key_moments, caption_concise, caption_detailed, classify_single, classify_multi, and vqa. Grounded tasks write real FiftyOne label objects (fo.Detections, fo.Keypoints, fo.TemporalDetections, and fo.Classification(s)), and captioning and VQA write text fields.fiftyone plugins download https://github.com/harpreetsahota204/minimax-m3-fiftyone, pip install openai opencv-python pillow numpy, set an HF_TOKEN, and launch the App. You can also register it as a Zoo model and call foz.load_zoo_model("minimax/minimax-m3", task="detect") in a notebook.
