Analysis
X Algorithm Analysis: Post Reach and Visibility
A code-citing technical reference to the X For You feed, current as of the November 19, 2026 tree.
This document tracks research into the X “For You” feed algorithm to understand how post reach and visibility are determined, and what factors influence these metrics. Current as of the Nov 19, 2026 tree (commit b551517; the Aug 13βNov 19 history is preserved in the addenda below).
Key Questions to Answer#
- How does the algorithm determine reach or visibility for a post? What factors increase or diminish reach?
- What account attributes affect reach of a post?
- How truthful is the README documentation? β See Section 11
- How does account reputation factor into reach, and how is reputation determined? β See Section 12
Related documents: this is the primary, code-citing technical reference. account-analysis.md is the technical addendum that answers scenario-specific questions against this code, and user-guide.md is the non-technical plain-English playbook distilled from both.
DOC ACCURACY NOTE (August 2026 revision): This document was rewritten for the Aug 13, 2026 release (commit
a389166, +2053 files / +363K lines). The previous (July 2026) revision is archived below. The Aug release is a fundamentally different codebase:
- The scoring weights are no longer hidden. All weight values (favorite 0.5, reply 5.0, report β234.0, β¦),
MAX_POST_AGE = 48h,TOP_K_CANDIDATES_TO_SELECT = 50,OonWeightFactor = 0.75,AuthorDiversityDecay = 0.5, andUAS_WINDOW_TIME_MS = 5 minare now open in code (home-mixer/params/param.rs,home-mixer/params/config.rs). Section 5’s old “weight configuration is missing” finding is obsolete.visibility-filtering/(the VF service) is now open source β every drop rule, the ALLOW/INTERSTITIAL/DROP verdict model, and the user/tweet safety labels are visible (Sections 3, 11, 12.5).vm-ranker/is now open source, but the proprietary value-model inference inside it was redacted: the released server only runs a DPP (determinantal point process) diversity filter over multimodal embeddings. The head-weight formula now lives client-side in home-mixer (Section 5.3, 12.5).phoenix/now ships the production training code (phoenix/xrex/), real model configs (8 layers, emb 2560, history 1022, candidate 64, SID 6Γ256), and synthetic-data training; the old artifacts zip is gone β no pretrained checkpoint is included (Section 5.1, 10).- Account reputation is no longer “absent from the codebase”:
user-cred-v2/(PageRank credibility),abuse-enforcement-service/,scarecrow/(BotMaker + spam-label store),bdsm/,agatha/, and user-level safety-label aggregation are all released (Section 12).- The old doc’s July claims (dead-code
weighted_scorer/oon_scorer/author_diversity_scorer, VMRanker “omission”, “no reputation score anywhere”) are superseded. The old release’s scoring chain (RankingScorer + VMRanker override) survives, but with new heads, modes, and values.
AUG 14, 2026 ADDENDUM (commit
c65aa17): A follow-up commit the next day (21 files, +2,271/β32 lines) added: (1) official comments inparam.rs/ranking_scorer.rs/README clarifying that weights scale predicted probabilities, not raw counts (“one report cancels 468 likes” is called out as an incorrect interpretation; the report head is β234 because the baseline report probability is >1,000Γ lower than a like), and that mass block/report campaigns mostly affect similar users because recommendations are personalized and only Home-Timeline-served impressions count; (2)Brazil2026ElectionFilterβ a new pre-scoring filter (no feature switch) dropping posts from 665 hardcoded user IDs reported to Brazil’s Electoral Court (plus their retweets/quotes/thread ancestors), unless the viewer follows the account (Section 3, 9); (3) three new serving-side model bool features βIS_AUTHOR_FOLLOWED_BY_VIEWER,IS_AUTHOR_FOLLOWING_VIEWER(both follow directions, candidate + history tokens) andIS_STALE_POST14D(zeroes engagement counts for posts > 14 days old whenenable_stale_post, ships off); their indices live in the internalFEATURE_CONFIG_PYconfig, not the shippedfeature_config.py(Section 5.1, 12.6); (4) grox blast-radius thresholds raised 15,000 β 30,000 (Section 9, 12.3); (5) a cold-start Thompson-sampling variant (off by default) (Section 5.3); (6) a newvideo_14daySID retrieval window (now 10) (Section 5.4); (7) bdsm enforcement-note structure now visible (dominant bot head > 0.5, β₯ 30 actions; templates redacted) (Section 12.4). None of this overturns the analysis; Section 1.B, 3, 5.1, 5.3, 5.4, 9, 12.3, 12.6 and 13.3 carry the details.
SEPT 2, 2026 ADDENDUM (commit
85ac72a, 13 commits Aug 17βSept 2): scoring weights moved (video_open0.05β0.07,vqv0.05β0.0, dwell probability head 0.0β0.05); MPN scoring removed (replaced by a narrowerMultiplierPreOffsetgate, default off); the VMRanker request was stripped to ids + score (no follower counts, head weights, or slate context); cold-start eligibility now counts Home-Timeline views and the Thompson variant keeps top-2;FOLLOWING_MAX_RESULT_SIZE100β110; UAS aggregation is nowDENSE_WITH_LONG_DWELLfor scoring and retrieval; the default ads blender ismulti_riskwith brand-safety verdict V2 on; grox blast-radius gates stepped 30Kβ120K; the Brazil list grew 665β2,388 IDs (README notes an Aug 27 update); rankall went Main 14β17 / SID 10β15 windows (age-banded1fav_video_*, grok pool removed, min-age gating with prefloor sidecars) and the training EVERGREEN corpus moved tovideo_4to14day; model training moved to a Muon dense optimizer (emb lr 0.28, half-lives 2500/5000) with engagement-count MLP plus follow-direction and stale-post features enabled inhome_direct_packed(bool indices now shipped); the VFrules/rewrite is behavior-preserving (same 28+26 rules, pinned by a golden corpus) with a now-dynamic NSFW country list; enforcement gained apanda_reports_embedding_v10_rough_spam β suspendrule plus a GrowthBook-gated generic-action path; the bdsm action-count gate is now redacted (999999, was 30). Sections 1.AβB, 1.G, 2, 3, 5.1, 5.3β5.4, 6, 9β13 carry the details. None of this overturns the analysis.
OCT 13, 2026 ADDENDUM (commit
e1a0375, 28 commits Sept 3βOct 13, +114K/β23K lines across 664 files): the scoring engine was re-architected, not retuned:home-mixer/scorers/ranking_scorer.rs(1,571 lines) andvalue_model_gate.rs(dwell-regret gate) are deleted; the weighted-score + diversity + OON math now lives in a sharedxai-value-model/crate (scoring.rs/weights.rs) consumed by both home-mixer (scorers/value_model.rs, local fallback) and a much thicker vm-ranker (scoring/value_model.rs,scoring/dpp_model.rs,params.rsβ all scorer params moved server-side, incl. OON 0.75/0.5/0.00001, diversity 0.5/0.25,MultiplierPreOffset,OonRescore). Dwell-regret modes are gone. VMRanker now optionally computes the value model itself (ComputeValueModel, ships off) and carries DPP seed-pinning plus per-head weight perturbation (perturbed(sigma,salt,user_id)). Weight moves inhome-mixer/params/param.rs(sync Oct 12): favorite 0.5β0.6, video_open 0.07β0.02, click 0.4β0.3, post_unexplored 0.02β0.015, cont_dwell 0.004β0.006, cont_click_dwell 0.0β0.4 (new active head), not_interested β43.2ββ47.52; new zero-weightReactWithVideohead; multiplicative-post-unexplored, click-dwell-penalty, MPN/DwellRegret params deleted. Cold start rewritten: follower cap 1,000β50,000, impression threshold 1,000β200, max age 24hβ2h, position ratio 0.85β0.97, Thompson sampling on by default (EnableColdStartThompsonSampling=true); viewer-cold-start flag new. Pipeline: 19th pre-scoring filter (FavHoldoutFilter, off; fav-scaled 0β15% holdout) + post-selectionAncestorVideoFilter+VideoCarouselFilter(last, off; up to 30 extra candidates, carousel max 5/min 3, position 6); two new sources βPopularPostsSource(on, 500 results, most-followed-active shared list) andSidSource(off; SID-prefix retrieval, 800 max, 50 seeds); OON replies now optionally servable (EnablePhoenixOonReplies, off); VF verdict model widened to Allow/Interstitial/Drop/Tombstone/Downrank/Avoid/NotEvaluated with a new level-82TimelineHomeHydrationclause set, Article lifecycle drops (draft/soft-deleted β DROP), tombstone + age-prompt blurs + limited-engagement prompts (treatment.rs), retweetβsource verdict merging (retweet.rs), andEvaluateTweetsRPC; golden corpus split into 22 files. Brazil list 2,388β2,800 (README: Sept 29). Grox gates 120Kβ250K split (GROK_GEMMA_FOLLOWER_SPLIT=250K; spam-screen + reply-rank at 250K, coordinated-spam at 180K, new Gemma reply-spam minimum 125K) plus a multi-step reply-spam flow (demotes to 0.0 + labels), v8-search multimodal embeddings, and key-frame/leading-frame/infrared video-safety lanes. Rankall Main 17β19 / SID +1fav_reply/32fav_reply(reply index windows;MmMetadatapipeline deleted), retrieval adds a 3rdld(recap-dwelled) head (w_ld=0.75) + MoL/temperature/negative-masking knobs (off) + SID-retrieval microservice, training adds a 1-layer reply two-tower v2 ranker + dwell-bucket/click-dwell/web-conversion/purchase-value plumbing (purchase loss weight 0.0 = off). Enforcement: no label/threshold change except a mockvery_high_follower_count β₯12345.67 β skip; three new default-off guardrails β page-managedaction_control(Recording/Live),off_switchkill-switch,overturn_holdvs newabuse-ledger-service.FEED_MODULE_SLOTS4β5 (FOR_YOU_MAX 47β48). Sections 1.AβB, 1.G, 2, 3, 5.1, 5.3β5.4, 6, 7, 9β13 carry the Oct details. None of this overturns the analysis β but the cold-start rewrite and the VF verdict expansion are the two changes with the largest user-visible potential.
NOV 7, 2026 ADDENDUM (commit
3b6919f, 19 commits Oct 14βNov 7, +95,795/β41,866 lines across 552 files): another large, mostly structural update. (1) Weights moved again (home-mixer/params/param.rs, sync2026-11-06): favorite 0.6β0.708, click 0.3β0.274, share 2.0β2.118, cont_dwell 0.006β0.0075, cont_click_dwell 0.4β0.385, not_interested β47.52ββ53.07, and video_open 0.02β0.0 β the video-open head no longer scores at all;react_with_videois renamedpost_videoand joined by a zero-weightgrok_analyzehead (both 0.0). (2)Brazil2026ElectionFilteris deleted β the README’s Oct 26 note says the election is over; the pipeline no longer contains it. (3) The pipeline grew to 21 pre-scoring filters:ZeroFavCodivertFilter(2nd; a viewer/author-bucketed 0-fav retrieval A/B, ships off) andNewUserSoftNsfwFilter(4th; drops OONSOFT_NSFW-labeled posts for accounts younger than 7 days with <100 scoring actions, ships off), plusFairShareRankingFilter(last, ON by default) β it proportionally caps each retrieval source’s share of the 2,800-candidate Phoenix budget, so earlier sources (Thunder) can no longer crowd later ones (Phoenix retrieval/topics/MoE/SID/cache) out of the scored pool. (4) Three new scorers:ImmersiveVideoScorer(carousel-only Phoenix pass),PageDecodeRerankScorerandPageRlExplorationScorer(page-level RL decode + PlackettβLuce exploration, both off).EnableHomeExcursionScoresnow ships true andPhoenixAggregationTypemoved toDENSE_WITH_LONG_DWELL_HOME_EXCURSION, so the viewer’s history now carries video-continuation/profile-visit signals. (5) Video carousel rebuilt (still off): immersive ranking, v8-search embedding dedupe (cosine β₯0.72), media-id/MD5 dedupe against the main feed, brand-safety risk filters (high-risk on), a seen-carousel cache, and an optional CTR model served by vm-ranker. (6) VMRanker gained a user-debias model (per-head probability recentering from the viewer’s action sequence, off), a carousel-model RPC,top_headsin responses (Under-the-Hood only), and DPP is now skipped for topic requests. (7) VF was rewritten around an action model: 9 safety levels (addsImmersiveExpandedRecommendations,TimelineHomePromotedHydration,QuotedTweetRules, notification levels), 7 action kinds (Drop / Tombstone / Appealable / SoftIntervention / LabelInterstitial / MediaRestriction / LimitedEngagement), rule/catalog/expansion engines, FOSNR actions split by follower status (followers get a notice, non-followers a drop), reaction-video propagation (a video reacting to adult/gore media is blurred), client-version switches, DDG experiments, and a sharedsafety-label-source/crate. (8) Phoenix: ranking adds time-gap recency features (19 log-spaced buckets, lrΓ4) and home-excursion history vectors tohome_direct_packed, plus 2 candidate-only layers, FP8 GEMMs, and a 0.5-weight view head; gen-recs and SID-retrieval model families are deleted from the release; page-RL models (page_decoder/page_critic/page_rl) ship; retrieval gains ahome+ld-only two-tower config, ad ID-list features, and new actions/labels. (9) Enforcement gains confidence accounting, per-head/per-classifier head guards (hourly/daily ceilings that trip the off switch), backfill runs over past decisions, and new bot-label rules (bb1_*), and drops theinauthentic_detection_v45/cluster_spamfamilies;llm_slopSpamHighRecall TTL 30dβ7d. (10) Under the Hood now reports top-3 score contributions per post plus appeal text and (optional) post IDs. Sections 1.AβB, 1.DβE, 2β3, 5.1, 5.3β5.4, 6β13 carry the details. None of this overturns the analysis.
NOV 19, 2026 ADDENDUM (commit
b551517, 8 commits Nov 10β19, +30,960/β13,930 lines across 346 files): a consolidation release with no scoring-weight changes (param.rssync2026-11-18T16:00:39Z; favorite 0.708, click 0.274, share 2.118, cont_dwell 0.0075, cont_click_dwell 0.385, not_interested β53.07, video_open 0.0 all unchanged). (1) The value model gained a multiplicative mode (ValueModelMode="multiplicative",NotDwelledPowerExponent/Offset,MultiplicativeVmEngageBias):score = skip Γ engage Γ time, whereskip=(P(not_dwelled)+offset)^exponent,engage=Ξ£ weighted heads + bias, andtime=1+max(0, cont_dwellΓdwell); negative engage falls back to the additive floor, and the offset normalizer drops to the four negative behavioural heads. Ships off; both consumers ignore the cachedweighted_scorewhen it is on. (2) Class factors (OonEngagementWeightFactor,OonDwellTimeWeightFactor,InNetworkβ¦, all β1.0 = sentinel/off) split a candidate’s engagement and dwell terms for OON vs in-network, layered on the OON level. Off. (3) DPP sliding window (DppWindow=0,DppWindowHalfDistance,DppWindowSidLambda=0,DppGreedyOrder=false): when on, each pick is penalized against the previous N picks and the viewer’s 10-minute served history by0.5^(d/half_distance), plus a SID-overlap term served by a newSidClient. Off. (4) The ads blender default changedmulti_risk β slot_spread, and slot-spread now thins ad slots per user state and page-score tier (UserAdLoad; params empty by default). This does not change organic scores but shifts organic positions around the ads. (5) A teacher-score side effect ships ON (EnableTeacherScoreSideEffect=true): prod For-You requests write the served posts’ 21 action probabilities to a Redis teacher cache (TTL 3,000 s) keyed by prediction request id β a distillation/telemetry artifact, not a ranking input. (6) Phoenix now returns candidate ids only (return_candidate_tweet_id_only=trueon both Home and Gallery prediction calls), and production requests no longer retain the retrieved-candidate vector (retain_retrieved_candidatesreturnsis_debug_request) β a memory/logging change. (7) The post int64 feature block grows 17β22 with six DPA-SID level slots (dpaSidLevel0..5, slots 3 and 17β21; 256-code codebooks), so every ranking model’s input and parameter shapes change even thoughenable_dpa_sid_featstill ships false. (8) Retrieval:xrecsys_two_tower_home_onlynow trains its recall-dwelled head with click-only dwells removed (ld_label_exclude_clicks=true; requires a video-quality view or β₯10 s dwell plus a qualifying non-click action), and six new AI-video SID windows (ai_video_self/ai_video_detected, 24h/48h/7d, each excluding posts in anynsfw_videowindow) join rankall’s Sid pipeline (18β24 windows; Main still 19), fed by a newcontentDisclosure/C2PA/Grok-signature classifier inphoenixRankAllCandidateProcessor.strato. No training config selects the new windows yet andSidSourcestays off. (9) New model code:page_critic_only.py(2-layer critic, Gumbel-top-k decode, TD-Ξ» loss; optionalserve_decode="critic", default staysactor), along_term_author_profiledataset (plumbing only), recent-positive history value scaling (off), per-field ad-side-id embeddings (off), per-request CREAD decode parameters (off), and aRetrieveTopKUsersRPC (Rust host only, off). (10) Grox moderation: a new many-media PTOS flow chunks posts with >4 images (own or quoted) into 4-image batches when the post has β₯256 favs or a prior adult-content violation (and is not already NSFW); spam policy now distinguishes PromotesClicks / PromotesReplies / PromotesShares β only Shares writesSpamHighRecall, while Clicks/Replies write the newPromotes*labels with no spam label.is_pastedand the reply-spam flow are untouched. (11) Enforcement/VF: 12 new post-label page actions (NsfaHighPrecision,NsfaLimitedInventory,NsfwHighPrecision,SoftNsfwΓ 7d/30d/permanent) with post-label carry/dedup in abuse-enforcement; test-account paths (TestUserStatus, role/e-mail/CIDR policies) skip account-ending actions;follower_threshold: 0now means “stored off” (ceilingβ1) instead of literal zero;DO_NOT_AMPLIFYandMALICIOUS_URLaudiences changed from non-authors to non-followers (both rules live only on OON levels, so this is a semantic cleanup with no feed effect today); Like/Repost notifications also judge the retweet source, Quote notifications get their own rule set, media-tagged notifications gain spam/risky-reply drops, and retweet verdict merging now picks the higher-severity withholding. (12) Safety-label cache: not-found TTL 10β60 min, two-step 30/90 s refresh β one 60 s step, and stale labels are served on any remote failure inside a request deadline. No filter was added or removed (still 21, same order); no candidate source was added or removed (a new Jetfuel frame source ships off); no weight moved. Sections 1.AβB, 1.E, 3, 5.1, 5.3β5.4, 9, 12.4β12.5 carry the details. None of this overturns the analysis.
ARCHIVED DOC ACCURACY NOTE (July 2026 revision, prior release): Earlier sections described
weighted_scorer.rs,oon_scorer.rs, andauthor_diversity_scorer.rsas the active scoring pipeline. These files were dead code in that release (removed fromhome-mixer/scorers/mod.rs). The active scorer chain wasPhoenixScorer β RankingScorer β VMRanker.RankingScorermerged weighted scoring + author diversity + OON weighting into one scorer, andVMRanker(a proprietary gRPC value-model ranker) ran after it and could override the final score entirely. In the Aug 2026 release,weighted_scorer.rs,oon_scorer.rs, andauthor_diversity_scorer.rsare deleted outright,vm-ranker/is open (but head-model-redacted), and RankingScorer gained new scoring modes.
What’s New in the Aug 13, 2026 Release (Executive Summary)#
| New system | What it is | Relevance to reach |
|---|---|---|
home-mixer/params/ (open) | All weight values + structural constants in code | Removes the #1 limitation of prior analysis (exact weights known) |
visibility-filtering/ | The post-ranking content gate: drop / tombstone / appealable / soft-intervention / label-interstitial / blur / limited-engagement, across 9 safety levels | Full rule catalogue now auditable (Section 12.5) |
vm-ranker/ | Reranking service (DPP diversity; optional server-side value model, user-debias, carousel model) | DPP can zero-out candidates; diversity layer in final ranking (Section 5.3) |
user-cred-v2/ | Daily PageRank “credibility mass” over the follow+engagement graph | Reputation score now exists; gates enforcement & spam treatment (Section 12) |
abuse-enforcement-service/ | Rule-based enforcement (suspend/label/challenge) from model scores | Account-level actions with 500K/day caps (Section 12.4) |
scarecrow/ + botmaker/ | BotMaker rule engine; spam-label store (TTL 26h), rate limits | Real-time labeling of posts (SPAM etc.) (Section 12.5) |
phoenix/ (production code) | Training code, real configs, synthetic data; no checkpoint | Model architecture now fully known (Section 5.1) |
phoenix-rankall/ | Kafkaβparquet retrieval-index maintenance (engagement-threshold windows) | Which posts enter candidate pools (Section 5.4) |
simclusters/ source in home-mixer | 3rd OON source (cluster similarity, 20M users/145K clusters) | Additional OON distribution layer (Section 1.D) |
| SID (semantic IDs) | 6Γ256 residual-quantized codes from multimodal embeddings | Post representation for retrieval & ranking (Section 5.1) |
| Scoring: new heads/modes | video_open, open_link, quoted_vqv, post_unexplored, not_dwelled, dwell_time (dwell prob 0.05; Oct: cont_click_dwell 0.4 active; Nov: video_openβ0.0, post_video (renamed from react_with_video) + grok_analyze at 0.0), MultiplierPreOffset gate, author cold-start boost, bidirectional-follow boost (dwell-regret DELETED Oct) | New explicit levers (Sections 1, 5.3) |
under-the-hood/ | Transparency report (labels per account/post, 365d age + 10 posts/mo eligibility) | External accountability surface |
Aug 14, 2026 addendum (commit c65aa17): official weights-interpretation comments (weights Γ predicted probabilities, not counts) Β· Brazil2026ElectionFilter (665 hardcoded IDs, follower-exempt, no feature switch) Β· 3 new serving-side bool features (both follow directions + 14-day stale-post flag) Β· grox blast-radius thresholds 15K β 30K Β· cold-start Thompson-sampling variant (off) Β· SID video_14day window Β· bdsm enforcement-note structure. See Section 1.B, 3, 5.1, 5.3, 5.4, 9, 11, 12.3.
Sept 2, 2026 addendum (commit 85ac72a, 13 commits Aug 17βSept 2): weight moves (video_open 0.07, vqv 0.0, dwell prob 0.05) Β· MPN removed (MultiplierPreOffset, off) Β· VMRanker request slimmed to ids + score Β· grox gates 30K β 120K Β· Brazil list 665 β 2,388 IDs Β· rankall Main 17 / SID 15 windows Β· Muon training + stale-post/follow/count features enabled in home_direct_packed Β· VF rules refactor (behavior-preserving) with dynamic NSFW country list Β· new panda-reports suspend rule + generic-action path Β· bdsm count gate redacted. See the Sept addendum above and Sections 1.AβB, 1.G, 2, 3, 5.1, 5.3β5.4, 6, 9β13.
Oct 13, 2026 addendum (commit e1a0375, 28 commits Sept 3βOct 13): scorer moved to xai-value-model/ (dwell-regret deleted; server-side value-model compute optional) Β· weights to favorite 0.6 / video_open 0.02 / click 0.3 / post_unexplored 0.015 / cont_dwell 0.006 / cont_click_dwell 0.4 / not_interested β47.52 Β· cold-start rewritten (50K/200/2h/97%, Thompson on) Β· 19th filter (FavHoldout, off) + AncestorVideo/VideoCarousel post-selection + PopularPosts (on)/SidSource (off) sources Β· VF tombstone/Article/age-prompt/limited-engagement/retweet-merge/level-82 expansion Β· Brazil 2,800 Β· grox 250K split + multi-step reply-spam + v8-search + video lanes Β· rankall Main 19 + reply windows Β· retrieval ld head + sid-retrieval service + reply two-tower v2 Β· enforcement guardrails (action_control/off_switch/overturn_hold + ledger). See the Oct addendum above and Sections 1.AβB, 1.DβE, 2, 3, 5.1, 5.3β5.4, 6, 7, 9β13.
Nov 7, 2026 addendum (commit 3b6919f, 19 commits Oct 14βNov 7): weights to favorite 0.708 / click 0.274 / share 2.118 / cont_dwell 0.0075 / cont_click_dwell 0.385 / not_interested β53.07 and video_open 0.0 Β· post_video + grok_analyze heads (0.0) Β· Brazil filter deleted Β· 21 filters (ZeroFavCodivert, NewUserSoftNsfw, FairShareRanking ON β proportional per-source budget) Β· 3 new scorers (ImmersiveVideo, PageDecodeRerank, PageRlExploration) Β· EnableHomeExcursionScores on + DENSE_WITH_LONG_DWELL_HOME_EXCURSION aggregation Β· carousel rebuilt (embedding/media dedupe, brand-safety, immersive, CTR model) Β· VMRanker user-debias + carousel model + top_heads + no DPP on topic requests Β· VF action-model rewrite (9 levels, 7 action kinds, FOSNR follower split, reaction-video propagation, notifications, client switches, DDG experiments) + safety-label-source/ Β· Phoenix time-gap + home-excursion history features, candidate-only layers, FP8, view head; gen-recs/SID-retrieval deleted; page-RL models Β· enforcement head guards/confidence/backfill + bb1_* rules Β· Under-the-Hood top-3 score contributions. See the Nov addendum above and Sections 1.AβB, 1.DβE, 2β3, 5.1, 5.3β5.4, 6β13.
Nov 19, 2026 addendum (commit b551517, 8 commits Nov 10β19): no weight changes Β· multiplicative value model (off) + per-class engagement/dwell factors (off) + DPP sliding window with SID history (off) Β· ads blender default multi_riskβslot_spread Β· teacher-score side effect ON Β· Phoenix returns ids-only and drops retrieved-candidate retention Β· post int64 width 17β22 (DPA-SID slots) Β· ld label excludes click-only dwells for the home-only two-tower Β· six AI-video SID windows (Sid 18β24) Β· page_critic_only + long-term author profile + recent-positive scaling (all off) Β· many-media PTOS + Promotes* spam labels Β· 12 post-label page actions, test-account skips, follower-threshold semantics Β· DO_NOT_AMPLIFY/MALICIOUS_URL audiences cleaned up (OON-only; no effect today) Β· notification source/quote passes Β· safety-label cache 60-min negative TTL + stale-on-failure. Filters (21), sources, and weights are unchanged. See the Nov 19 addendum above and Sections 1.AβB, 1.E, 3, 5.1, 5.3β5.4, 9, 12.4β12.5.
System Architecture Overview#
The X For You feed algorithm is a recommendation system combining:
- In-Network Content (Thunder): posts from accounts the viewer follows, held in-memory with 48h retention
- Out-of-Network Content: Phoenix retrieval (two-tower transformer, SID-based) + SimClusters (cluster-similarity ANN) + PhoenixTopics/PhoenixMoE/CachedPosts
- Ranking (Phoenix transformer): predicts per-action probabilities, blended by
RankingScorerweights, then DPP-diversified byVMRanker - Visibility Filtering (VF): a separate service that decides ALLOW / INTERSTITIAL / DROP per post Γ viewer (now open source)
File Location: README.md β system overview (Aug 13, 2026); see Section 11 for the truthfulness audit.
Two pipelines compose the feed:
- Post Pipeline (
home-mixer/candidate_pipeline/phoenix_candidate_pipeline.rs): find β hydrate β filter β score β select β VF. - Blending Pipeline (
for_you_candidate_pipeline.rs): adds ads, Who to Follow, prompts, push-to-home at fixed positions (WHO_TO_FOLLOW_POSITION = 6,PROMPTS_POSITION = 0), thenBlenderSelectorblends ads β defaultslot_spreadsince Nov 19 (wasmulti_risk; slot-spacing with per-user ad load, risk classes, BSR-aware slotting, brand-safety verdict V2 on) β andAdAdjacentServedFilterre-sorts for ad adjacency (never drops organic posts; drops the ad if no swap candidate).
Servers (home-mixer/lib.rs): ForYouFeedService, FollowingFeedService, RankedFollowingFeedService, ScoredPostsService, PhoenixScoresService (+ reverse-chron following pipeline).
1. Post Reach and Visibility Determination#
A. Scoring Pipeline (current)#
Location: xai-value-model/scoring.rs + xai-value-model/weights.rs (shared crate, NEW Oct) + home-mixer/scorers/value_model.rs + scorers/phoenix_scorer.rs + scorers/vm_ranker.rs
Scorer chain (phoenix_candidate_pipeline.rs, scorers vec): PhoenixScorer β ImmersiveVideoScorer β VMRanker β PageDecodeRerankScorer β PageRlExplorationScorer β TopKScoreSelector (50). Oct: RankingScorer (1,571 lines) is deleted β the weighted-score + diversity + OON math moved into the shared xai-value-model/ crate; home-mixer computes a local weighted_score fallback (scorers/value_model.rs, diversity/OON hardcoded off locally) and VMRanker applies cold-start, then sends a full RankRequest (scorers/vm_ranker_request.rs: viewer + per-candidate author/in-network/reply/mutual flags, cold_start_lift_to_rank, phoenix_scores, weighted_score, SIDs, experiment payload) with value_model_id="dpp" + compute_value_model flag; the server (vm-ranker/scoring/value_model.rs) now owns all scorer params (OON 0.75/0.5/0.00001, diversity 0.5/0.25, MultiplierPreOffset, OonRescore) and can compute the value model itself (ComputeValueModel, ships off) with per-head weight perturbation (weights.perturbed(sigma,salt,user_id), md5 salt:user:head). Dwell-regret modes and value_model_gate.rs are deleted. Nov: two new post-scoring stages ship off β PageDecodeRerankScorer sends page-decode params in the Phoenix request and rewrites candidate order from the model’s slot picks (EnablePageDecodeRerank=false), and PageRlExplorationScorer samples a PlackettβLuce order over the top-64 scores at temperature 0.2 (EnablePageRlExploration=false); ImmersiveVideoScorer runs a second Phoenix pass with the immersive aggregation for carousel-eligible videos only.
1. PhoenixScorer β calls the Phoenix gRPC ranker (PredictNextActionsRequest, up to PHOENIX_CLIENT_MAX_CANDIDATES = 2800 β phoenix_request.rs:11), returns logits for 60 action heads + 8 continuous heads (see Section 5.1). Scores are applied to the original tweet id (candidate_scores(&get_original_tweet_id()), phoenix_scorer.rs:109; Sept: backbone scores + slate context stamped the same way).
2. Value model β compute_weighted_score (xai-value-model/scoring.rs:69-140; Oct: same formula, new home):
terms = [
favoriteΓ0.708, replyΓreply_weight_for(c), retweetΓ1.0,
photo_expandΓ0.05, video_openΓ0.0, clickΓ0.274, open_linkΓ0.2,
profile_clickΓ0.0, vqvΓvqv_weight, shareΓ2.118, share_via_dmΓ5.0,
share_via_copy_linkΓ20.0, dwellΓdwell_weight_for(c), quoteΓ5.0,
quoted_clickΓ0.05, quoted_vqvΓquoted_vqv_weight (0.0; duration-checkable), dwellΓdwell_weight_for(c) (**0.05** + mutual-follow originals), dwell_timeΓ0.0075,
click_dwell_timeΓ0.385, active_secs_5m_residualΓ0.0, follow_authorΓ4.0,
post_unexploredΓ0.015 (additive, in-network only),
post_videoΓ0.0 (renamed), grok_analyzeΓ0.0 (NEW head, zero),
video_continuation/user_video_continuation/profile_visit_secs (Phoenix excursion heads, now produced because EnableHomeExcursionScores=true; weights still 0.0),
not_interestedΓβ53.07, block_authorΓβ31.2, mute_authorΓβ58.8,
reportΓβ234.0, not_dwelledΓβ0.02,
]
pos = Ξ£ positive terms, neg = βΞ£ negative terms
offset_score(posβneg): combined < 0 β (combined + negative_sum)/total_sum Γ 0.001
else β combined + 0.001 (NEGATIVE_SCORES_OFFSET)
3. Post-scoring adjustments (xai-value-model/scoring.rs:183-327; Oct: moved server-side, same math):
- Author diversity:
multiplier = (1β0.25)Β·0.5^k + 0.25for the k-th post of an author in the pool (decay 0.5, floor 0.25, enabled β now invm-ranker/params.rs; home-mixer local fallback hardcodes diversity off). - In-network reply/retweet deboost is gated, not always-on. Replies and reposts from accounts the viewer follows get the 0.75 OON factor only when
EnableOonRescoreForInNetworkRepliesRetweetsis on (nowvm-ranker/params.rs:190, ships true) β the README’s blanket “replies and reposts from accounts the viewer does follow are discounted” overstates it. Still unknown whether prod enables it. Oct: OON replies from Phoenix retrieval are additionally gated by newEnablePhoenixOonReplies(ships off) +PhoenixOonRepliesRootOnly. - New-user OON factor: viewers with account age <
NewUserAgeThresholdSecsand following β₯NEW_USER_MIN_FOLLOWING = 5getNEW_USER_OON_WEIGHT_FACTOR = 0.00001(vs 0.75) β a 75,000Γ stronger in-network bias. DefaultNewUserAgeThresholdSecs = 0(off unless feature-switched; Oct: params moved tovm-ranker/params.rs:221-227). MultiplierPreOffset(falseby default; Oct: moved tovm-ranker/params.rs): when on, applies diversityΓOON multipliers only to the positive net pre-offset (mΒ·netif net β₯ 0, else net), then cold-start; the default path runs cold-start β diversity β OON.- Author cold-start boost (
author_cold_start.rs): one eligible post is lifted to slot 15 (see Section 5.3 β Oct rewrite: cap 50K followers / 200 views / 2h age / top-97%, Thompson on).
4. VMRanker (vm_ranker.rs + NEW scoring/value_model.rs, scoring/dpp_model.rs, params.rs, vm_ranker_request.rs β Oct: the scorer moved here) β home-mixer computes a local weighted_score fallback, applies AuthorColdStart::apply_with_decisions, and sends a full RankRequest (viewer id + context, per-candidate tweet/retweeted ids, author/in-network/reply/mutual flags, cold_start_lift_to_rank, phoenix scores + weighted_score, SIDs, hard value_model_id="dpp", compute_value_model flag, DPP params theta=0.65, max_selected_rank=150, seed-tweet pinning) to the vm-ranker/ service; the server resolves params via RankingConfig, optionally recomputes the value model itself (ComputeValueModel, ships off), then runs DPP (dpp.rs: seed pinned with q=1.0, greedy-MAP, non-selected β 0.0) and returns per-candidate {score, weighted_score} (fallback: local score). A candidate DPP didn’t select still scores 0.0 β i.e., DPP acts as a hard diversity cap. (Plus RerankerHeadTag=0, backbone scores, served slate context with SID/recon fields, exact_k/exact_gap, debias/pacing payloads.) Nov additions: (a) topic_request now disables DPP (dpp_enabled = DppEnabled && (!topic_request || TopicDppEnabled), TopicDppEnabled=false) β topic feeds are ordered by raw score; (b) an optional user-debias stage (vm-ranker/scoring/debias.rs + user_debias_store.rs): an offline-trained per-user/per-head logit model reads the viewer’s action sequence, computes debiased_probability(p, user_logit, global_logit, clip=2.0), and blends (1βΞ±)Β·p + Ξ±Β·debiased per head at Ξ±=0/off (heads: fav, reply, quote, retweet, dwelled, not_dwelled, vqv, photo_expand, click, profile_click, share, open_link, video_open β the default list also names bookmark, which has no score field yet and is skipped); artifacts are hot-reloaded and rejected when older than 3,600 s; (c) a GetCarouselModel RPC serves a linear CTR model (lr_v8_search_video_1_2026-11-03_no_high_risk) for the video carousel; (d) when include_top_heads is set (Under-the-Hood requests only) the response carries each candidate’s top-3 weighted-score contributions (head, contribution, share β₯10%).
Nov 19 additions (all off by default): the shared value model can run in a multiplicative mode (ValueModelMode="multiplicative": skip Γ engage Γ time, with the score floor still additive) and can apply separate engagement/dwell factors for OON vs in-network candidates (four β¦WeightFactor params, β1.0 sentinel); the VM ranker’s DPP can use a sliding window over the last N picks plus the viewer’s 10-minute served history with a SID-overlap term (DppWindow, DppWindowHalfDistance, DppWindowSidLambda, plus GreedyOrder), served by a new SidClient; and PageDecodeRerank can consume the same multiplicative parameters. The live changes in this layer are outside the per-post math: the ads blender default is now slot_spread (the new per-user-state/page-tier thinning knobs ship empty; the algorithm change is live), the Phoenix calls request candidate ids only, and production requests no longer retain the fetched-candidate vector (debug requests do). No scorer stage was added to or removed from the chain.
B. Weights β NOW KNOWN (home-mixer/params/param.rs; Nov 2026 values, sync Nov 18)#
Nov 2026 moves (param.rs, sync 2026-11-06): favorite 0.6β0.708, click 0.3β0.274, share 2.0β2.118, cont_dwell 0.006β0.0075, cont_click_dwell 0.4β0.385, not_interested β47.52ββ53.07, and video_open 0.02β0.0 (the head no longer scores; it survives only through the implicit discovery signal and the carousel ranker). react_with_video was renamed post_video (still 0.0) and a new grok_analyze head (0.0) was added. The moves are small and not uniform (like Γ1.18, share Γ1.06, dwell Γ1.25, click Γ0.91, click-dwell Γ0.96, not-interested Γ1.12) β directionally the same objective as Oct, with video-open fully removed. (Sept/Oct moves retained: vqv 0.0, dwell prob 0.05, cont_click_dwell introduced at 0.4, post_unexplored 0.015.) Dwell-regret/MultiplicativePostUnexplored/click-dwell-penalty params remain deleted. Note: vm-ranker/params.rs (server copy) still ships the Aug-era favorite 0.5 / video_open 0.07 / cont_dwell 0.004 / post_unexplored 0.02 / not_interested β47.52 β the two crates have diverged; which copy the server resolves at runtime is config-dependent.
Nov 19: no weight moved. The 2026-11-18 sync changes only non-weight params (see Section 9); every value in the table below is current for both the Nov 7 and Nov 19 trees.
| Weight | Default | Weight | Default |
|---|---|---|---|
| favorite | 0.708 | not_interested | β53.07 |
| reply | 5.0 (+15.0 bidirectional boost) | block_author | β31.2 |
| retweet | 1.0 | mute_author | β58.8 |
| photo_expand | 0.05 | report | β234.0 |
| video_open | 0.0 | not_dwelled | β0.02 |
| click | 0.274 | follow_author | 4.0 |
| open_link | 0.2 | post_unexplored | 0.015 |
| profile_click | 0.0 | cont_dwell_time | 0.0075 |
| vqv | 0.0 | cont_click_dwell_time | 0.385 |
| share | 2.118 | active_secs_5m_residual | 0.0 |
| share_via_dm | 5.0 | dwell (prob head) | 0.05 |
| share_via_copy_link | 20.0 | quote | 5.0 |
| quoted_click | 0.05 | quoted_vqv | 0.0 |
| post_video | 0.0 (renamed) | grok_analyze | 0.0 (new) |
Interpretation: the strongest positive levers are shares via copy link (20), reply (5, +15 if mutual-follow original), quote/share-via-DM (5), follow-author (4.0), share (2.118). Likes are now 0.708 β ~7Γ weaker than a reply, down from the Oct 10Γ. The dominant negatives are report (β234), mute (β58.8), not-interested (β53.07), block (β31.2). Note: profile_click, active_secs_5m_residual, quoted_vqv, vqv, video_open, post_video, and grok_analyze are zero in defaults. Continuous dwell_time (0.0075) is sigmoid-activated and de-normalized by norm-scale 30, so predicted dwell is capped at ~30 s β max contribution β 0.225 (was β0.18) β still under a third of a like; the click-dwell term (0.385) remains the large continuous lever (per-second value depends on the log1p/60 normalization in recsys_feature_prep.py). The dwell probability head stays 0.05.
Aug 14 official clarification (param.rs, xai-value-model/scoring.rs, README): the weights scale predicted probabilities (or predicted continuous values), not raw engagement counts β so weight ratios are not count equivalences, and “one report cancels 468 likes” is explicitly called an incorrect statement. The report head is β234 because the baseline probability of a report is >1,000Γ lower than a like β the weight exists to give a rare action enough influence to matter. The same logic applies symmetrically to the positive weights (platform framing echoed by the VP of Product on Aug 15, 2026): copy-link share (20.0) and DM share (5.0) are rare actions, so their weights are inflated relative to the common like (0.708); every term’s contribution is weight Γ P(action), so “reply is ~7Γ a like” is a weight comparison, not a value equivalence β the ranking of levers depends on the (unpublished, per-viewer) predicted probabilities. The same comments argue that mass block/report campaigns cannot “significantly suppress reach” because (1) predictions are personalized β reports from bad actors “primarily affect recommendations for users who are similar to the bad actors,” not everyone’s ranking of the post; and (2) only actions on posts served in the Home Timeline count toward ranking (direct navigation, e.g. via groupchat coordination, has no ranking impact). This reframes the negative-weight math in Section 14 of the companion account-analysis
: the β234 is a bound on a per-viewer prediction, not a subtraction from a global score.
C. Dwell-Regret Scoring Mode (DELETED Oct 2026)#
Oct: ValueModelMode, all DwellRegret* params, and scorers/value_model_gate.rs (537 lines) are deleted β grep dwell_regret in the Oct tree returns nothing. The only scoring mode is the weighted sum in xai-value-model/scoring.rs. The archived description is preserved below for history:
ValueModelMode = "weighted" | "dwell_regret_sigmoid" | "gated_dwell_regret". In dwell-regret modes, RankingScorer replaced the weighted sum with (ranking_scorer.rs:535-600, Sept tree):
score = dwell_time Γ 2Β·sigmoid(Ξ£ Ξ±Β·(P_i/mean_i β 1) / T) Γ exp(neg.min(0)/T)
i.e., dwell time modulated by a candidate’s engagement probabilities relative to the candidate pool’s mean, with negative-action exponential damping (T = 10.0, dwell floor 1.0). A logistic gate model (value_model_gate.rs, 19 viewer features: seq length, action counts over 28d/7d/1d, active days, account age, followers/following; threshold β0.634264, bias 1.033918) decides which viewers get it (“gated” mode).
D. Candidate Sources#
phoenix_candidate_pipeline.rs (sources vec) β nine sources (Oct: +2; Nov: PopularPosts gets its own served type):
- Thunder (in-network, up to 1200) β @grok/@gork/@products posts are blacklisted server-side (thunder_service.rs:29-36); Nov: the
ThunderClusterIdparam and cluster resolution are removed (the client picks a channel itself) - PopularPostsSource (on β
EnablePopularPostsSource=true, 500 results; shared most-followed-active list viapopular_posts.rs/popular_authors.rs+ background jobs; Nov: served asForYouPopularPostsrather than being folded intoForYouPhoenixRetrieval) - TweetMixer (800; source ships disabled β
EnableTweetMixerSource=false) - Simclusters (800 max,
LOG_FAV_LONGEST_L2_EMBEDDING_TWEETquery βLOG_FAV_BASED_TWEETcandidates,MODEL_20M_145K_2020, min score 0.5, β€48h old, driven by explicit+implicit engagement signals; Oct: 48h cap now a param, shared ANN cache client, per-seed parallelism, retrieval-source provenance) - Phoenix retrieval (1000; Oct: new
PhoenixColdStartMaxResults=200,PhoenixRetrievalExcludeSeenPosts=false, cold/zero-fav dataset mapping, seen+served+impression/bloom exclusions; Nov: the retrieval request carries an optionalvalue_model_weightsmap, include/exclude author ids, and the zero-fav dataset maps toForYouPhoenixRetrievalZeroFav) - PhoenixTopics (1000, reuses
PhoenixMaxResults) - PhoenixMOE (ships disabled β
PhoenixMOEMaxResults=0; Oct: clusterExperiment1FouβExperiment2Memy04, cold-start cap 200, kill-switch decider, gated on SID source off) - SidSource (NEW Oct, off β
EnableSidSource=false; SID-prefix retrieval overseed_post_idsfrom post signals, 800 max / 50 seeds / 100 per seed / min prefix depth 3, served by the newxai-recsys-sid-retrievalmicroservice) - CachedPosts (no cap; cache size
MaxPostsToCache=750)
Nov: the pool is no longer first-come-first-served. FairShareRankingFilter (ON by default) runs last among the pre-scoring filters and trims the combined pool to PhoenixScoresResultSize (2,800) by proportional fair share per ServedType (largest-remainder allocation). Because PhoenixScorer then takes the first 2,800 candidates in pool order, the filter is what stops the early sources (Thunder 1,200 + PopularPosts 500 + TweetMixer 800 + Simclusters 800 β 3,300 alone) from pushing Phoenix retrieval/topics/MoE/SID/Cached out of the scored set. It is skipped for cached-post requests and when the Phoenix kill switch is on.
Nov 19 adds no candidate source. The frame/blending layer gained a Jetfuel /grokbot/suggested_tasks card at slot 3 (off), a direct Jetfuel frame source (off), and an ads-only timeline for an explicit viewer list (off); the default ads blender changed to slot_spread (Section 1.A).
E. VF (Visibility Filtering) β now open source#
Final gate, per post Γ viewer: nine safety levels (FilterAll, TimelineHome, TimelineHomeRecommendations, TimelineHomeHydration, TimelineHomePromotedHydration, ImmersiveExpandedRecommendations, QuotedTweetRules, NotificationsFollow, NotificationsWriterV2) and seven action kinds (models/verdict.rs, treatment.rs, rules/levels.rs + action_kind.rs): Drop and Tombstone withhold; Appealable and SoftIntervention are freedom-of-speech-not-reach notices; LabelInterstitial, MediaRestriction (blur), and LimitedEngagement fill show-with-treatment slots (first writer wins per slot, with the other slots still available). The rule engine was rewritten in Nov (rules/{catalog,defaults,engine,expansion,action_kind}.rs + a typed rule_spec/ with feature leaves); the verdict is now a fold over user rules then post rules, with explicit precedence (drop beats tombstone beats interstitial/limit). In-network posts are evaluated on the TimelineHome rule set; OON posts on TimelineHomeRecommendations (base rules + OON-only drops that can only drop); level-82 TimelineHomeHydration carried only by EvaluateTweets; promoted ads have their own hydration level; immersive gallery requests get ImmersiveExpandedRecommendations; notifications have two dedicated levels (follows; writer-v2 mentions/replies/media-tags). New Oct behavior retained: Article lifecycle drops, tombstones (age-verification / app-update / local-regulations), age-prompt blurs, limited-engagement prompts (14 LimitedEngagementReason variants incl. LocalTweet, StaleTweet, RootAuthorBlockedViewer), retweetβsource verdict merging, and split NSFW-gating/age-verification/tombstone/local-regulations country lists. Nov behavior (Section 12.5): FOSNR actions are split by follower status; reaction videos inherit a blur from the media they react to; a SOFT_NSFW label can drop OON posts for brand-new viewers; interstitials now distinguish public-interest label notices from emergency copy; VF is gated by client app/version switches and DDG experiments. Golden corpus is now 25 files. Full rule catalogue in Section 12.5. Home-mixer sends candidates in 50-chunk batches (XAI client default 400 ms timeout; fails closed: missing results = UnspecifiedReason drop).
Nov 19: levels and action kinds are unchanged (9 / 7). The only feed-level precedence change is that merged retweet verdicts now pick the higher-severity withholding rather than a drop-vs-tombstone special case. The DO_NOT_AMPLIFY and MALICIOUS_URL audience narrowing (non-authors β non-followers) is semantic: both rules run only on OON levels, where every viewer is already a non-follower. The rest is notification plumbing (Like/Repost also judge the retweet source; Quote gets its own rule set; media tags gain spam/risky-reply drops) and a response-surface change: FilterTweets no longer returns the per-post safety_labels map.
F. Post-Type Handling: Native Articles Are Not a Distinct Type#
X’s native Articles (long-form posts) are not a distinct type in the ranking/retrieval path β verified absence at every scoring layer (the one exception is a visibility lifecycle gate, added Oct 2026):
- Model input:
TweetInfocarries no article flag;TweetBoolFeaturesis onlyis_for_you_page / is_promoted_tweet / is_reply / is_retweet / is_quote / has_media(recsys.proto:1105-1112). No article feature exists in the training code orfeature_config.py. - Scoring: no article weight, no article head, no article adjustment in
RankingScorer. In thunder, an article counts toward the original bucket (MAX_ORIGINAL_POSTS_PER_AUTHOR = 50; the per-author caps only distinguish replies 30 / originals 50 / videos 100 β thunder/config.rs:5-7). - Eligibility: no article filter; articles obey the same
MAX_POST_AGE = 48h; no article retrieval window exists in phoenix-rankall (evergreen pools are video-only), and no article engagement signal (the implicit-signal hydrator covers photo-expand/VQV/video only). - Analytics: the
TweetTypeMetricsHydratorbitset hasRETWEET/REPLY/SUBSCRIPTION_POST/VIDEO/β¦but no ARTICLE bit (tweet_type_metrics_hydrator.rs:22-132). - VF: Oct: Articles now have a VF rule β the TES
get_is_articlestub is replaced by a realarticle_client.rs(lifecycle + media/cover hydration) andARTICLE_TWEET_CONTENT_DROP_UNSPECIFIED(unpublished article β DROP on level 82; golden-corpusarticle.rsmatrix covers published/unpublished Γ NSFW body/cover Γ viewer age/opt-in). No article scoring signal still β the type signal exists only as a VF eligibility gate. - Where articles do appear β content understanding only: grox hydrates article metadata (title, media, cover) and feeds it into the multimodal-embedding renderers (
grox/flows/mm_emb/renderer.py:65-70, 88-89,renderer_v82.py:317-320,phoenix/reference/mm_encoder.py:143-166) and tells its LLM classifiers the post is an article ("[Article Post]β¦",grox/core/lm/post.py:95; still no[Video Post]label anywhere in grox). So an article is judged by its content β title + body media + cover enter the post’s embedding and therefore its SID β exactly like any other post’s text/media, just with no type signal.
Empirical check (sample-1; collector flag IsArticle = bit 2, PF = {IsQuote:1, IsArticle:2, HasPhoto:4, HasVideo:8, HasURL:16, HasMention:32}): 37 of 10,000 posts are articles, 23 of them from one account (ilex_ulmus). Pooled medians: views/follower 25.95% vs 9.65% for all other posts; engagement/view 0.86% vs 1.67%; median views 3,935 vs 5,051. The delivery advantage is real but rests on a tiny, one-account-skewed sample, and the dominant article poster is a small account β the cold-start boost alone can explain most of the gap. “Longer dwell” is an inference (dwell isn’t in the dataset), not a measured signal (Section 13.5).
G. The Role of Followers in Distribution: the unconditional seed channel#
Follow status is the only thing that makes distribution unconditionally offered: a post is entered into every follower’s candidate pool without passing any engagement or index gate. Whether it is actually shown is still a full per-viewer decision (below). Everything else in this section is about the gated channels; this subsection is about the one channel with no gate:
The follower channel (Thunder) β unconditional candidacy, per-request, but still per-viewer prediction. Your post enters every follower’s candidate pool the moment they request the feed: in-memory 48h retention, no engagement threshold, no index membership, no discovery prediction β subject only to the MAX_POST_AGE = 48h window and per-author caps (1200 posts total / 50 originals per author). Which followers actually see it is still decided by the full per-viewer pipeline: the seen/served filters (a follower who was already shown it β or was served it in the last 10 minutes β is excluded; previously_seen_posts_filter.rs, previously_served_posts_filter.rs), muted keywords and blocks (both directions), the per-viewer Phoenix prediction Γ weights, author-diversity decay, the DPP diversity step (it can be zeroed if the follower’s feed already picked similar content), the TopK-50 selection, and VF for that viewer (follower-exempt labels don’t apply to followers; exclusive content still drops unless subscribed). So the initial audience is not “whoever is online” β it is the subset of followers who (a) request the feed within 48h, (b) haven’t been shown it yet, (c) are not filtered, and (d) have a per-viewer predicted-engagement score high enough to survive the top-50 cut against ~1,000 candidates β in practice: currently-active followers with a recent engagement history with you (the model predicts them best). Passive followers, and followers who already saw the post in an earlier request, are ranked out or filtered out.
The out-of-network channels are all gated (phoenix-rankall-strato, eventProcessing.strato):
- Admissions (fav-only; videos excepted): the
1fav,32fav,1fav_topic,1fav_reply,32fav_replywindows are built when a favorite event fires (phoenixRankAllCandidateProcessor.strato) β replies, reposts, quotes, and views never open an index window (20 favs + 100 replies does not open32fav_1day) β and the training pipeline’s HOME retrieval corpus is literally the 1-fav pool (post_sid_v8_256x6_snapshots/1fav_1day.parquet, retrieval_dataset.py). A 0-fav post is absent from the primary OON index. Membership is eligibility, not serving: an indexed post still has to win each viewer’s top-50 against ~1,000 candidates with the 0.75 OON discount. - The 0-fav OON entries are the fresh window, the video creation-time windows, and the small-account tail:
post_creation_1day(24h retention, no favs required β config/mod.rs:142) is built from the post-creation event, which arrives with a 2-minute delay via the content-understanding topic (postCreationEventForwarder.strato:17); videos β₯10s enter video windows at creation with 0 favs (buildVideoIndex on the PostCreation event); authors under 1,000 followers get theSidTailpool at 0 favs. All three are weak candidacy only (content-only signals, OON discount applies). Index admission also consults VF atTimelineHomeRecommendations(shouldDropPostByVF, eventProcessing.strato:246-265) β posts VF would drop for non-followers never enter the OON index at all. - Maintenance reprocessing (not admissions) β power-of-two thresholds with a 1-minute cooldown: the first favorite triggers index processing immediately (the uncached path has no time throttle β only age <49h and fav count <2Β³β°; favoriteEventProcessor.strato:136-148). Each later favorite re-triggers processing only when all five checks in the cached path pass (favoriteEventProcessor.strato:122-134, eventProcessing.strato:42-60): the fav count has reached the next power-of-two threshold above the last processed count (
nextPowerOfTwoβ 2, then 4, then 8, then 16, then 32; favoriteEventProcessor.strato:14-21), β₯1 minute has passed since the last processing round, <48h have elapsed since the first round (maxTimeSinceFirstUpdate), the post is <49h old (maxTweetAge), and the fav count is still below 2Β³β° (maxEngagements). Consequences: the thresholds are relative, so triggers land at approximately the 2nd, 4th, 8th, 16th and 32nd favorites (the intermediate rounds merely refresh the index); a second favorite within the same minute as the first is coalesced (no extra processing β but also no need: the first favorite already built1fav_1day); favorites that don’t push the count to a power of two (the 3rd, 5thβ7th, 9thβ15th, 17thβ31st) cause no re-processing at all; and the first processing round that observes β₯32 favorites opens32fav_1day(32 is a power of two, so the 32nd favorite triggers it on schedule β or the next round after it, if the 1-minute cooldown delays it). Crucially, these triggers never expand any audience pool: favorites play no role in the in-network channel at all (thunder serves followers’ candidates from creation, fav-independent), and no OON window admits the post between 1 and 32 favs β the 2/4/8/16 rounds are index maintenance (metadata/embedding retries), not admissions. Audience growth with accumulating engagement of any type (favs, replies, reposts, quotes, views) runs through the model’s live engagement-count features (60-second cache, bucketed + log2-MLP β Section 5.1), which raise predicted engagement for every viewer in later requests: the momentum loop. The first two days decide a post’s OON index membership.
The emergent “followers-first” pattern. The code never restricts a new post to followers β but the initial audience is overwhelmingly followers, and it is structural, not accidental: (1) thunder enters the post into every follower’s pool instantly with no gate; (2) the main OON pool requires β₯1 fav, and the 0-fav fresh window is 2 minutes delayed and secondary (a content-only candidate); (3) even when proposed OON, a 0-engagement post must outscore in-network candidates while carrying the 0.75 OON discount. The typical lifecycle is therefore: followers see it first β their favorites open the OON pools (the first fav triggers the 1fav_1day build immediately β pipeline latency, not a 1-minute wait; the 32nd fav opens 32fav_1day) β discovery expands β or doesn’t, if the seed audience doesn’t engage. The empirical signature: dawn_com’s 0.07% V/F is follower-only delivery with zero OON lift; the >100% V/F accounts (celestepoasts, atmoio) are the mirror β OON dominated because their posts cleared the gates. The user-experience intuition that “new posts are only shown to your followers” is directionally right as a first-phase description, wrong as a rule: followers are the entry ticket, not the audience.
How large is the initial pool? The code defines no “initial audience size” β the seed is every follower feed request while the post is <48h old, winnowed by four per-viewer gates: (a) activity β the follower must actually request the feed in that window; (b) seen-dedup β followers already shown the post are excluded (10-min served exclusion + seen filters), so the pool shrinks as it converts; (c) filters β muted keywords, blocks both directions, VF per viewer; (d) per-viewer ranking β the post must survive the top-50 cut of that follower’s ~1,000-candidate pool. The only hard numeric caps are per-request/per-viewer (thunder: 1200 posts total, 50 originals per author), and the cold-start “low impressions” definition (<200 Home-Timeline views, Oct: was <1,000; β€2h old; β€50K followers). The size advantage of larger follower bases is real but operates through velocity, not delivery probability: more followers offered β more early favorites β faster crossing of the 1-fav/32-fav gates β earlier and larger OON expansion; the model never reads follower count, so per-viewer delivery probability is size-independent. Two seed-stage asymmetries: videos β₯10s enter their video OON windows at creation with 0 favorites (buildVideoIndex on the PostCreation event β phoenixRankAllCandidateProcessor.strato:468-473), while text/photos need β₯1 fav for the main OON pool β so video posts start with a broader initial OON pool as well as longer eligibility (video windows up to 5 years); and the mutual-follow +15 and cold-start boost raise the fraction of followers whose top-50 the post survives.
Factors That Increase Reach#
- High predicted engagement per head (weighted sum above)
- Mutual-follow status: original posts from authors who follow you back get reply weight 5β20 (documented in
docs/BIDIRECTIONAL_BOOST_CHANGE.md; launched Jul 13 2026 at 20, reduced to 15 Jul 24) - Cold-start boost: posts by authors with β€50,000 followers and <200 Home-Timeline views (Oct rewrite: was β€1,000 / <1,000), non-reply/retweet, β€2h old (was 24h; now universal, not treatment-arm-only), ranking within the top 97% of the scored pool (position ratio < 0.97; was 0.85) β one post lifted to slot 15 (author_cold_start.rs). Thompson sampling now on by default (
EnableColdStartThompsonSampling=true; Beta(0.75+favs, 49.25+viewsβfavs), top-2). - In-network status (no OON discount) and “post unexplored” boost (in-network only, 0.015)
- Continuous dwell time (0.0075, capped β0.225 max) + dwell probability head (0.05) + click-dwell (0.385) β modest but real; click-dwell is the large continuous term. (Nov:
video_open0.0 β video earns no direct score term.) - Shares via copy link / DM / quote / reply predictions
- Video quality views (
vqv, gate: video duration strictly >MinVideoDurationMs = 10,000ms β an exactly-10-second video clears the implicit-signal gate but not this one; a viewer with β₯10K followers is exempt from the duration gate; Sept: the VQV scoring weight is now 0.0, so the gate feeds only the implicit discovery signal, not the score) - DPP: candidates with distinct embeddings (esp. multimedia) survive diversity selection better
Factors That Diminish Reach#
- Negative predictions: report (β234) > mute (β58.8) > not-interested (β53.07) > block (β31.2)
- OON discount 0.75 (0.5 topics; 0.00001 for eligible new users); replies/reposts from followed accounts also discounted (only when
EnableOonRescoreForInNetworkRepliesRetweetsis on) (gated βEnableOonRescoreForInNetworkRepliesRetweets) - Author repetition (diversity decay 0.5^k, floor 0.25) β and DPP diversity can zero out similar posts entirely
- VF drops (11 user labels, 17 tweet labels β see 12.5), nullcast, paywall, age > 48h
- Old posts:
MAX_POST_AGE = 48h; model age buckets cap at 80h
2. Account Attributes Affecting Post Reach#
Direct Account Attributes (gizmoduck_hydrator.rs)#
Fetched per candidate author: followers_count, screen_name, retweeted screen name, NSFW flags (nsfw_admin || nsfw_user || NSFW_HIGH_PRECISION label β nsfw_author; + POSSIBLY_NSFW_ACCOUNT β nsfw_author_ads), plus a new nsfw_author_phoenix model input (nsfw_user || nsfw_admin || NSFW_HIGH_PRECISION || POSSIBLY_NSFW_ACCOUNT β stamped as SAFETY_BIT_AUTHOR_NSFW on non-retweets; gizmoduck_hydrator.rs:139-149, candidate.rs:238-244). Still no verification status β the algorithm never reads is_blue_verified (Section 12.6; empirically confirmed in Section 13.7). Sept: on retweets, the model no longer receives the author context at all β is_following_user/followers are now None when a retweet id is present (candidate.rs:265-274).
New account-level inputs (AugβNov 2026)#
- Bidirectional follow (
bidirectional_follow_hydrator.rs):is_mutual_follow_authorβ mutual follows get the +15 reply-weight boost on originals. - Engagement counts (
engagement_counts_hydrator.rs): fav/reply/repost/quote/view/bookmark counts per post (60 s TTL cache) β fed to the model and to cold-start eligibility. - Author cold-start eligibility (
author_cold_start.rs):author_followers_count β€ 50000+ Home-Timeline viewsview_count_on_home < 200(Oct rewrite: was β€1,000 / <1,000) + not reply/retweet + β€2h old (was 24h treatment-arm-only; now universal) + ranked within top 97% (was 85%) β one boost per request (slot 15/16). Thompson sampling now on by default. NewEnableViewerColdStartviewer-side flag. - Semantic IDs (
semantic_id_hydrator.rs): 6-level SID codes per post (multimodal-derived). - Media info (
media_info_hydrator.rs):has_media,min_video_duration_ms(replaces the removedhas_media/video_durationhydrators). - New-account viewer filter (
new_user_min_engagement_filter.rs): for viewer accounts < 30 min old (or resurrected within 30 min), OON posts below an engagement threshold (metricfav, ratio mode off) are dropped. The filter ships disabled (EnableNewUserMinEngagementFilter=false). - Author-level experiment buckets (
author_rules.rs): authors can be bucketed into experiments (e.g., MoE codivert treatment/control) that change whether their posts’ scores survive. - AI-trend feedback context (
ai_trend_feedback_context_hydrator.rs, Sept): whenEnableAiTrendFeedbackContextis on (ships off), the top-2 trending topics among eligible originals are hydrated as post context (ai_trend_name/id) β a possible future discovery signal, currently inert. - Post engagers (
post_engagers_hydrator.rs+clients/post_engagers_client.rs, NEW Nov, ships off βEnablePostEngagers=false): up to 32 positive + 8 negative visible engagers per post (user id, action bitset, age seconds; the viewer is excluded) are stamped intoTweetInfo.postEngagers, so the model can condition on who engaged and how recently. Backed by a Redis cache (post-engagers-cache, 200 ms timeout). This is the first per-engagement identity feature in the ranking input. - Recent search queries (
recent_search_queries_query_hydrator.rs, NEW Nov, ships off): the viewer’s recent queries (text + timestamp + bf16 embedding) are added toUserContext; also a newhydrate_history_url_embeddingsrequest flag. - Video media + embeddings (
video_media_hydrator.rsreplacingvideo_aspect_ratio_hydrator.rs,video_embedding_hydrator.rs, NEW Nov): media id, MD5 hashes, and the v8-search video embedding (from a Manhattan client) are hydrated post-selection for carousel dedupe/CTR. - Home-excursion context (Nov): with
EnableHomeExcursionScores=true, the Phoenix response now carrieshome_video_continuation_secsandhome_profile_visit_secs, and the UAS aggregation isDENSE_WITH_LONG_DWELL_HOME_EXCURSIONβ i.e., the viewer’s own post-impression video-continuation/profile-visit behaviour is part of their history sequence. - History topics (
scoring_sequence_query_hydrator.rs, NEW Nov, ships off): the top engaged topics over a 28-day window are extracted from the scoring sequence and optionally sent to vm-ranker (VMRankerSendHistoryTopics,HistoryTopicsEnabled,HistoryTopicsWindowDays=28). - Soft-NSFW label (
new_user_soft_nsfw_label_hydrator.rs, NEW Nov, ships off with the filter): fetches theSOFT_NSFWtweet label for OON candidates only when the new-viewer filter is active.
Indirect account influence#
- User-cred score (Section 12.2): gates spam labeling and enforcement, not feed scoring directly.
- Safety/user labels (Section 12.5): 11 user labels β VF drops (e.g., SpamHighRecall, Compromised, Impersonation, NSFW*, ReadOnly, DoNotAmplify β the last two exempt followers).
- Subscription status (subscription_hydrator.rs + VF
DropExclusiveTweetContentRule). - Viewer-side: followed topics/starter packs, muted keywords, installed apps, inferred gender, IP geo β all sent in the Phoenix request context.
3. Filtering and Content Eligibility#
Pre-Scoring Filters (in order; 21 in the Nov tree)#
| # | Filter | Removes |
|---|---|---|
| 1 | DropDuplicates | same post from multiple sources |
| 2 | ZeroFavCodivert (NEW Nov, off) | Runs only when the viewer is in the treatment arm; then drops ForYouPhoenixRetrievalZeroFav-only posts whose author is not in the author treatment arm (author rules: treatment/control/unbucketed). Non-zero-fav candidates are never touched |
| 3 | CoreDataHydration | posts that failed text/metadata load |
| 4 | NewUserSoftNsfw (NEW Nov, off) | OON posts with the SOFT_NSFW label when the viewer account is < 7 days old and has < 100 scoring actions (For You, non-topic only) |
| 5 | AgeFilter | posts > 48h (MAX_POST_AGE, config.rs:36) |
| 6 | SelfTweet | viewer’s own posts |
| 7 | OONRetweetReply | OON reposts, OON replies, replies with missing parents (Oct: Phoenix-retrieved OON replies servable when EnablePhoenixOonReplies on) |
| 8 | OONNsfwSimclusters | SimClusters posts by NSFW authors (viewer not following) |
| 9 | RetweetDeduplication | duplicate reposts |
| 10 | IneligibleSubscription | subscriber-only posts |
| 11-13 | PreviouslySeen Γ2 (+backup), PreviouslyServed | already shown/served (bloom filters) |
| 14 | ViewerMutedKeyword (renamed; was MutedKeyword) | posts matching viewer’s muted keywords (tokenizer-based) |
| 15 | AuthorSocialgraph | viewer blocks/mutes author, author blocks viewer, quoted/retweeted author blocks viewer |
| 16 | Video | video posts when request excludes video (quote-embedded video counts β has_video includes quoted_max_duration) |
| 17 | TopicIds | out-of-topic / excluded topics (1502 lines; topic filtering PostBased50Pct) |
| 18 | NewUserMinEngagement | new-account viewer: low-engagement OON posts (ships disabled) |
| 19 | InventoryHoldout | deterministic % holdout per (post, viewer) via splitmix hash, 100 buckets (ships disabled β EnableInventoryHoldout=false) |
| 20 | FavHoldout (Oct, off β EnableFavHoldout=false) | fav-scaled holdout 0β15% (STEPS: 1 favβ0%, 2β3%, β¦, 10Kβ15%) via InventoryHoldoutFilter::is_held_out on the original tweet id |
| 21 | FairShareRanking (NEW Nov, ON β EnableFairShareRanking=true) | trims the pool to PhoenixScoresResultSize (2,800) by proportional fair share per ServedType (largest-remainder), so no retrieval source is truncated out of the scored pool. Skipped for cached-post requests and when the Phoenix ranker kill switch is on |
Nov pipeline notes: Brazil2026ElectionFilter is deleted (README Oct 26: the 2026 Brazilian election is over; posts it covered are no longer filtered). NewUserSoftNsfw is the first filter keyed on scoring-sequence length (a proxy for “brand-new viewer”) rather than account age alone. FairShareRanking is the first filter whose job is composition rather than removal β it changes which sources reach the scorer, not whether a post is eligible. DropDuplicates/RetweetDeduplication still merge retrieval_sources provenance (first copy wins, sources appended); VFFilter keys on visibility_action (Allow|Interstitial|Avoid|Downrank pass; Drop|Tombstone|NotEvaluated drop); VideoFilter drops quote-embedded video when videos are excluded. Nov 19: the filter set and order are unchanged (21 filters); no filter was added, removed, or default-flipped in the Nov 10β19 commits β the diff is refactors (partition helpers, a muting ASCII prefilter) plus carousel-only changes (EnableVideoCarouselMediumRiskFilter and β¦SeenFilter now default true, but the carousel itself still ships off).
Sept: the Following pipeline gained parity filters β FollowingViewerMutedKeywordFilter (mutes also match quoted-tweet and reply-ancestor text, via new quoted_post_text/ancestor_texts hydrators), FollowingBlockedByHydrator (author-blocks-viewer now enforced), and its own VFFollowingCandidateHydrator (TimelineHome-only verdicts); night-owl sources now populate retweeted_user_id to feed these. The For You order above is otherwise unchanged β Oct: 19 filters; Nov: 21 (the README table still lists far fewer).
Post-Selection (in order)#
VFCandidateHydrator (in-network candidates and retweeted source ids β TimelineHome rules; OON candidates + ancestors/quotes β Recommendations rules; 50-chunk, 400 ms) β VFFilter (Oct: drop on Drop|Tombstone|NotEvaluated actions) β AncillaryVFFilter (drop posts whose ancestors/quotes/reposts dropped) β AncestorVideoFilter (Oct: drop ancestor_has_video when request excludes video) β DedupConversationFilter β VideoCarouselFilter (Oct, last, off: pulls up to 30 extra video candidates into a carousel, max 5/min 3, position 6; Nov: immersive ranking, embedding/media dedupe, brand-safety gates, seen-cache, optional CTR model).
Repost semantics for the filters: AgeFilter computes age from the candidate’s own snowflake id (age_filter.rs:16-20) β a fresh repost passes 48h even when its original is old β but the model’s post-age features are computed from the original id that home-mixer sends (candidate.rs get_original_tweet_id() β util.rs original_age_sec, xai-recsys util.rs:505), so the repost is scored with the original’s age. PreviouslySeenPostsFilter also checks the retweeted id (related_post_ids_iter now also covers reply parents β candidates_util.rs:13-17), so reposts of posts a viewer already saw are dropped; RetweetDeduplicationFilter keeps the first-seen copy per original per request (first wins, original included β retweet_deduplication_filter.rs:19-29). Net effect for a self-repost of a poorly performing post: the same post id β same embedding and author state, re-scored with the original’s (worse) age features, deboosted Γ0.75 in-network (if the rescore switch is on), removed for everyone who saw the original β with no freshness reset; the only structural benefit is a fresh retrieval-index entry (the repost is a new id in the fresh windows). See account-analysis Part 22.7.
Other pipelines’ filters#
SelfReplyChainFilter+ConversationGapAncestorHydrator(reverse-chron pipeline only β Sept: ancestor-expand failures now fall back to original ancestors instead of erroring, and the hydrator also fillsancestor_textsfor mute matching),FollowingRetweetDeduplication,AdAdjacentServed(blending),PushToHomeDedup,InvalidConversationModule,PopularTopicsAuthorDedup.
4. Key Algorithm Insights#
- No vocabulary β hash embeddings everywhere: 2 hash functions each for user/item/author (moduli 2_859_568_897 / 2_361_375_383 / 631_860_353). New posts representable immediately. (README claim #3 β)
- Candidate isolation in ranking: candidates cannot attend to each other; right-anchored RoPE positions; scores consistent and cacheable. (β)
- Multi-action prediction: 60 actions + 8 continuous heads (Nov:
react_with_videoβpost_video, plus agrok_analyzehead; both zero-weight); weighted blend is an explicit, tunable, fully visible step. - Ranking and visibility are separate services (README design decision #4):
home-mixerranks;visibility-filtering/decides show/drop. The VF service is now open and its rules are enumerable. - The “no hand-engineered features” era is over as a claim: the README no longer makes it, and the code is full of hand-built features (engagement-count buckets, tweet-type metrics, VQV duration gates, follower thresholds, mutual-follow boosts, cold-start heuristics, minhash Jaccard).
- Continuous training code is released; no checkpoint ships. Training data pipeline:
reference/dump_gen.pysynthetic dumps +world.pysimulation (10,000 users / 2,000 authors / 50,000 posts / 16 topics). - UAS (user action sequence): aggregated per-viewer sequences are the model’s main input; window is 5 minutes (
UAS_WINDOW_TIME_MS = 300_000) withMaxSeqLengthScoring/Retrieval = 1024.
5. HIGH IMPACT FACTORS: Deep Dive Analysis#
5.1. The Phoenix Model β now fully specified (production configs)#
Ranking (home_direct_packed, phoenix/xrex/configs/xrecsys.py:239-313):
| Config | Value |
|---|---|
| layers | 8 |
| emb_size | 2560 (emb_table_width 1024) |
| attention | 20 query / 4 KV heads, key_size 128, qk_norm, right-anchored RoPE, causal=False (candidate isolation), flash-attention v3/v4 (Pallas/CUTEDSL) |
| history_seq_len | 1022 (seqpack, packed per-user events, block 128) |
| candidate_seq_len | 64 |
| output | 64 action logits (60 used) + 8 continuous heads (DWELL_TIME MAE/tweedie) |
| SID | 6 levels Γ 256-code codebook, 1024-dim, hash-level + optional cross-attention |
| vocab | user 100M, item 100M, author 30M, IP 10M (hashed, 2 hashes each) |
| training | total 1e11 samples, rowwise-AdaGrad embeddings (lr 0.28, half-life 2500 H100 / 5000 GB300, lazy decay + small wd β Sept), Muon dense (Sept; was AdamW: 7.1e-4 H100 / 5e-4 GB300), checkpoint every 150/300, copy_port 9988 |
| post age | POST_AGE_MAX_MINUTES = 4800 (~80h) bucketed hourly (82 buckets incl. overflow) |
Sept: home_direct_packed now learns from live counts + follow directions + stale posts. The production feature-prep config enables enable_engagement_counts (live fav/reply/repost/quote/view counts through a log2(1+c)/32 β MLP-64 β embedding path β velocity now shapes predictions directly, not just via buckets), enable_is_author_followed_by_viewer + enable_is_author_following_viewer, and enable_stale_post (recsys_feature_prep.py:628-667, xrecsys.py:272-292). The bool-feature indices are now shipped (feature_config.py:42-44: isStalePost14d=0, isAuthorFollowedByViewerSeq=1, isAuthorFollowingViewerSeq=2; STALE_POST_14D_TTL_SEC = 1,213,200).
Features (recsys_feature_prep.py:130-217, 382-592, 609-667): user token (hashed user id + IP) + user-features token (country 250, language 200, location Fourier 32-d, gender, inferred gender, age brackets, installed apps 64-multihot); history/candidate tokens = post (2 hashes) + author (2 hashes) + action multihot + product surface (16) + post-age buckets + timezone (32) + hour-of-day (25) + SID + multimodal-embedding projection (v5, 1024-d) + engagement-count buckets (32; fav capped 18, reply 13, repost 15, quote 13, view 25). Sept: home_direct_packed additionally feeds live counts through a log2(1+c)/32 β MLP-64 embedding path plus both follow-direction flags (recsys_feature_prep.py:609-667) β velocity and mutual-follow shape predictions directly, on top of the buckets. Negative-feedback masking: any of 9 negative actions on a candidate zeroes all other (non-negative) heads (recsys_model.py:2781-2823) β ships disabled: mask_candidate_positive_when_negative_action_present=False by default and no shipped config enables it.
NEW (Aug 14, shipped since Sept): three serving-side bool features (phoenix/crates/common/xai-recsys/src/util.rs): (a) IS_AUTHOR_FOLLOWED_BY_VIEWER β viewer follows the candidate’s author, and (b) IS_AUTHOR_FOLLOWING_VIEWER β the author follows the viewer (mutual-follow direction), both stamped on candidate and history tokens from isAuthorFollowedByUser / author_info.is_following_user; and (c) IS_STALE_POST14D β since the Sept update, home_direct_packed sets enable_stale_post: True (xrecsys.py:272), so candidates older than STALE_POST_14D_TTL_SEC = 1,213,200 s (~14.04 days; now shipped in feature_config.py:74) have their engagement counts zeroed and the flag set.
Nov: home_direct_packed gains recency + excursion history, and goes FP8/candidate-only. The prod ranking config now enables (xrecsys.py:239-313):
enable_time_gap(lr scale 4.0): a 19-bucket log-spaced “how long ago did this history event happen relative to this request” feature (edges at 2^k minutes, k=0β¦17), stamped on history tokens. The model’s view of the viewer’s sequence is now time-aware, not just order-aware.enable_home_excursion_history:HOME_VIDEO_CONTINUATION_SECSandHOME_PROFILE_VISIT_SECSfrom history are log1p-normalized (scale 300) and projected through zero-initialised learned vectors onto history tokens β the model learns from what viewers did after leaving the timeline.- Two candidate-only layers (
num_candidate_only_layers=2,cutedsl_ranker_candidate_only_attn) β extra transformer layers that only update candidate tokens (cheaper candidate capacity without touching the user/history representation). - FP8 GEMMs (
fp8_gemms=ALL), seqpack candidate packing (candidate_count=[128], mean history 414). view_loss_weight=0.5β a view/implicit-impression auxiliary objective alongside the action heads.- Home-excursion continuous losses:
home_video_continuation_loss_weight=0.013,home_profile_visit_secs_loss_weight=0.0035(CREAD, conditioned onVIDEO_OPEN/CLICK_PROFILE). - Optimizer: global-norm clip 1.4, embedding weight decay 5.6e-4 with a 37-step exempt window and stochastic rounding.
Nov: the released model families shrank. recsys_gen_recs_model.py, recsys_sid_retrieval_model.py, xrecsys_gen_recs.py, xrecsys_sid_retrieval.py, their trainers/runners, and the gen-recs reference artifact generator are deleted from the repo; the xai-recsys-sid-retrieval service (and SidSource) remain. New model code in their place is page-level RL: page_decoder.py (set decoder over the ranked page), page_critic.py (discounted suffix-value critic), page_rl.py (reward = dwell/engagement/completion β serve-score, Ξ³=0.8), configured by PageRlConfig and trained in the immersive_page_rl* configs. Nov 19 adds page_critic_only.py β a standalone 2-layer critic (d_model 256, 8 heads, 48-step horizon, TD-Ξ» 0.9, exploration-return weight 9.0) with a Gumbel-top-k decode (Ξ· / author-overlap / SID-overlap / prior-Ο knobs); it can be selected as serve_decode="critic" (default remains "actor") and is trained by the new immersive_page_rl_cread_critic_only config.
Retrieval (two-tower, recsys_two_tower_model.py + xrecsys_two_tower.py): user tower = transformer over history + user features (no learned per-user embedding, use_user_embedding=False); candidate tower builds representations from SID + hashed author + post features (max_posts 10.24M table, 28.67M combined); cosine top-K, in-batch negatives + 64 global negatives, log-Q correction, learned temperature; three heads (Oct: was 2): home (positive = favorite) + immersive (8 positive actions) + ld (recap-dwelled, w_ld=0.75, evals swap to (0,1) weighting); num_candidate_heads 2β3, HOME_COLD/HOME_HOT map to head 0, new immersive_only model family. Oct adds (all default-off in prod): fixed-temperature, own-negative logit offset, positive-count user weighting, in-batch false-negative masking, recency-gated user_query_heads, and a MoL (mixture-of-logits) item scorer with fused int8 CUDA kernel (xrex/cuda/mol_epilogue/). Retrieval serves top-K from an index stored inside the checkpoint (phoenix/README.md:114-117) β maintained by phoenix-rankall (Section 5.4). Oct: SID snapshots moved v5βv8 (post_sid_v8_256x6_snapshots), HOME_COLD/HOME_HOT share the 1fav_1day parquet with cold-pool + fresh-post masks. Nov: a new xrecsys_two_tower_home_only config (2 heads home+ld, HOME only) narrows the read-through objective to the home surface; an ad tweet_click head plus objective-conditioned slot weights (objective_head_weights, OBJECTIVE[@FORMAT]:w) let one post table serve multiple ad objectives; the retrieval dataset gains IMMERSIVE_32FAV_VIDEO_14_TO_30_DAY (type 19) and the corpus reader now carries per-row objective/video bits; the embedding LR doubles to 0.2, cold_start_max_age_seconds=7200, and dense-parameter EMA (0.99) is added; gen_recs/sid_retrieval datasets are gone.
Nov 19 model/feature changes (mostly shape and plumbing, not objective): the post int64 feature block grows 17β22 with six DPA-SID slots (dpaSidLevel0..5 at indices 3 and 17β21; 6 levels Γ 256 codes, read from dpaSidCodesSeq and stored +1 so 0 means missing). The embedding is gated by enable_dpa_sid_feat (default false), but POST_INT64_FEATURE_SIZE is consumed by the model runner, so every ranking model’s input/parameter shape changes β old checkpoints need retraining or migration. The xrecsys_two_tower_home_only config now sets ld_label_exclude_clicks=true, so a click-only dwell no longer counts as recap-dwelled unless the row also has a video-quality view, or β₯10 s of dwell plus a qualifying non-click action (a live label change for that model). New but unconsumed: a long_term_author_profile reader (8 batch keys: hashed author ids, counts, strong counts, last-engaged, in-window, dwell stats), recent-positive history value scaling (identity at defaults), per-field ad-side-id embeddings (dim 0), and per-request CREAD decode power/scale (defaults 1.0). Two-tower serving adds int8 row padding (auto-64 on SM80) and an opt-in shard-local top-k; training readers move to RustParquetDataset for both offline dumps and retrieval corpora. The proto adds RetrieveTopKUsers (Rust host only, --enable-post-to-user), AI-video dataset types 20β29, and new aggregation enums (IMMERSIVE_LONG_DWELL_GALLERY_3S, IMMERSIVE_RECENT_IMPRESSIONS, DENSE_WITH_LONG_DWELL_AUTHOR_PROFILE), none yet referenced by Python configs.
Gen-Recs & SID retrieval (REMOVED Nov 2026): the Oct release described two experimental families β recsys_gen_recs_model.py (autoregressive retrieval over multimodal embeddings) and recsys_sid_retrieval_model.py (seq2seq next-post SID generation). Both are deleted from the Nov tree, together with their configs, trainers, and runners. What remains is the production path: the two-tower retrieval model, the xai-recsys-sid-retrieval microservice (prefix index over SIDs, caps 5,000 results / 2,000 per seed) backing SidSource (off), and the 1-layer reply two-tower v2 (xrecsys_reply_two_tower_v2.py + recsys_reply_two_tower_v2_model.py; history 127 / candidate 64 / emb 256, SID 16 levels, focal-parent placement via reply_features.py, dwell-tweedie + negfb-weighted loss) β this release adds ad-heads/objective plumbing to it rather than removing it.
Serving: Rust engine (xai-recsys-engine: admission control, SID hydration, embedding gather, checkpoint distribution, retrieval top-K) + Python/JAX inference (xrex/inference/model_runner.py). gRPC RecsysPredictor.PredictNextActions / RecsysRetrievalPredictor.RetrieveTopKCandidates; request carries UserActionSequences (Arrow columnar), TweetInfo[] (author flags, engagement counts, semanticIds, language, media bools), UserContext (age/gender/geo/apps/followers/following/blockers/muters counts).
No checkpoint ships: TRAINING.md:90 “No pretrained checkpoint is included”; training is on synthetic data (QUICKSTART seed 20260721, 12,288 rows; world.py sim). The old oss-phoenix-artifacts.zip (2.9 GB, 128-dim/4-layer) is gone.
5.2. Filtering Pipeline: Binary Impact Factors#
Unchanged in structure; now with exact values: MAX_POST_AGE = 48h (was “unknown, likely 6-24h” β the estimate was wrong), MinVideoDurationMs = 10,000, inventory holdout % (0 by default β ships disabled), NewUserMinEngagementFilter (30-min account age window; ships disabled), Nov: NewUserSoftNsfw (7-day account, <100 actions; off) and FairShareRanking (per-source 2,800 budget; on). VF adds the binary post-selection gate (Section 12.5).
5.3. Scoring Multipliers β with real numbers#
- VMRanker DPP (highest impact, post-scoring): greedy-MAP determinantal point process over v5 multimodal embeddings (f16, 1024-d). Quality score
qf = exp(Ξ±Β·(score/max_score)), kernelL_ij = qf_iΒ·qf_jΒ·cos(emb_i, emb_j),Ξ± = ΞΈ/(2(1βΞΈ)), ΞΈ = 0.65 (client) / 0.5 (server). Candidate pool = topmax_selected_rankby score (150 client-sent / 100 server default); greedy selection stops attop_k= 50 or epsilon; non-selected candidates are scored 0.0 β effectively removed. Missing embeddings β random unit vector. Duplicate/identical embeddings collapse to one survivor; orthogonal content is promoted. This is the only diversity mechanism that can remove in-network posts. - Author diversity (0.5^k, floor 0.25) β per-pool exponential decay, resets per request.
- OON factor (0.75 / 0.5 topics / 0.00001 new-user) + in-network reply/retweet deboost (gated by
EnableOonRescoreForInNetworkRepliesRetweets). - Cold-start boost: lift one eligible post to slot 15 (
ColdStartSlotMin=15, Max=16) β eligibility (Oct rewrite): author β€ 50,000 followers, < 200 Home-Timeline views, β€2h old (universal), position ratio < 0.97 (top 97%).EnableViewerColdStart=true. Thompson sampling now ON by default (EnableColdStartThompsonSampling=true) draws a Beta(Ξ±0 + favs, Ξ²0 + viewsΒ·scale β favs) reward per eligible post (Ξ±0=0.75, Ξ²0=49.25, scale=1.0), keeps the top-2 sampled rewards, and promotes the highest-scoring post among them β favoring the least-proven candidate when data is near-zero (author_cold_start.rs). - Bidirectional reply boost +15 (reply 5 β 20) β the largest single positive weight in the system.
- Dwell-regret mode β DELETED Oct (
value_model_gate.rsremoved; weighted sum is the only mode). - Fair-share source allocation (NEW Nov, ON): before scoring, the 2,800-candidate Phoenix budget is split proportionally across
ServedTypes (largest-remainder). A source that over-supplies (Thunder ~1,200, Simclusters ~800, PopularPosts ~500) is thinned rather than allowed to crowd out the discovery sources. This is a composition control, not a per-post multiplier β but for OON candidates it materially raises the chance of being scored at all. - Topic requests skip DPP (NEW Nov):
dpp_enablednow requires!topic_request || TopicDppEnabled(the latter ships false), so topic feeds are ordered purely by score β removing the old diversity zeroing on topic surfaces. - User debias (NEW Nov, off): when
UserDebiasEnabledand a model artifact is loaded, 14 head probabilities are recentered toward a global baseline using the viewer’s own sequence (debiased_probability(p, user_logit, global_logit, clip=2.0), blended at Ξ±), then the value model is recomputed. Artifacts older than 3,600 s are skipped. Ship state: disabled (Ξ±=0). - Multiplicative value model (NEW Nov 19, off): when
ValueModelMode="multiplicative", the weighted sum is replaced byskip Γ engage Γ timeβskip=(P(not_dwelled)+offset)^exponent(1.0 when the exponent is 0),engage=Ξ£ positive/negative weighted heads excluding dwell/not-dwelled + engage_bias, andtime=1+max(0, cont_dwellΓdwell). A negativeengagefalls back to the additive path, and the negative-sum normalizer uses only the four behavioural negatives (not-interested, block, mute, report). Ship state:"weighted"; exponent/offset/bias are 0.0, and cachedweighted_scoreis ignored if the mode flips. - Per-class engagement/dwell factors (NEW Nov 19, off):
OonEngagementWeightFactor,OonDwellTimeWeightFactor,InNetworkEngagementWeightFactor,InNetworkDwellTimeWeightFactor(all β1.0 sentinel) let the server applylevel Γ (engagement_factor Γ (score β dwell) + dwell_factor Γ dwell)with dwell clamped to[0, score]; only author diversity is re-applied (the OON multiplier is baked intolevel). Skipped in multiplicative/pre-offset modes and for topic requests. - DPP sliding window (NEW Nov 19, off):
DppWindow(0),DppWindowHalfDistance,DppWindowSidLambda(0),DppGreedyOrder(false). When enabled, the greedy DPP re-penalizes each pick against the previous N picks and the viewer’s 10-minute served history (0.5^(d/half_distance)), adding a SID-overlap term when history shares the candidate’s L1 SID; a newSidClient(32 lazy channels, 30 ms, 10 M/48 h cache) supplies the codes. Default output is unchanged.
5.4. phoenix-rankall: what gets into the retrieval index (Nov 2026: Main 19 / SID 24 + reply windows)#
Kafka (phoenix_rank_all_indexing_event) β windowed parquet indexes per window name (config/mod.rs; Oct: Main 17β19 with 1fav_reply_1day + 32fav_reply_1day reply windows, SID gains 1fav_reply_1day, MmMetadata pipeline deleted; test asserts 19. Nov 7: SID 17β18 with the two 6-hour video windows; Nov 19: SID 18β24 with six AI-video windows; Main unchanged):
- Main (19): post_creation_1day Β· 1fav_1day Β· 1fav_2day Β· 1fav_video_2day Β· 1fav_video_2to4day Β· 1fav_video_4to14day Β· 1fav_video_4to30day Β· 1fav_reply_1day Β· 32fav_reply_1day (NEW Oct β fav’d non-self replies fan out to reply indexes + metadata dump) Β· 32fav_1day Β· video_2/4/7/14/30day Β· nsfw_video_2/7day Β· evergreen_video_1825day (5 years!) Β· evergreen_nsfw_video_1825day
- Topic (7): 1fav_1day + topic variants Γ5; SID (24): 1fav_1day, 1fav_reply_1day, 1fav_video_2day/2to4day/4to14day/4to30day, video_6h, video_6hto2day (Nov 7), video_2/4/14day, nsfw_video_2/7/14day, evergreen_video 5y, imagine_4day, ai_video_self_1day/2day/7day + ai_video_detected_1day/2day/7day (NEW Nov 19; each excludes any post in an
nsfw_videowindow), video_4to14day, nsfw_video_4to14day (NO grok pool); SidTail: long-tail authors (< 1000 followers, β₯ 0 favs β notetail_min_fav_count = 0). - Nov mechanics: window names are now generated from hours when the span is sub-day (
video_6h,video_6hto2day,video_6to12h) instead of integer days; reply indexing events now carryinReplyToPostId, so reply indexes can be joined to their parent post. Nov 19: the fav-event and post-creation processors emitai_video_self(user-declared AI disclosure) orai_video_detected(C2PA, Grok signature, or agent-applied) index events for immersive videos; a repost inherits its source’s AI-video index. Compaction removes any post from anai_video_*window when it appears in annsfw_videowindow. No existing window threshold changed. - Sept mechanics β min-age windows + 0-fav emission: bounded
*4to14day-style windows only serve posts older thanmin_age(young posts are held in aprefloor_*sidecar and survive restarts from it;PREFLOOR_SLACK_SECS=6h); the fav-event processor now emits1favunconditionally (0-fav race no longer misses the index) andbuildMMEmbMetadataDumpcovers 0-fav posts too; the fresh-post path moved to the v3 content-understanding topic (2-min delay preserved).
Engagement thresholds decide corpus membership β a post with β₯1 fav enters the 1-day index; β₯32 favs additionally enters 32fav_1day (favorites only β replies/reposts/quotes/views never open a window); videos have their own windows up to 5 years (evergreen + age-banded 1fav_video_* slices; videos enter at creation with 0 favs). Membership is eligibility, not serving. Blacklist files (blacklisted_post_ids.txt, blacklisted_author_ids.txt) applied in the topic pipeline. The Analysis pipeline archives served phoenix scores (home_mixer_phoenix_scored_candidates, prefix phoenix.HomeExperiment1Lap7., 7-day retention, parquet; Sept: the cached score payload no longer carries per-head prediction scores).
Index admission mechanics (phoenix-rankall-strato, Section 1.G): the post_creation window is built from the content-understanding unified-posts topic with a 2-minute forwarder delay (postCreationEventForwarder.strato:17; Sept: topic moved to v3, key-first (Long, Event)), no engagement required, 24h retention. The 1fav / 32fav / 1fav_topic / video / nsfw_video windows are built only on favorite events (phoenixRankAllCandidateProcessor.strato:455-469; Sept: fav’d immersive videos fan out to both video and the new 1fav_video streams, and build1FavIndex now emits unconditionally β the 0-fav race no longer misses the index, counted via 1fav_uec_count_race); the first favorite triggers immediately, and later favorites re-trigger only when the count crosses the next power-of-two threshold (2, 4, 8, 16, 32, β¦ β favoriteEventProcessor.strato:14-21) at most once per minute, stopping 48h after the first update / 49h after creation (eventProcessing.strato:28-40). Index admission consults VF at TimelineHomeRecommendations (shouldDropPostByVF) β posts VF would drop for non-followers never enter the index. The training pipeline’s HOME retrieval corpus is the 1-fav pool (1fav_1day); post_creation_1day is the default global-negative pool in the ranking data pipeline (retrieval_dataset.py:231-235, parquet_recsys.py:863-865).
6. Reach Optimization: Hierarchy of Influence (updated with real values)#
π TIER 1: CRITICAL (binary elimination)#
- VF drops (11 user labels, ~20 tweet labels incl. SPAM/PDNA/BOUNCE/FOSNR, legal takedowns, nullcast, NSFW gating, exclusive content) β Section 12.5; Nov: FOSNR is follower-tiered (followers see a notice, non-followers a drop), and reaction videos can inherit a blur
- Post age > 48h (
MAX_POST_AGE) - Author suspended/deactivated/protected + viewer blocks (both directions)
- Muted keywords, seen/served dedup, self posts
- DPP non-selection (score forced to 0) β especially for near-duplicate content; Nov: no longer applies to topic requests
π₯ TIER 2: HIGH IMPACT (large multipliers)#
- Report/mute/not-interested/block predictions (β234/β58.8/β53.07/β31.2)
- In-network vs OON (0.75; 0.5 topics; 0.00001 new users) + in-network reply/retweet deboost (gated; Phoenix OON replies additionally gated off)
- Mutual-follow original posts (+15 reply weight β effectively 20 vs 5)
- Cold-start boost (β€50K followers + <200 Home-Timeline views + β€2h β slot 15; Thompson on)
- Share via copy link / DM / quote / reply predictions (20/5/5/5)
- Click-dwell (0.385) + dwell time (0.0075, capped ~0.225 β modest; Section 1.B)
π₯ TIER 3: MEDIUM#
- Video duration β₯ 10 s (VQV gate, implicit-signal minimum β the VQV weight is 0.0, so the gate matters only via the implicit discovery signal; Nov:
video_openis also 0.0, so video earns no direct scoring weight at all β only its structural eligibility and the carousel/immersive path) - Author diversity decay (0.5^k)
- Follow-author prediction (4.0)
- Engagement-count features (log2-MLP path + buckets) + follower count (as model inputs & cold-start gate)
- New-user OON lockdown (if enabled)
- Fair-share source allocation (Nov, on) β determines whether your OON post is in the scored 2,800 at all
π TIER 4: LOW#
- Likes (0.708), photo expand (0.05), click (0.274), open link (0.2)
- Product surface / timezone / hour-of-day context
profile_click(0.0 default),quoted_vqv(0.0),active_secs(0.0),video_open/post_video/grok_analyze(0.0)
7. Important Files Summary#
| File | Purpose | Notes (Aug 2026) |
|---|---|---|
README.md | System overview | Rewritten Aug 13, 2026; see Section 11 |
home-mixer/params/param.rs | ALL weights + FS defaults | OPEN: exact values in Section 1.B |
home-mixer/params/config.rs | Structural constants | MAX_POST_AGE=48h, TOP_K=50, RESULT_SIZE=35, UAS window 5 min, NEGATIVE_SCORES_OFFSET=0.001 |
home-mixer/scorers/ranking_scorer.rs | Active scoring | DELETED Oct β replaced by xai-value-model/scoring.rs + weights.rs (shared crate) consumed by home-mixer/scorers/value_model.rs (local fallback) and vm-ranker/scoring/value_model.rs (server; owns OON/diversity/MultiplierPreOffset params). Dwell-regret gate deleted |
home-mixer/scorers/phoenix_scorer.rs | ML client | 2800 candidates max (phoenix_request.rs), cluster routing, stamps backbone scores + slate context (+ Oct: home-excursion heads when EnableHomeExcursionScores) |
home-mixer/scorers/vm_ranker.rs | VMRanker client | Full RankRequest (ids + author/in-network/reply/mutual flags + cold_start_lift_to_rank + phoenix/weighted_score + SIDs; compute_value_model flag; hardcoded “dpp”; theta 0.65 / max_selected 150; seed pinning) β server computes value model (optional) + DPP |
home-mixer/scorers/author_cold_start.rs | Cold-start boost | Oct rewrite: β€50K followers, <200 Home-Timeline views, β€2h, top-97% β slot 15/16; Thompson-sampling on (top-2) |
home-mixer/scorers/value_model_gate.rs | Dwell-regret gate | DELETED Oct |
home-mixer/filters/brazil_2026_election_filter.rs | Brazil 2026 election filter | DELETED Nov 7, 2026 (README Oct 26: election over). Was 2,800 Electoral-Court-reported IDs, follower-exempt (Oct) |
home-mixer/filters/fav_holdout_filter.rs (Oct, off) | Fav-scaled holdout | 0β15% holdout by fav count |
home-mixer/filters/fair_share_ranking_filter.rs (NEW Nov, on) | Per-source budget | Proportional ServedType allocation of the 2,800-candidate Phoenix budget |
home-mixer/filters/zero_fav_codivert_filter.rs + new_user_soft_nsfw_filter.rs (NEW Nov, off) | Zero-fav codivert + new-viewer soft NSFW | Author/viewer-bucketed 0-fav A/B; OON SOFT_NSFW drop for <7-day-old viewers with <100 actions |
home-mixer/filters/video_carousel_filter.rs + ancestor_video_filter.rs (Oct; carousel rebuilt Nov, still off) | Video carousel + ancestor-video | Carousel (last, 30 extra, max 5/min 3, embedding/media dedupe, brand-safety gates, immersive scoring, optional CTR model); ancestor-video drop when videos excluded |
home-mixer/scorers/page_decode_rerank_scorer.rs + page_rl_exploration_scorer.rs (NEW Nov, off) | Page-level RL rerank + exploration | Page-decode slot rewrite; PlackettβLuce exploration over top-64 at T=0.2 |
home-mixer/sources/sid_source.rs (Oct, off) + popular_posts_source.rs (Oct, on) | SID retrieval + popular posts | SID-prefix (800/50/100/depth-3) via sid-retrieval service; popular (500, most-followed-active, own ForYouPopularPosts type since Nov) |
home-mixer/filters/viewer_muted_keyword_filter.rs + following_viewer_muted_keyword_filter.rs | Mute filters | Sept rename + Following variant (matches quoted/ancestor text) |
vm-ranker/ | Reranking service | OPEN + THICKENED Oct/Nov: value-model scoring shared (xai-value-model/), optionally computed server-side; DPP with seed pinning; per-head weight perturbation; Nov: user-debias model, carousel-model RPC, top_heads, DPP off for topic requests |
visibility-filtering/ + -client/ | Content gate | OPEN, REWRITTEN Nov: 9 levels, 7 action kinds, rule catalog/engine/expansion, FOSNR follower split, reaction-video propagation, notifications, quote pass (shadow), client-version switches, DDG experiments |
safety-label-source/ (NEW Nov) | Shared safety-label source | L1 cache + dragonfly cache + Manhattan fallback + warmer; used by VF and the VF client |
user-cred-v2/ | PageRank credibility | OPEN: score = 165.2 + 7.07Β·ln(mass) (Section 12.2) |
abuse-enforcement-service/ | Enforcement | suspend/label/challenge; cred β₯ 50 exempt (mock 12.34 in the shipped YAML); Sept: panda-reports rule + generic-action path; Oct: action_control/off_switch/overturn_hold + abuse-ledger-service; Nov: confidence accounting, per-head head guards, backfill runs, bb1_* bot rules, 7-day SpamHighRecall TTL |
scarecrow/ + botmaker/ + botmaker-rules/ | BotMaker rules | tweet labels, spam-label store (TTL 26h); Sept: new XReview human-review reporting path |
bdsm/ | Inauthentic-behavior model | 8 heads incl. LegitimateUser; thresholds redacted; Sept: the β₯30 action-count gate is now redacted too (999999) |
agatha/ | User-label modeling | PMI features over graph signals |
safety-label-user-agg/ | Postβuser label rules | POSSIBLY_NSFW_ACCOUNT etc. |
phoenix/xrex/ | Training + inference code | real configs (8L/2560), synthetic data; no checkpoint. Nov: time-gap + excursion history, candidate-only layers, FP8, page-RL models; gen-recs/SID-retrieval deleted |
phoenix/xrex/configs/xrecsys*.py | Prod model configs | Section 5.1 table |
phoenix-rankall/ | Retrieval index maintenance | windowed parquet, SID hydration, analysis store; Nov: 6-hour video windows, inReplyToPostId |
simclusters/ + home-mixer/sources/simclusters_source.rs | 3rd OON source | MODEL_20M_145K_2020, min score 0.5 |
thunder/ | In-network store | 48h retention, 2-min trim, 8K QPS, @grok blacklist |
grox/ | Content understanding | reply-spam buckets β€100/β€500/β€1000; upa deluxe fav β₯ 64; Oct: blast-radius 250K split (spam-screen + reply-rank at 250K, coordinated 180K, Gemma-spam min 125K); multi-step reply-spam flow; v8-search embeddings; key/leading/infrared video lanes. Nov: new PTOS categories (FraudAndFinancialCrime, LocalLawSensitiveContent, NonConsensualNudity), media-tag-aware mention counting, Grok 4.5 internal model, XReview intake config |
under-the-hood/ | Transparency tool | labels per account/post; 365d age + 10 posts/mo; Oct: + legal-compliance (takedown) reporting + jetfuel/ report page; Nov: top-3 score contributions per post, appeal text, optional post IDs |
home-mixer/scorers/weighted_scorer.rs etc. | Legacy scorers | DELETED in this release |
8. Strategic Recommendations for Reach (updated)#
π― HIGH PRIORITY (must optimize)#
- Stay under 48h post age; freshness matters (model age buckets + AgeFilter).
- Avoid any safety/spam label: SPAM, SPAM_HIGH_RECALL, BOUNCE, PDNA, FOSNR, NSFW*, DO_NOT_AMPLIFY β VF drops (author-exempt for some; follower-exempt for AbusiveHighRecall/DoNotAmplify; Nov: FOSNR abuse-insults shows followers a notice but drops non-followers).
- Mutual follows: get authors of large accounts you want to reach to follow you back β original posts get reply weight 5β20 (plus a dormant +2.0 dwell-bonus mechanism that ships at 0). (Replies/RTs of mutual authors do NOT get the boost.)
- Avoid actions that train report/mute/block predictions (β234/β58.8/β53.07) β these dominate the negative side.
- Drive reply + share-via-DM/copy-link + quote + follow conversions β the highest positive weights.
π MEDIUM PRIORITY#
- Small accounts: exploit the cold-start boost β accounts β€50K followers with <200-view Home-Timeline posts (Oct: was β€1K/<1K) β€2h old (was 24h) ranked in the top 97% (was 85%) can get one post lifted to slot 15 per request (non-reply/non-retweet; Thompson sampling on). Low-impression posts benefit structurally β and the eligible class is now 50Γ wider with a much shorter freshness window.
- Video is now structural, not scored: the VQV scoring weight is 0.0 (Sept) and video_open fell to 0.0 in Nov β video earns no direct score term. What remains is eligibility: β₯10 s opens the implicit discovery signal and the long-lived video pools, and carousel/immersive ranking is a separate surface (carousel ships off). Click-dwell (0.385) is the large continuous lever for any format.
- Dwell time: modest but real (0.0075 Γ predicted dwell, capped ~30 s β 0.225 max). Long reading sessions help, but replies/shares dwarf them.
- In-network growth (followers) β OON 0.75 penalty, and new-user viewers get 0.00001. Nov: fair-share allocation also decides which sources reach the scorer.
- Distinct content (avoid near-duplicates) β DPP zeroes similar posts (topic feeds excepted since Nov).
π LOW PRIORITY#
- Likes (0.708) β signal, but ~7Γ weaker than reply weight.
- Profile completeness / verification β still not read by the feed algorithm (affects enforcement-exemption indirectly via user-cred premium teleport; Section 12.2/13.7).
9. Discovered Constants vs Unknown Values (Aug 2026)#
β FOUND VALUES (in code, defaults)#
Scoring (home-mixer/params/param.rs; Nov 2026, sync Nov 18 β no weight changes since Nov 6): favorite 0.708 Β· reply 5.0 Β· retweet 1.0 Β· photo_expand 0.05 Β· video_open 0.0 Β· click 0.274 Β· open_link 0.2 Β· profile_click 0.0 Β· vqv 0.0 Β· share 2.118 Β· share_via_dm 5.0 Β· share_via_copy_link 20.0 Β· dwell (prob) 0.05 Β· quote 5.0 Β· quoted_click 0.05 Β· quoted_vqv 0.0 Β· follow_author 4.0 Β· post_unexplored 0.015 Β· post_video 0.0 (renamed from react_with_video) Β· grok_analyze 0.0 (new) Β· cont_dwell_time 0.0075 Β· cont_click_dwell_time 0.385 Β· active_secs_5m 0.0 Β· not_interested β53.07 Β· block_author β31.2 Β· mute_author β58.8 Β· report β234.0 Β· not_dwelled β0.02 Β· bidirectional reply boost +15.0 Β· dwell boost 0.0 Β· AuthorDiversityDecay 0.5 Β· Floor 0.25 (server vm-ranker/params.rs: favorite still 0.5 / video_open 0.07 / cont_dwell 0.004 / post_unexplored 0.02 / not_interested β47.52 β the two copies diverge) Β· OonWeightFactor 0.75 Β· TopicOon 0.5 (server-side) Β· NEW_USER_OON_WEIGHT_FACTOR 0.00001 Β· NEW_USER_MIN_FOLLOWING 5 Β· NewUserAgeThresholdSecs 0 (off) Β· NEGATIVE_SCORES_OFFSET 0.001 Β· MinVideoDurationMs 10,000 Β· EnableQuotedVqvDurationCheck false Β· dwell-regret / multiplicative-unexplored / click-dwell-penalty params DELETED Β· Nov 19: ValueModelMode “weighted” (multiplicative off; exponent/offset/bias 0.0) Β· class factors β1.0 sentinel (off) Β· DppWindow 0 / half-distance 0.0 / SID Ξ» 0.0 / DppGreedyOrder false Β· AdsBlenderType “slot_spread” Β· EnableTeacherScoreSideEffect true (TTL 3,000 s, 21 heads)
Structure (params/config.rs): MAX_POST_AGE = 48h Β· TOP_K_CANDIDATES_TO_SELECT = 50 Β· RESULT_SIZE = 35 Β· FOR_YOU_MAX_RESULT_SIZE = 48 (35+5+8; FEED_MODULE_SLOTS 5) Β· RANKED_FOLLOWING_MAX_RESULT_SIZE = 38 Β· FOLLOWING_MAX_RESULT_SIZE = 110 Β· UAS_WINDOW_TIME_MS = 300,000 (5 min) Β· MaxSeqLengthScoring/Retrieval = 1024 Β· EngagementSignalsMaxPerType = 15 Β· WHO_TO_FOLLOW_POSITION 6 Β· PROMPTS_POSITION 0 Β· VIDEO_CAROUSEL_MAX/MIN = 5/3 Β· VIDEO_CAROUSEL_EXTRA_CANDIDATES moved to a param (default 30) Β· PhoenixScoresResultSize = 2,800 (fair-share budget)
Cold start (Oct rewrite): ColdStartImpressionThreshold 200 (was 1000; Home-Timeline views) Β· ColdStartSlotMin/Max 15/16 Β· ColdStartFollowerCap 50000 (was 1000) Β· ColdStartMaxPostAgeSecs 7200 (was 86400; now universal) Β· LowImpressionsMaxPositionRatio 0.97 (was 0.85) Β· EnableViewerColdStart true (new viewer flag) Β· EnableColdStartThompsonSampling true (was false) Β· ColdStartBetaAlpha0 0.75 Β· ColdStartBetaBeta0 49.25 Β· ColdStartTsTopK 2 Β· ColdStartImpressionScale 1.0 Β· ColdStartTrackedIds ""
VM ranker (Oct: params moved to vm-ranker/params.rs): cluster “Experiment3” Β· hardcoded value_model_id="dpp" Β· ΞΈ 0.65 Β· max_selected_rank 150 Β· 400 ms timeout Β· server: top_k 50, ΞΈ 0.5, max 100, embedding dim 1024 (v5 f16) Β· full request (ids + author/in-network/reply/mutual + cold_start_lift_to_rank + phoenix/weighted_score + SIDs + compute_value_model flag) Β· optional server-side value-model compute (ships off) + weight perturbation + DPP seed pinning Β· Nov: DPP off for topic requests (TopicDppEnabled=false); user-debias model off (Ξ± 0.0, clip 2.0, max artifact age 3,600 s); GetCarouselModel RPC; top_heads (top-3, share β₯10%) only for Under-the-Hood Β· dwell-regret deleted Β· Nov 19: DPP sliding window (off; 10-min served history + SID Ξ») + greedy-order; class-factor split for OON/in-network (off)
| training | total 1e11 samples, rowwise-AdaGrad embeddings (Nov: lr 0.2 for two-tower; ranking lr 0.28, half-life 2500 H100 / 5000 GB300, lazy decay + small wd β Sept), Muon dense (Sept; was AdamW: 7.1e-4 H100 / 5e-4 GB300), checkpoint every 150/300; Nov ranking adds time-gap (lrΓ4) + excursion-history vectors, 2 candidate-only layers, FP8 GEMMs, view loss 0.5; dense-param EMA 0.99; Nov 19: post int64 width 17β22 (six DPA-SID slots; checkpoints must match) |
Retrieval (two-tower): history 1023 Β· candidate 64 Β· emb 1024 Β· 8 layers Β· 16/4 heads Β· max_posts 10.24M (28.67M combined) Β· 64 global negatives Β· 3 heads (Oct: home/immersive/ld, w_ld=0.75; Nov: xrecsys_two_tower_home_only = home+ld) Β· pos = fav (ld pos = recap-dwelled) Β· hard negs 7 (report, not-interested, see-fewer, unfollow, block, mute, not-relevant) Β· soft neg = not-dwelled Β· attention normalized + uncapped Β· SID snapshots v8 Β· new MoL/temperature/negative-mask/query-head knobs (all off) + sid-retrieval microservice + 1-layer reply two-tower v2 Β· Nov: ad tweet_click head + objective-conditioned slot weights; IMMERSIVE_32FAV_VIDEO_14_TO_30_DAY dataset; gen-recs/SID-retrieval model families deleted Β· Nov 19: xrecsys_two_tower_home_only sets ld_label_exclude_clicks=true (click-only dwells <10 s removed unless a VQV); six AI-video SID snapshot datasets (self Γ 1/2/7d, detected Γ 1/2/7d, model Γ 1/2/7/14d) added to the corpus readers but unused by configs; offline + retrieval readers move to RustParquetDataset
user-cred-v2: jump probability Ξ± 0.2 Β· engagement blend Ξ² 0.5 Β· 7-day engagement window Β· convergence 0.001 / max 50 iters Β· score 165.2 + 7.07Β·ln(mass), clamp [0,100] Β· VIT threshold 55 Β· gates Β±5% (rows, mass [0.99,1.005], VIT count) Β· premium = blue/gray/gold/org/affiliate
Enforcement: 500,000/day + 500,000 post/day caps (MAX_ENFORCEMENTS_PER_DAY, MAX_POST_ENFORCEMENTS_PER_DAY, config.rs:241-244) Β· 24h dedup Β· 15-min skip dedup Β· cred skip: score β₯ 50 or is_high (shipped YAML mock threshold 12.34, annotated as a mock) Β· scarecrow high-PR threshold 54 Β· fallback followers > 25,000 (only when no score and no PREDICTED_HIGH_PAGE_RANK label) Β· spam-label store TTL 26h Β· HLL precision 60 s / TTL 25 h Β· panda-reports rule + GrowthBook generic-action path Β· Oct: mock very_high_follower_count β skip (removed Nov) + action_control (Recording/Live, default off) + off_switch + overturn_hold vs abuse-ledger-service (default off) Β· Nov: confidence/evidence model, per-head head_guard hourly+daily ceilings (tripping the off switch), backfill runs over past decisions, bb1_* bot-label rules, impersonation_scam_classifier_v1_suspend, SpamHighRecall TTL 30dβ7d for llm_slop, inauthentic_detection_v45/cluster_spam rule families removed Β· Nov 19: 12 post-label page actions (NsfaHighPrecision, NsfaLimitedInventory, NsfwHighPrecision, SoftNsfw Γ 7d/30d/permanent), post-label carry/dedup (already_post_labeled), TestUserStatus/role/e-mail/CIDR test-account skips, follower_threshold: 0 reads as stored-off (ceilingβ1), action-control record v6β9
VF: adult age 18 Β· NSFW-gating 16 / age-verification+tombstone 30 EU / local-regulations ["in"] (split lists, params.rs) Β· Nov: 9 safety levels; 7 action kinds (Drop / Tombstone / Appealable / SoftIntervention / LabelInterstitial / MediaRestriction / LimitedEngagement); 119 post rules + 38 author rules (Nov 19; 117/38 before); golden corpus 25 files Β· Article lifecycle drops Β· reaction-video media propagation (adult/gore reaction β blur; protected original author β drop) Β· SOFT_NSFW new-viewer drop (soft_nsfw.yml DDG experiment, ships off) Β· FOSNR follower split (followers: soft intervention; non-followers: drop; author: appealable with limited actions) Β· retweetβsource merging Β· level-82 hydration + promoted-hydration + immersive + notification levels Β· quote pass (shadow only) Β· client app/version switches (freedom_of_speech_not_reach_*, stale_tweet_*, community_tweet_viewer_removed_*, media-blur, age-verification per capability) Β· safety-label source: 10 s local hot key; Dragonfly found 24 h / not-found 60 min (Nov 19; was 10 min), single 60 s refresh, stale-on-any-remote-failure within a request deadline Β· exclusive-content rule Β· follower-exempt labels: AbusiveHighRecall, DoNotAmplify Β· Nov 19: DO_NOT_AMPLIFY and MALICIOUS_URL audiences changed from non-authors to non-followers (OON-only rules; semantic, no feed effect); notification reply = mention list, media-tagged gains spam/risky-reply, new quote + retweet-source passes; FilterTweets no longer returns the safety-label map; retweet merge = higher-severity withholding
Thunder: retention 172,800 s (48h) Β· trim 2 min (5 shards, 10-min cycle per user) Β· max 1200 posts / 600 videos / reply 30Β·author / original 50Β·author / video 100Β·author Β· 8,000 QPS Β· blacklist @grok/@gork/@products
Rankall windows (Nov 19, 2026: Main 19 / SID 24): post_creation_1day Β· 1fav_1day/2day Β· 1fav_video_2day/2to4day/4to14day/4to30day Β· 1fav_reply_1day + 32fav_reply_1day Β· 32fav_1day Β· video_2/4/7/14/30day Β· nsfw_video_2/7day Β· evergreen_video_1825day (5y) Β· evergreen_nsfw_video_1825day Β· sid: above + video_6h + video_6hto2day (Nov 7) + ai_video_self_1/2/7day + ai_video_detected_1/2/7day (Nov 19; exclude nsfw_video) + video_2/4/14day + video_4to14day + nsfw_video_2/7/14day + nsfw_video_4to14day + evergreen_video_5y + imagine_4day + 1fav_reply_1day Β· min-age gating w/ prefloor sidecars Β· tail: authors < 1000 followers, min fav 0 Β· training EVERGREEN corpus video_4to14day Β· MmMetadata pipeline deleted Β· analysis: 7-day retention Β· commit_after_dump, versions_to_keep 3, dump every 180 s Β· Nov: reply indexing events carry inReplyToPostId; sub-day window names (video_6h); AI-video index emission from contentDisclosure/C2PA/Grok-signature; ai_video_model annotation index
grox: reply-spam buckets β€100 / β€500 / β€1000 Β· spam-detection follower threshold 1000 Β· blast-radius thresholds 250K split (Oct; was 120K) β TaskSpamFilter + TaskReplyRankingFilter at GROK_GEMMA_FOLLOWER_SPLIT=250K, TaskCoordinatedSpamFilter at 180K, new GEMMA_REPLY_SPAM_MIN_FOLLOWERS=125K (constants.py; task_filter.py) Β· upa deluxe fav threshold 64 Β· reply scores only move down, per-post dedupe removed, LLMs see follower counts, PTOS safety overhaul (adult-Hard dual-model agreement, ChildSafety fail-closed, ViolentMedia fail-soft, high-fav video lane gated on motion-reveal, off) Β· Oct+: multi-step reply-spam flow (root β₯1K, depth β₯2, same-author parent; Gemma-2 thread scorer needs β₯2 same-author flags incl. newest β label + ReplyRankingScore=0.0), v8-search multimodal embeddings (4 frames + backfill pipeline), key-frame (fav β₯128; β₯1024 unconditional) / leading-frame (β₯30s or broadcast + prior PTOS; fav β₯512) / infrared-injected-adult (fav β₯512; SpamManipulatedMedia) video lanes, AgeGatingHarmfulText category + Gemma-for-Spam/Illegal/AgeGating routing Β· Nov: new PTOS categories FraudAndFinancialCrime, LocalLawSensitiveContent, NonConsensualNudity; mention counting includes media-tag handles; Grok 4.5 internal model; XReviewIntakeConfig Β· Nov 19: many-media PTOS (chunks >4 media into 4-image batches; β₯256 favs or prior adult violation; skips already-NSFW), PromotesClicks/PromotesReplies/PromotesShares policies (only Shares β SpamHighRecall), high-page-rank lookup cached 30 min; is_pasted and reply-spam unchanged
New-account filter: MaxAccountAgeSecs 1800 (30 min) Β· resurrection 1800 Β· metric fav (default) Β· OON-only
SimClusters: max 800 results Β· MIN_SCORE 0.5 Β· MODEL_20M_145K_2020 Β· query LOG_FAV_LONGEST_L2_EMBEDDING_TWEET Β· candidate LOG_FAV_BASED_TWEET Β· ANN scan 50 clusters Β· 2M-entry cache/600 s
Dwell-regret (DELETED Oct): temperature 10.0 Β· dwell floor 1.0 Β· gate bias 1.033918 Β· threshold β0.634264 Β· hysteresis 0.0 Β· WINDOW 28 days β params and value_model_gate.rs removed; weighted sum is the only mode.
Post types: Article lifecycle is now a VF gate (unpublished β DROP on level 82) but still no ARTICLE bit in tweet-type metrics, no article retrieval window, no article model feature (bool flags are reply/retweet/quote/media only), no article weight. Thunder caps classify posts as replies (30) / originals (50) / videos (100) only β articles fall in the original bucket (Section 1.F).
β STILL UNKNOWN (redacted or runtime)#
| Constant | Function | Status |
|---|---|---|
| Production FS overrides | Actual live weights | Read from /usr/local/config/features/home-mixer/main/rust_home_mixer.yml (config.rs:1) β code defaults are “synced” to prod by cron (README.md:404) but not guaranteed |
| NewUserAgeThresholdSecs prod value | New-user OON lockdown | Default 0; prod likely > 0 |
| VM ranker head-value model | The actual “value model” | Redacted β server reads no scores; DPP only |
| bdsm operating thresholds | Inauthentic detection | Sentinel 9.99 in code |
| Grox LLM prompts | Spam/quality classification | j2 files not published (README.md:418) |
| Some botmaker rules | Labeling | Not published |
| Phoenix training data windows/cadence | Continuous retraining | Training code open (Kafka-streaming feeds), but no production data window/cadence constants |
| Follower-count threshold for reply-ranking (grox) | Reply ranking gate | 250K split (GROK_GEMMA_FOLLOWER_SPLIT, task_filter.py; spam-screen + reply-rank at 250K, coordinated-spam at 180K, Gemma-spam min 125K β Oct; was 120K) |
MultiplierPreOffset prod value | Replacement for removed MPN mode | Ships false (Oct: lives in vm-ranker/params.rs); prod unknown |
ComputeValueModel / DppEnabled prod values (NEW Oct) | Server-side value-model compute + DPP toggle | Ship false/true respectively; prod unknown |
EnableSidSource / EnablePopularPostsSource / EnableFavHoldout / EnableVideoCarousel / EnablePhoenixOonReplies prod values (NEW Oct) | New sources/filters/carousel/OON-replies | Ship false/true/false/false/false; prod unknown |
EnableFairShareRanking prod value (NEW Nov) | Per-source budget allocation | Ships true; prod unknown (kill switch can disable) |
EnablePageDecodeRerank / EnablePageRlExploration prod values (NEW Nov) | Page-level RL reranking / exploration | Both false |
UserDebiasEnabled + artifact prod values (NEW Nov) | Per-viewer probability recentering | Ships false / Ξ± 0.0; prod unknown |
EnablePostEngagers / EnableRecentSearchQueriesHydration / HistoryTopicsEnabled prod values (NEW Nov) | New model inputs | All false |
EnableNewUserSoftNsfwFilter prod values (NEW Nov); zero-fav codivert ViewerZeroFavCodivertIsTreatment + author rules | New-viewer soft-NSFW drop; zero-fav A/B | Filter ships false; codivert viewer/author arms all false |
New bool-feature indices (IS_AUTHOR_FOLLOWED_BY_VIEWER, IS_AUTHOR_FOLLOWING_VIEWER, IS_STALE_POST14D) | Model inputs | Shipped since Sept (feature_config.py:36-44,74, incl. STALE_POST_14D_TTL_SEC = 1,213,200) |
enable_stale_post prod value | 14-day stale-post zeroing | Ships True in home_direct_packed (xrecsys.py:272) since Sept; grpc feed forces it off |
ValueModelMode / class factors / DPP window prod values (NEW Nov 19) | Multiplicative value model; OON-vs-in-network engagement/dwell split; DPP sliding window + SID history | All off in code ("weighted", β1.0 sentinels, window 0, SID Ξ» 0.0, greedy-order false); prod unknown |
AdsBlenderType / UserAdLoad prod values (NEW Nov 19) | Ad slot algorithm and per-user ad thinning | Blender default slot_spread; user-load keep/spacing lists empty (no thinning) |
EnableTeacherScoreSideEffect prod value (NEW Nov 19) | Exports served head probabilities to a teacher cache | Ships true (prod-gated); TTL 3,000 s; not a ranking input |
enable_dpa_sid_feat / AI-video training windows (NEW Nov 19) | DPA-SID embedding; AI-video retrieval corpora | Feature ships false; no training config selects the AI-video datasets; rankall emission/windows are live |
10. π Engagement History Windows: Account Recovery Timeline (updated)#
Known windows (all now open)#
| Window | Value | Side | Source |
|---|---|---|---|
| UAS aggregation window | 300,000 ms = 5 minutes | Viewer | params/config.rs:34 |
| Max sequence length (scoring/retrieval) | 1024 | Viewer | param.rs (MaxSeqLengthScoring/Retrieval) |
| Model history capacity (ranking) | 1022 (seqpack) | Viewer | xrecsys.py |
| Model history capacity (retrieval) | 1023 | Viewer | xrecsys_two_tower.py |
| Post age max (model) | 4800 min (~80h) | Post | recsys_model.py:93 |
| Post age max (feed) | 48h | Post | params/config.rs:36 |
| Thunder retention | 48h | Post | thunder/args.rs:48 |
| Cold-start post age | 7200 s (2h; Oct: was 86400) | Post | author_cold_start.rs |
| Engagement signals (per type) | max 15, recency-sorted | Viewer | param.rs |
| VF safety-label cache | 30 s (fresh posts) / 60 s | Post | visibility-filtering |
| user-cred engagement window | 7 days | Account | UserCredV2Config.scala:25 |
| Credit snapshot freshness | β€ 7 days (warm start) | Account | UserCredV2App |
| Spam label store TTL | 26 h | Post | scarecrow SpamLabelStoreRegistry |
| Stale-post flag | 14 days (STALE_POST_14D_TTL_SEC = 1,213,200 β shipped in feature_config.py:74 since Sept; home_direct_packed enables it, xrecsys.py:272) β zeroes engagement counts + sets flag | Post | xai-recsys/src/util.rs |
Architecture change: UAS hydrator replaced#
The old user_action_seq_query_hydrator.rs is deleted. It’s now:
- Scoring/retrieval sequences:
scoring_sequence_query_hydrator.rs/retrieval_sequence_query_hydrator.rscall the UAS aggregation service (UserActionAggregationClient) with 5-minute window,MaxSeqLength = 1024, aggregationDENSE_WITH_LONG_DWELL_HOME_EXCURSIONfor scoring (Nov; wasDENSE_WITH_LONG_DWELL) andDENSE_WITH_LONG_DWELLfor retrieval, sourceNEW_UUA_ARROW, separateUaasModelType::{Ranking, Retrieval}. The sequence is serialized to Arrow and sent to Phoenix. Nov: a secondImmersiveScoringSequenceQueryHydratorfetches anIMMERSIVE_LONG_DWELLsequence for the carousel; the scoring hydrator can also derive 28-day engaged history topics and optionally attach them to the vm-ranker payload (off). - Engagement signals (for SimClusters source + new-user handling):
explicit_engagement_signals_query_hydrator.rs(favorite/retweet/reply/bookmark/share/self-tweets from Twistly, 300 ms) andimplicit_engagement_signals_query_hydrator.rs(photo-expand, VQV, immersive VQV; video views require β₯10 s and non-promoted).
Account status change implications (revised)#
- Viewer-side forgetting is fast: the UAS window is 5 minutes β the model’s view of a viewer’s recent behavior is a 5-minute rolling window (truncated to last 1024 actions). Long-horizon viewer features come from the user-features token (profile age/gender/country/apps) and the 28-day action-count features in the dwell-regret gate β not from the transformer history.
- Author-side “reputation”: still no decay in the model (Section 12.7); now the user-cred PageRank provides a daily-updated account score that gates enforcement, with a 7-day engagement window β i.e., credibility responds to engagement trends over ~7 days of favs/retweets.
- Recovery logic from the old section (recent actions matter most, window-based forgetting) still holds but the window is minutes, not days.
11. README Truthfulness Audit (Aug 13, 2026 README, + Aug 14 / Sept 18 / Oct updates)#
The README was rewritten (482 lines, “Latest update β August 14th, 2026”, README.md:30). It no longer makes the two most-contested claims of the old release (256-dim/2-layer model; “no hand-engineered features”). The Aug 14 commit added a “Latest Updates” section (weights clarification + Brazil 2026 election filter announcement). Sept: the section was renamed “Notable Updates” and the Brazil line gained "(Account list updated August 27, 2026.)" β matching the 665β2,388 list growth in the filter file. Sept 18: a legal-compliance Under-the-Hood notice was added (takedown visibility in reports + jetfuel/ page). Oct: the weights link moved to xai-value-model/scoring.rs, the Brazil note read "(Account list updated September 29, 2026.)" (matching 2,388β2,800), and the filter table added FavHoldoutFilter β but it still missed the 9-source reality, the level-82/EvaluateTweets/tombstone/Article VF expansion, the cold-start rewrite, and all Oct weight moves. Nov: three new “Notable Updates” entries were added β Oct 8 (“Ranking predicts time spent much better” β loss_recsys.py), Oct 13 (“Retrieval trained to surface posts people read all the way through” β recsys_two_tower_model.py), and Oct 26 (Brazil filter removed) β and the Brazil row was deleted from the filter table. The README still does not reflect the 21-filter pipeline, the Nov weight moves, fair-share allocation, the page-RL scorers, the VF action-model rewrite, or the user-debias/carousel-model additions. Verified against code:
β VERIFIED TRUE#
| # | README Claim | Evidence |
|---|---|---|
| 1 | Two sources: Thunder (in-network, in-memory) + Phoenix retrieval & SimClusters (OON) (README.md:60-63, 99-104) | thunder/ (48h in-memory store) + simclusters_source.rs + phoenix_source.rs β (code now runs 9 sources: + PopularPosts (on) + SidSource (off) alongside TweetMixer/Topics/MoE/CachedPosts β README simplifies to the 2 advertised families) |
| 2 | Request-path stages: query hydration β sources in parallel β hydration β pre-scoring filters β scoring β TopK β post-selection VF | Exact match to phoenix_candidate_pipeline.rs chain β |
| 3 | “user action sequence β the main input to the model” (README.md:92-93) | scoring_sequence_query_hydrator.rs (UAS, 5-min window, Arrow) β |
| 4 | Phoenix predicts per-action probabilities; list of heads (README.md:328-332) | 60 action heads incl. all listed (favoriteβ¦not dwelled) β |
| 5 | “weights are in home-mixer/params/param.rs” (README.md:341) | TRUE β this was the previous release’s biggest omission; now shipped β |
| 6 | Three adjustments: author diversity (decay to floor), OON discount (factor < 1, incl. replies/reposts from followed accounts), new-author boost (impressions below threshold lifted to target position) (README.md:347-349) | xai-value-model/scoring.rs + vm-ranker/params.rs (0.5^k, floor 0.25; 0.75; cold-start β Oct: scorer moved out of ranking_scorer.rs) β |
| 7 | VMRanker now documented: “a separate service that reorders the result” / DPP over embeddings (README.md:276, 351) | vm-ranker/ open; DPP over v5 embeddings β (README understates: DPP zeroes non-selected candidates; Oct: server optionally computes the value model itself β no longer purely “redacted”) |
| 8 | Visibility filtering: ALLOW/INTERSTITIAL/DROP; recommendations-only rules that “can only drop”; same post allowed to a follower (README.md:203-206, 211-213, 393-394) | visibility-filtering/rules/registry.rs β exact for Aug β; Oct: verdicts widened (Tombstone/Downrank/Avoid/NotEvaluated), level-82 hydration, Article drops, age-prompt blurs, limited-engagement, retweet merging β README doesn’t reflect any of it |
| 9 | Filter tables (17 pre-scoring + 3 post-selection, in order) (README.md:358-376, 384-388) | Exact match to the Aug 13 code order β (AgeFilter 48h β) β drifted Aug 14: code now has 18 pre-scoring filters (Brazil2026ElectionFilter added; README not updated). Sept: README still lists 17 β and now also misses the ViewerMutedKeyword rename, the Following-pipeline additions, the multi_risk blender, and every weight/structure change below. Oct: README added FavHoldoutFilter but the code had 19 (+ AncestorVideo/VideoCarousel post-selection, 9 sources, new hydrators). Nov: Brazil row removed but the code has 21 pre-scoring filters (ZeroFavCodivert, NewUserSoftNsfw, FairShareRanking all missing from the table) |
| 10 | Labeling path: content understanding (grox, media-model-proxy, clip, agatha, bdsm, user-cred-v2) β labeling rules (scarecrow/botmaker, abuse-enforcement-service, safety-label-user-agg) β storage β VF | All components exist β |
| 11 | Hash-based embeddings, no vocabulary (README.md:460-462) | 2 hashes/entity β |
| 12 | Candidate isolation in ranking (README.md:456-458) | causal=False + right-anchored RoPE, candidates can’t attend to each other β |
| 13 | Multi-action prediction (README.md:452-454) | β |
| 14 | Thunder blacklists @grok/@gork/@products (not claimed in README, but in code) | thunder_service.rs:29-36 β a notable transparency point |
| 15 | “What’s not in this repo”: Grox prompts, some botmaker rules (README.md:412-427) | Honest disclosure β |
| 16 | Under the Hood transparency tool + eligibility (README.md:438-442) | under-the-hood/ (365d age, 10 posts/mo) β |
| 17 | “no pretrained checkpoint is included” (phoenix/TRAINING.md:90); synthetic data for training | β β artifacts zip removed |
β οΈ PARTIAL / UNDERSTATED#
| # | Claim | Reality |
|---|---|---|
| 1 | Architecture diagram shows only Thunder + Phoenix/SimClusters as sources | Pipeline actually runs 9 sources (Thunder, PopularPosts (on), TweetMixer, Simclusters, Phoenix, PhoenixTopics, PhoenixMOE, SidSource (off), CachedPosts) β README deliberately presents the two primary families |
| 2 | VMRanker “reorders the result” | Understated: DPP removes (scores 0) all non-selected candidates; Oct: the server can now also compute the value model itself (ComputeValueModel, off) with weight perturbation β README doesn’t disclose either the zeroing or the new compute path |
| 3 | “aim is for experiments at β₯10% traffic to be visible in the repo” (README.md:402) | Aspirational; several major mechanisms ship at defaults that are off (MultiplierPreOffset, ComputeValueModel, new-user OON, SidSource/FavHoldout/VideoCarousel/OON-replies, dwell-regret deleted, cold-start Thompson now on) with the real prod values living in the excluded FS YAML |
| 4 | Weights “in param.rs” | The values are defaults mirrored from prod via cron; live prod values are feature-switch driven (README discloses the sync mechanism β). Aug 14: the README + code now carry an explicit clarification that weights scale predicted probabilities, not raw counts (Section 1.B) β |
| 5 | “replaces the Phoenix demonstration model with the code used to train the models the feed uses” (README.md:43) | True in structure, but the repo cannot reproduce production models: no pretrained checkpoint, no production data pipeline (synthetic only) |
| 6 | Filter tables (17 pre-scoring + 3 post-selection, in order) (README.md:358-376, 384-388) | Drifted by the Aug 14 commit: the code now has 18 pre-scoring filters β Brazil2026ElectionFilter was added between AuthorSocialgraphFilter and VideoFilter, but the README table still lists 17 and the “Latest Updates” section doesn’t mention the filter-table change (it does announce the filter itself). Same for the new model bool features and grox 30K thresholds: announced in code, absent from README’s table. Sept: the drift is now much larger β the README table misses the Brazil count (665β2,388), the mute-filter rename/additions, the 0.07/0.0/0.05 weight moves, MPNβMultiplierPreOffset, the slimmed VMRanker request, LONG_DWELL aggregation, ColdStartTsTopK 2, FOLLOWING_MAX_RESULT_SIZE 110, the multi_risk blender, grox 120K gates, and the enabled stale-post/count features. Oct: README added FavHoldoutFilter but still missed the 19th filter, the AncestorVideo/VideoCarousel post-selection pair, the 9-source pipeline, the xai-value-model move, the Oct weight moves (0.6/0.02/0.3/0.015/0.006/0.4/β47.52), the cold-start rewrite, the VF verdict/level-82/Article/tombstone expansion, grox 250K gates + multi-step reply-spam + v8-search, rankall reply windows, the ld retrieval head, and all three enforcement guardrails. Nov: README added three dated notes and removed the Brazil row, but still misses the 21-filter pipeline, the Nov weight moves (0.708/0.274/2.118/0.0075/0.385/β53.07 and video_openβ0.0), FairShareRanking, the page-RL scorers, the VF action-model rewrite (9 levels / 7 action kinds / FOSNR follower split / reaction video), user-debias + carousel-model, the Phoenix time-gap/excursion features, and the enforcement head guards/backfill |
β FALSE OR MISLEADING β none found (major improvement over old release)#
The two old false claims are gone:
“256-dim embeddings, 2-layer transformer”β no model numbers in README; actual prod = 8 layers / 2560-dim (Section 5.1).“eliminated every hand-engineered feature”β claim removed entirely.
π TRUTHFULNESS VERDICT (Aug 2026)#
~80% accurate β still the most transparent release, but the README keeps drifting. The README’s architecture, scoring description, and labeling path match the code (as of the Aug 13 commit, plus the Sept 29 Brazil note and the Nov Oct 8/13/26 notes). Remaining gaps: (1) the 9-source reality vs 2-source diagram, (2) VMRanker’s zeroing + optional server-side compute + the Nov debias/carousel RPCs, (3) prod FS values not shipped (structural: the sync mechanism is disclosed), (4) Nov drift: the README’s filter table still lists far fewer than 21 filters; the weights section misses the Nov moves (0.708/0.274/2.118/0.0075/0.385/β53.07, video_openβ0.0); and nothing reflects FairShareRanking, the page-RL scorers, the VF action-model rewrite (9 levels / 7 action kinds / FOSNR follower split / reaction-video propagation), user-debias, the Phoenix time-gap/excursion features, or the enforcement head guards/backfill. Nov 19 adds no README change at all, so the same document also omits AI-video labeling and retrieval windows, the 12 post-label page actions, the ads-blender default change, teacher scores, the multiplicative value model, and the new notification passes. The Aug 14 commit’s strongest transparency point is the explicit correction of a public misconception (weights β count equivalences) in the code itself. Compared to the old release (model-size error + missing VMRanker + false “no heuristics” claim), this is a qualitatively different document.
12. Account Reputation & Reach: How Reputation Determines Visibility (major revision)#
12.1 The headline finding (revised)#
The July claim β “there is no reputation score anywhere in this codebase” β is now false. The Aug release ships a full account-health stack. But the nuance survives: no reputation score feeds the For You scoring directly. Reputation acts through four layers:
| Layer | System | Effect on reach |
|---|---|---|
| A. Learned author embeddings | Phoenix model (ranking + retrieval) | Implicit, per-author engagement propensity (Section 12.2 of old doc still applies; vocab now user/item 100M, author 30M, 2 hashes) |
| B. Explicit credibility score | user-cred-v2 (PageRank mass) | Gates spam labeling, enforcement, challenge/suspend; NOT fed to feed scoring |
| C. Follower count + graph | gizmoduck, socialgraph, bidirectional follow | OON status, mutual-follow boost, VF rules, cold-start eligibility, model inputs |
| D. Labels (post + account) | grox, scarecrow, safety-label-user-agg, abuse-enforcement, media models β VF rules | Drop/interstitial with exact rule catalogue |
12.2 B (new): user-cred-v2 β the explicit credibility score#
user-cred-v2/ (Scalding daily batch, prod since 2026-04-27):
- Algorithm: PageRank-style mass propagation over the follow graph (edges keep their followerβfollowee orientation: each user distributes
mass/outDegreeto the accounts it follows, and accumulates mass from its followers β the.swapatUserCredV2App.scala:295is only a validation step, not an orientation flip), with engagement-weighted teleport: fav/retweet edges (7-day window) weight the teleport distribution, blended with a uniform distribution over premium users (blue/gray/gold verified, org, affiliate) at Ξ² = 0.5. Jump probability Ξ± = 0.2; convergence at L1 diff β€ 0.001 (max 50 iterations). - Score:
165.2 + 7.07Β·ln(mass), clamped to [0, 100]. Daily Manhattan snapshot. - Sanitization: suspended/deactivated/erased/restricted users dropped; linked-account (expansion) edges scrubbed; near-zero users flagged; warm starts keep prior mass.
- Gate: Β±5% checks on row count, mass sum [0.99, 1.005], VIT count (score β₯ 55) vs prior snapshot; failures quarantine.
Consumers (this repo):
abuse-enforcement-service:pagerank_skipped: cred.is_high || cred.score >= 50.0β accounts with credibility β₯ 50 are exempt from automated enforcement rules (user rules yaml:26-31; post rules require additionally!skip_author_credibility_prechecks).scarecrow:IsHighPageRankUser= cred score β₯ 54 OR labelPREDICTED_HIGH_PAGE_RANK; fallback followers > 25,000 when no score and no label. High-PR authors are skipped by spam labeling (e.g. bot 3226 URL-verdict SPAM labeling).- VF user-label aggregation excludes high-PR authors from
POSSIBLY_NSFW_ACCOUNT/NSFW_HIGH_PRECISIONuser labels (safety-label-user-agg).
Reach interpretation: credibility is a shield β it exempts accounts from spam labels, suspensions, and challenges. It does not directly boost feed position. Notably, blue verification matters here: premium users get extra teleport mass (making the cred graph lean toward verified accounts), so blue-check accounts are more likely to hold high cred β enforcement exemption. This is the only place verification enters the algorithm (Section 13.7’s “verification has zero effect” finding was about feed delivery and still holds).
12.3 C (updated): follower count & graph signals#
- Follower count is hydrated per candidate (gizmoduck) and used: (a) cold-start eligibility (Oct: β€50K), (b) SimClusters/rankall tail eligibility (authors < 1000 followers), (c) tweet-type metrics buckets (0-100/1K/10K/100K/1M) and the served-metrics bitset (RETWEET/REPLY/VIDEO/β¦ β no ARTICLE bit), (d) model
UserContextfollower counts. (e) grox’s reply/PTOS LLMs now see follower counts in their prompt context (not a score β prompt text). - grox reply ecosystem: three blast-radius gates in
grox/flows/reply_spam/task_filter.pyβ basic LLM spam screening runs when the in-reply target AND thread root are both at/below the split (Oct:GROK_GEMMA_FOLLOWER_SPLIT=250K; was 120K); coordinated-spam detection runs only when root β₯ 180K followers (Oct: was 1,000); LLM reply-ranking runs only when target or root > 250K followers (was 120K). Oct adds a multi-step reply-spam flow (plan_multi_step_reply_spam.py: reply, depth β₯2, same-author parent, root β₯1K, not high-PR/grey β Gemma-2 thread scorer needs β₯2 same-author flags incl. newest β spam label +ReplyRankingScore=0.0) and aGEMMA_REPLY_SPAM_MIN_FOLLOWERS=125Krouting floor. So replies in small conversations are the ones screened; replies in big threads are ranked instead. Reply scores still only move down; per-post per-minute dedupe removed; v3 unified-posts topic; reply/PTOS LLMs see follower counts. - Bidirectional follow (NEW, big): mutual-follow authors’ original posts get reply weight +15. This is the strongest explicit graph-derived multiplier in the system, documented in
docs/BIDIRECTIONAL_BOOST_CHANGE.md(A/B Jul 10; launch Jul 13 at 20; reduced to 15 Jul 24). - Block/mute both directions: viewerβauthor + quoted/retweeted authors (author_socialgraph_filter.rs:27-56) + VF
ViewerBlocksAuthorRule/ViewerMutesAuthorRule/MutedRetweetsRule/AuthorIsProtectedRule(logged-out or non-follower). - New-user viewer lockdown: viewers < NewUserAgeThresholdSecs with β₯5 followings get OON factor 0.00001 (if enabled) β new-account viewers see an almost purely in-network feed.
12.4 D (new): enforcement β the sharpest reputation lever#
abuse-enforcement-service/ (Rust): consumes Kafka model-score results (dedup classes: none/label/challenge/suspend) β fact gathering (gizmoduck, cred, grey badge, UAS) β CEL rule pipeline (rules mirrored from GrowthBook xai_abuse_enforcement_service_rule_user/post) β rate limits β AIS actions:
| Action | Effect |
|---|---|
suspendUser(perm:false, PlatformManipulation) / suspendUser(perm:true, Cse) | Account suspended |
addLabelsV2 (SpamHighRecall, TTL 30d) | User label β VF drops for non-followers |
addPostLabelsV2 (RiskyHighVizReply, SpamHighRecall β 30d) | Post labels β VF drops |
bounceArkose / bounceCaptcha / spamLivenessCheck | Challenge flows |
| caps | 500,000/day + 500,000 post/day; 24h dedup; 15-min skip dedup |
User rules: anchor_campaign_suspend, llm_slop_user β SpamHighRecall (TTL 7d since Nov), SpamEmbeddingMajorityPoster, panda_reports_embedding_v10_rough_spam β suspend, terminal act_suspend. Nov: the inauthentic_detection_v45_* and cluster_spam* families and the very_high_follower_count mock skip are removed; new bb1_post_spam_bot_* / bb1_reply_spam_bot_* / bb1_user_* label rules (suspend or 30-day SpamHighRecall + liveness check), liveness_check_and_spam_high_recall_7d, spam_high_recall_7d, and impersonation_scam_classifier_v1_suspend. Post rules: llm_slop_post β RiskyHighVizReply, gibberish_post/fast_reply_spam_post/anchor_campaign_post/SpamEmbeddingPtosDistilled β SpamHighRecall, cse_reports_embedding_v2 β suspend author (perm:true). Both pipelines keep the GrowthBook-gated generic-action step, decisions_json mirror, and scarecrow XReview reporting. Oct guardrails remain (page-managed action_control Rules/Recording/Live, off_switch, overturn_hold vs abuse-ledger-service); Nov adds the machinery that makes them operable at scale: a confidence/evidence model (confidence.rs: per-row Claim/RowOutcome/EvidenceOverride, a ConfidenceView API, adversarial tests), head_guard (per-head or head+classifier hourly and daily action ceilings; reaching one flips that scope’s off switch, resume by re-enabling; a PUT with no limits clears the guard), and backfill runs (backfill.rs, backfill_claims.rs: start/pause/resume/stop/done over past decisions, with a budget endpoint, persisted state, and claim-based work distribution). Enforcement-note structure, mock thresholds, and the bdsm redactions are unchanged. Nov 19 adds a post-label enforcement path and test-account guards: PAGE_POST_LABELS defines 12 page actions (NsfaHighPrecision, NsfaLimitedInventory, NsfwHighPrecision, SoftNsfw Γ 7d/30d/permanent) that write AddPostLabelsV2 with a source field; before writing, the service reads the post’s existing labels and skips as already_post_labeled when every requested label is already carried (dedup keys include the labels). Account-ending actions are refused for TestUserStatus (Temporary/Permanent) accounts and for known test accounts matching configured roles/confirmed e-mail domains/creation-IP CIDRs. follower_threshold: 0 no longer means “off”: it is stored as the ceilingβ1 (12,345 at the 12,346 ceiling), and raising a limit to the ceiling requires an exact confirm_follower_limit plus a typed confirmation. The action-control record format moves v6βv9 (post-label records v7), so older images cannot read new records.
bdsm/ (inauthentic behavior): transformer over user action sequences, 8 heads: FollowBot, LikeBot, EngagementAmplifier, ReplySpamBot, TweetSpamBot, RTBot, MultiActionBot, LegitimateUser; graduated actioning (challenge β suspend); operating thresholds redacted (sentinel 9.99). The enforcement-note structure is visible β a note fires when the dominant bot head scores > 0.5 over a policy-defined action count, naming the head (e.g. [model: FollowBot=0.91]); Sept: the count gate itself is now redacted β min_actions_for_enforcement ships as 999999 (was 30; sink_policy.yaml:13, version 2026-08-14), so only the > 0.5 pick logic is verifiable and the real gate value is withheld like the probability thresholds. The note templates and per-head key-action interpolators are the redacted sentinel "<redacted>" (score_results_sink_focal.py).
agatha/: PMI-based user-label modeling (spam_suspended, rate_based, nsfw, reports_per_fav, blocks_per_fav, favs) over graph signals (follow/fav/block/mute/abuse-report/DM/RT/list).
12.5 D (new): visibility-filtering rules β the full catalogue (now open)#
Verdict model (Nov: rewritten as an action fold): the engine now produces an ActionKind per matched rule β Drop, Tombstone, Appealable, SoftIntervention, LabelInterstitial, MediaRestriction, LimitedEngagement β with fixed severity and slot precedence (rules/action_kind.rs, rules/defaults.rs, rules/engine.rs). Drop/Tombstone withhold the post; Appealable/SoftIntervention are FOSNR notices (the post stays visible with a notice); the remaining three fill label-interstitial, media (blur), and engagement slots, and a rule is skipped when its slot is already filled (first writer wins per slot; the other slots remain available). rule_spec/ was split into typed feature leaves (article, author, community, compound, edges, post, reaction, request, tweet, viewer). The old Allow | Interstitial | Drop | Tombstone | Downrank | Avoid | NotEvaluated compatibility surface still exists in treatment.rs.
Levels (Nov: 9): FilterAll, TimelineHome, TimelineHomeRecommendations, TimelineHomeHydration (level 82), TimelineHomePromotedHydration (new; ad hydration), ImmersiveExpandedRecommendations (new; gallery), QuotedTweetRules (new; a quote-pass over quoted entries, VF_QUOTE_PASS=off|shadow), NotificationsFollow and NotificationsWriterV2 (new; per-notification-kind rules for mentions, replies, media tags). In-network = TimelineHome; OON adds TimelineHomeRecommendations (can only drop); retweetβsource verdicts are merged (retweet.rs); quote entries can rerender 8 specific rules rather than reusing the outer verdict. 119 post-rule constants + 38 author-rule constants (Nov 19; 117/38 before the two new post rules), golden corpus 25 files.
Base rules (both surfaces): suspended/deactivated/erased/offboarded author; protected author (logged-out or non-follower); viewer blocks/mutes author; muted retweets; tweet labels PDNA / BOUNCE / SPAM / FOR_EMERGENCY_USE_ONLY / FOSNR level-3 (hateful conduct, violent speech, abuse, civic integrity) β drop; nullcast tweets; superseded edits; legal takedowns (author-country matched; author-exempt; unenforced-takedown redaction filter); unpublished Articles β drop (level-82); NSFW age gating (viewer age < 18 β drop even with opt-in; logged-out β drop; no-stated-age in 16 gating countries [ar, au, br, ca, de, es, fr, gb, id, it, kr, mx, nl, ph, pt, th] β drop); NSFW tombstones (high-precision/gore/card/admin heuristics β Tombstone in 30-country age-verification/tombstone list or app-update prompts); age-prompt blurs (blur + AGE_VERIFICATION_PROMPT + SELFIE/PERSONA vs plain blur); exclusive (super-follow) content β drop unless viewer subscribes; interstitials for NSFW_HIGH_PRECISION / GORE_AND_VIOLENCE_HIGH_PRECISION / NSFW_CARD_IMAGE / nsfw-flagged author (when not opted-in).
Nov behavior changes in the rule set:
- FOSNR is follower-tiered.
FOSNR_ABUSE_INSULTSβ non-followers get aDrop(PossiblyUndesirable); followers get aSoftIntervention(notice + limited actions); the author sees their own post with anAppealablenotice carrying the policy (ABUSE / CIVIC_INTEGRITY / HATEFUL_CONDUCT / VIOLENT_SPEECH) and the level’s limited-action list (level-3 policy; abuse-insults level-1:EditTweetonly). AFosnrFallbackDropsclient switch adds a catch-all drop, and every FOSNR rule is gated by a client capability switch (freedom_of_speech_not_reach_rules_enabled) β old clients see different behavior. - Reaction-video propagation (
golden_corpus/reaction_video.rs): a post that reacts to a post labeled NSFW_HIGH_PRECISION / GORE_AND_VIOLENCE_HIGH_PRECISION inherits a media blur (sensitive/violence interstitial) for everyone, and is dropped if the reacted-to author is protected.Expansion::ReactedToplaces these rules on the hydration/promoted-hydration levels (they are not part of the plainTimelineHomeset). SOFT_NSFW(new label): OON posts with the label are dropped for new viewers when thesoft_nsfw_tweet_label_new_user_rule_enabledswitch is on β gated by a DDG experiment (soft_nsfw_ddg_1) and client capability; this is also the label the new home-mixerNewUserSoftNsfwFilterreads.- Label interstitials now distinguish public-interest notices (reason + optional limit β limited-engagement reason; e.g. civic misinfo limiting replies) from emergency dynamic copy (custom copy + link,
FOR_EMERGENCY_USE_ONLY). - Notifications are judged per kind: follow / follow-request-accepted (actor-level drops), like / repost (actor-on-recipient-post and, since Nov 19, the retweet source), mention / reply / media-tagged (actor post rules), and a new quote pass over the quoted/quoting post. Nov 19 makes the reply list identical to the mention list, adds
SPAM_HIGH_RECALL_OON/RISKY_HIGH_VIZ_REPLYdrops to media-tagged notifications, adds a notification-muted-keyword drop (with an excluding-followed-accounts partition), and lets an actor drop take precedence over a failed/not-found recipient-post evaluation. - Nov 19 audience cleanup:
DO_NOT_AMPLIFYandMALICIOUS_URLchanged fromNonAuthorstoNonFollowers. Both rules are listed only in the OON recommendation and immersive sets, where the candidate is by definition not from a followed account, so every viewer is already a non-follower and the change has no feed effect today; it matters only if the rules are reused on a surface that includes followers. Non-followers still getDrop(PossiblyUndesirable). - Nov 19 verdict merge: a retweet/source merge now picks the higher-severity withholding action rather than a special-case drop-vs-tombstone rule; restriction merges still take the source verdict.
- Nov 19 retweet media flag: the hydration patch that copied a source tweet’s
has_uploaded_mediaonto a merged retweet is removed. Media rules now see the retweet’s own flag, so a retweet edge can be treated differently from its media-bearing source (e.g., a text-only retweet entity of a media post).
OON-only drops: DMCA media; geo-restricted media (xx = worldwide); NSFW user/admin author flags; tweet nsfw flags; NSFW_HIGH_RECALL / NSFW_HIGH_PRECISION (OON) / NSFW_CARD_IMAGE / GORE_AND_VIOLENCE (OON); DO_NOT_AMPLIFY; MALICIOUS_URL; SPAM_HIGH_RECALL; NSFW_TEXT; FOSNR abuse-insults (non-followers only since Nov); DO_NOT_AMPLIFY and MALICIOUS_URL are also NonFollowers since Nov 19 (they were NonAuthors; both rules are OON-only, so this is semantic); plus 11 user labels (the table below):
| User label | Followers exempt? |
|---|---|
| NSFW_HIGH_RECALL, NSFW_HIGH_PRECISION, SPAM_HIGH_RECALL, COMPROMISED, READ_ONLY, IMPERSONATION_HIGH_PRECISION, NSFW_AVATAR_IMAGE, NSFW_BANNER_IMAGE, NSFW_NEAR_PERFECT | no (they are OON-only β allowed in-network β but OON they still drop followers) |
| ABUSIVE_HIGH_RECALL | yes (logged-in followers) |
| DO_NOT_AMPLIFY | yes |
Self-view exempt. Labels come from: the new shared safety-label-source/ crate (Nov) β per-label cache (10 s TTL), optional dragonfly tier, Manhattan fallback (jammer/safety_label_store/tweet_labels), a cache warmer, outage metrics, and feature-switch-driven cache controls β Nov 19: Dragonfly found entries 24 h / not-found 60 min (was 10 min), one 60 s refresh step (was 30/90 s), request deadlines with a 5 ms reply allowance, and stale entries served on any remote failure β plus gizmoduck user labels, TES (takedowns/edits), socialgraph, and the exclusive-content store. Label provenance includes GrokAnnotationAction{source} and BotmakerAction{rule_id}. Nov: VF also resolves client capabilities per request (app id + parsed user-agent version) to pick media-blur mode, stale-tweet/community limited-action support, and FOSNR rule sets β so the same post can get a different verdict on an old app than on a current one.
safety-label-user-agg/: post-level labels aggregate to user labels β POSSIBLY_NSFW_ACCOUNT (windowSize 10, minimumMatchingPosts 11 β as configured the rule is unsatisfiable, since only 10 posts are ever examined; 60-day max post/trigger age, 7d TTL; excludes high-PR/grey) and NSFW_HIGH_PRECISION (β₯3 of 5 posts) β written to gizmoduck by “Grox”.
scarecrow/: event-driven BotMaker rules apply tweet labels in real time (e.g., bot 3226: bad URL verdict β SPAM/SPAM_HIGH_RECALL on author’s tweet unless high-PR/gray), via AIS AddLabelsV2 with safetyLabelType; spam-label store TTL 26 h; per-rule rate limits (token bucket; limits come from runtime config β shipped rules carry empty actionLimits); HaltUntilHumanIntervention mode; rules β₯ 100,000 are transpiled. Sept: new XReviewIntake downstream + CreateXReviewReportProd/Staging units β BotMaker rules can now file proactive human-review reports (post/profile), not just labels.
12.6 What is NOT part of reputation (verified)#
- Verification: never read by feed scoring/ranking (gizmoduck hydrates no verification field). Only indirect: premium status feeds the user-cred teleport distribution (12.2) β affects enforcement exemption, not delivery. (Sept: the feed also now forwards an
nsfw_author_phoenixmodel signal β verification still absent.) - Engagement-per-impression ratio: not present; per-post counts hydrated and bucketed into the model (Sept: plus the log2-MLP count path in
home_direct_packed). - A standalone “author quality” feature: the model uses hashed author id only (plus SID, multimodal, counts) β though Aug 14 added the two follow-direction relationship flags (
IS_AUTHOR_FOLLOWED_BY_VIEWER,IS_AUTHOR_FOLLOWING_VIEWER) as model inputs (candidate + history), so “does the viewer know this author” is now part of the prediction itself, not just the +15 scoring boost. - bdsm/agatha/cred scores: not inputs to Phoenix or RankingScorer.
12.7 Bottom line: how reputation is actually determined (revised)#
- Explicit credibility (user-cred-v2): PageRank over follow + 7-day fav/retweet teleport; score 0-100 daily; premium-biased; exempts from enforcement β₯ 50, high-PR β₯ 54. This is the closest thing to a “reputation score” and it gates removal, not promotion.
- Labels: spam/abuse/NSFW/FOSNR labels (LLM + models + rules) β VF drops; follower-exemptions for two labels; post-level labels aggregate to account-level.
- Graph: follows (OON 0.75), mutual follows (+15 reply boost and, since Aug 14, follow-direction model features), blocks/mutes both directions, subscriptions, account age (new-user 0.00001), viewer age (VF NSFW gating).
- Learned author embeddings (unchanged analysis): 30M-author hash vocab, collision-diluted for small accounts; the candidate vector remains ~1 part author + post/age/SID/multimodal parts.
- Enforcement: model-score-driven rules can suspend, label (7d SpamHighRecall for
llm_slopsince Nov; 30d for the bot-label families), challenge, or cap at 500K/day per surface β and Nov’s head guards add an automatic circuit breaker at the head/classifier level.
12.8 The Learning Timescale (revised β training code is now open)#
The old section’s core finding (no training loop in the repo) is superseded: phoenix/xrex/ ships the production trainer. What we now know:
- Training: dense params via Muon since Sept (H100 lr 7.1e-4, GB300 5e-4; prod uses a tuned RMS-normalized-Adam derivative per TRAINING.md:14-17 for other surfaces); embeddings via sparse rowwise AdaGrad (lr 0.28, half-life 2500 H100 / 5000 GB300, lazy decay + small weight decay). Total data budget 1e11 samples; checkpoints every 150/300 steps with copy-port distribution (9988) to serving. (Plus a new StableHLO bundle-export path for rollout artifacts β no cadence change.)
- Feedback loop confirmed: home-mixer side effects publish every prediction (
PHOENIX_SCORES_TOPIC), 5%-sampled reranking scores (RERANKING_TOPIC), client events, served candidates β each engagement outcome is a training example for the next checkpoint. - Still unknown: production data windows, retrain/rollout cadence, learning-rate schedules in prod. The model has no author-side time dimension (unchanged): author embedding freshness = latest checkpoint. Post age (β€80h buckets) is the only post-time feature (timezone/hour-of-day features exist but are viewer-local context, not author-side) β plus
IS_STALE_POST14D, which zeroes engagement counts for posts > ~14 days old (now enabled inhome_direct_packed). - Quicksand verdict (updated): unchanged structurally (collision-diluted embeddings, post-heavy candidate representation, viewer-side 5-min UAS forgetting, content-based VF drops) β plus two recovery-friendly mechanisms: the cold-start boost actively promotes low-impression (β€50K-follower) authors’ posts toward slot 15, and credibility is re-computed daily from a 7-day engagement window, so enforcement shields can be earned in ~1 week and lost if behavior lapses. The Sept training changes strengthen this further: lazy decay + weight decay on the embedding optimizer (rows now shrink toward zero, not just decay history) β the author-embedding quicksand still depends on hidden retraining cadence.
13. Empirical Validation: 100-Profile Sample (x-filter-profile-stats-sample-1.json)#
13.1-13.2 Data, methodology, baseline β unchanged#
Baseline: median V/F 9.49% (Q1 3.81%, Q3 18.46%); median likes/view 1.22%. (Sample collected pre-Aug-2026 release; the scoring architecture these numbers exercise is the same RankingScorer family, now with exact weights.)
13.3 Finding 1: follower count negatively correlated with per-follower delivery β mechanism now quantified#
The code now gives exact numbers for the structural small-account advantages:
- Cold-start boost (author_cold_start.rs): accounts β€ 50,000 followers with posts < 200 Home-Timeline views and β€ 2h old (Oct rewrite; was β€1,000 / <1,000 / 24h / top-85%) ranked within the top 97% of the scored pool get a post lifted to slot 15 β a concrete, per-request promotion for small accounts. (Thompson sampling now on by default, top-2 β Section 5.3.)
- post_unexplored weight (0.015, in-network only) β extra credit for posts never seen.
- Hash-collision dilution (unchanged): 30M/100M author vocab, 2 hashes β small accounts average with collision partners (“average author” rather than “bad author”).
- New-user OON lockdown (0.00001) only applies to viewers, not authors.
- Follower count is never a Phoenix model input (Sept: the old
author_followers_countβ VMRanker request path is gone β the request carries no follower data at all) β though the Aug 14 commit added the relationship flagsIS_AUTHOR_FOLLOWED_BY_VIEWER/IS_AUTHOR_FOLLOWING_VIEWERas model inputs (Section 5.1; enabled inhome_direct_packedsince Sept): count, no; relationship, yes. (Sept caveat: on retweets the model now receives neither flag nor any follower count.)
So the β0.404 correlation and 22.86% (100-1K bucket β a single account) vs 3.65% (1M+) delivery gap are consistent with: no follower feature in the model + cold-start promotion for β€1K + follower-base quality effects.
13.4 Finding 2: the delivery ceiling β mechanism now identifiable#
dawn_com (1.38M followers, medV/F 0.065%, best post 0.52%) is structurally ineligible for every small-account mechanism:
- Cold-start requires β€ 50,000 followers (Oct; was 1,000) β 1.38M is still 28Γ over the cap.
- post_unexplored is in-network-only and tiny (0.015).
- The learned author embedding for a 1.38M-follower account is essentially “personal” (few collision partners) β low engagement propensity embeds permanently in its own bucket.
- R/V ~0.007% (near-zero replies) starves the highest positive weight in the system (reply 5.0, vs like 0.708).
The ceiling is the absence of every countervailing mechanism: the author embedding is a persistent negative and nothing in the pipeline (except a single VF-free, DPP-surviving viral post) can override it. The one-escape observation (drdrew’s 1.21M-view post) matches the design: a strong post-level prediction (post embedding + age + SID + multimodal) can outscore the author contribution because the candidate vector is post-heavy.
13.5 Finding 3: format effects β the ~48Γ dawn_com vs zerohedge gap#
Unchanged interpretation; now with weight evidence: reply weight 5.0 (vs like 0.708) quantifies why R/V discriminates (~0.007% vs 0.027% vs 0.13%); dwell_time (0.0075, capped ~0.225) modestly favors the article posts in the sample (collector flag IsArticle, bit 2 β median V/F 25.95% vs 9.65% baseline; n=37, 23 from one account; “longer dwell” is an inference, not a measured signal β see Section 1.F) while link posts’ low dwell/click-through starve cont_dwell_time and open_link (0.2) and share_via_copy_link (20.0) requires sharing β rare. VQV gate confirmed at 10 s. Sept: the VQV scoring weight is 0.0 and the dwell probability head is 0.05 β “watch time” now earns only the continuous dwell term plus the dwell-prob term, not VQV. Nov: video_open also fell to 0.0, so no format earns a direct video-open score term; click-dwell (0.385) is the large continuous term.
13.6 Finding 4: engagement rates don’t explain delivery β unchanged, weight-reinforced#
mmpadellan (L/V 6.09%, RT/V 1.25%, medV/F 3.78%) vs nikitabier (0.76%, 0.05%, 81.64%): the code confirms aggregate counts enter the model only as bucketed post features and the author embedding is trained on unobserved per-viewer outcomes. The observable-rate inversion remains unexplained by any shipped feature β the causal state (per-viewer propensity, dwell patterns, velocity) is still unobservable in aggregate data.
13.7 Finding 5: verification has zero effect β still true for delivery, with a nuance#
Unverified accounts outperform verified in every bucket from 1K to 1M (9.79% vs 18.76% at 1K-10K; 13.57% vs 17.11% at 10K-100K; 7.24% vs 10.08% at 100K-1M) β with one caveat: the only 1M+ unverified account (rt_com, 1.09%) trails the verified median (4.57%), so “every bucket” is three of four and the exception is a sample of one. The feed path still never reads is_blue_verified (gizmoduck hydrates no such field; model input has no verification feature). Nuance: user-cred-v2’s teleport distribution is uniform over premium users (blue/gray/gold/org) β so verification influences enforcement exemption (cred score, high-PR β₯ 54), which is a removal-shield, not a delivery boost. The empirical delivery finding stands.
13.8 Finding 6: virality is OON distribution β updated#
The OON layer now has five arms: Phoenix retrieval (two-tower over SID + embeddings; index windows per favorite threshold β 1fav_1day β¦ evergreen_video_5yr, plus age-banded 1fav_video_* slices and 1fav_reply/32fav_reply windows, rankall), SimClusters (min score 0.5, 800 max β cluster-similarity reach), PhoenixTopics/MoE, PopularPosts (on by default), and SidSource (off). The 50-2000Γ-followers viral tails match any of these. Note the rankall windows: a post needs β₯1 fav to enter the 1-day index and β₯32 favs to add the 32fav_1day pool β favorites only (20 favs + 100 replies opens no window) β a minimum fav threshold for OON candidacy, not a guarantee of serving (posts below 1 fav live only in post_creation_1day/video/tail windows β and since Sept, fav processing emits 1fav unconditionally, closing the 0-fav race gap). Nov: FairShareRanking additionally guarantees each discovery source a proportional share of the 2,800 scored candidates β under the old first-N truncation, the 3,300+ candidates from Thunder/PopularPosts/TweetMixer/Simclusters could crowd the Phoenix retrieval arms out of scoring entirely.
13.9 Finding 7: author-diversity decay β values now known#
AuthorDiversityDecay = 0.5, Floor = 0.25, k-th post multiplier (1β0.25)Β·0.5^k + 0.25 (2nd post Γ0.625, 3rd Γ0.4375). Still per-request, per-viewer slot allocation β not observable in aggregate view counts; unchanged conclusion. New related mechanisms also per-viewer/invisible in this dataset: DPP diversity (per-request embeddings), cold-start slot lift, bidirectional boost.
13.10-13.11 Observation β mechanism map (updated) + caveats β otherwise unchanged#
New map entries:
- Small-account delivery advantage β cold-start boost (Oct: β€50K followers, <200 Home-Timeline views, β€2h β slot 15; was β€1K/<1K/24h) + collision dilution + no follower feature in model
- dawn_com ceiling β ineligible for cold start (1.38M followers); personal (undiluted) negative embedding; near-zero reply signal vs reply weight 5.0
- Articles best format β no type signal exists (Section 1.F): the sample’s article posts over-deliver (25.95% vs 9.65% V/F) but that’s 37 posts, 23 from one small account β plausibly cold-start + content dynamics, not a format boost; dwell bonus (0.0075, capped ~0.225) + no VQV gate applies to all text-like posts
- Verification neutrality β no verification feature anywhere in feed path; premium only shapes cred teleport (enforcement shield)
- Viral tails β rankall index thresholds (β₯1 fav 1-day; β₯32 favs hot β favorites only; video windows to 5y) + SimClusters + Phoenix retrieval (membership is eligibility β serving still has to be won per viewer)
Caveats: dataset unchanged; all timing/positional effects in 13.9 remain age artifacts; causality caveat in 13.6 stands.
This analysis is based on the Aug 13, 2026 open-source release (commit a389166) plus the Aug 14, 2026 follow-up (commit c65aa17), the Sept 2, 2026 state (commit 85ac72a), the Oct 13, 2026 state (commit e1a0375), and the Nov 7, 2026 state (commit 3b6919f, 19 commits Oct 14βNov 7: weights to favorite 0.708 / click 0.274 / share 2.118 / cont_dwell 0.0075 / cont_click_dwell 0.385 / not_interested β53.07 and video_open 0.0; post_video + grok_analyze heads; Brazil filter deleted; 21 filters incl. ZeroFavCodivert, NewUserSoftNsfw, and FairShareRanking on; three new scorers incl. page-decode RL; EnableHomeExcursionScores on; carousel rebuilt; VMRanker user-debias + carousel model + top_heads + topic-DPP opt-out; VF action-model rewrite (9 levels / 7 action kinds / FOSNR follower split / reaction video / notifications / client switches) + safety-label-source/; Phoenix time-gap + excursion-history features, candidate-only layers, FP8, view head; gen-recs/SID-retrieval deleted; page-RL models; enforcement head guards/confidence/backfill + bb1_*; Under-the-Hood top-3 score contributions; and the Nov 19, 2026 state (commit b551517, 8 commits Nov 10β19: no weight changes; multiplicative value model + per-class factors + DPP sliding window all off; ads blender default slot_spread; teacher-score side effect on; Phoenix ids-only requests; post int64 width 17β22; six AI-video SID windows; page_critic_only; many-media PTOS + Promotes* labels; 12 post-label page actions; DO_NOT_AMPLIFY/MALICIOUS_URL now non-follower drops)). Unlike the July release, the scoring weights, feed sizes, age limits, history windows, and visibility-filter rules are now open; the remaining unknowns are production feature-switch values (rust_home_mixer.yml), the server-side value-model compute/pacing toggles, bdsm thresholds including the count gate, and the Phoenix training data windows/cadence.