{"data":{"protocol":{"package":"@cambridge-tcg/data-ingest","doctrine":"docs/methodology/source-protocol.md","catalog":"docs/connections/the-tributaries.md","audit_command":"pnpm audit:tributaries"},"counts":{"shipped":0,"partial":6,"planned":0,"blocked":5,"registered_total":11,"planned_slot_total":10},"ingest_runs_available":true,"sources":[{"id":"scryfall","name":"Scryfall","description":"Magic: The Gathering catalog and images through a policy-governed public API and bulk files. Value-added free use is permitted; raw republishing is not.","upstream":"https://scryfall.com","catalog_section":"the-tributaries.md#31-scryfall-mtg","access":"public-api","license":"proprietary","redistribute":false,"freshness":"catalog","canonical_effort":"low","status":"partial","games":["mtg"],"tos_notes":"Scryfall's API policy permits free value-added Magic software but forbids paywalling card data and simply repackaging, republishing, or proxying it. Magic material remains Wizards-owned. Use endpoint-specific limits and bulk files for large reads. https://scryfall.com/docs/api#use-of-scryfall-data-and-images ; https://scryfall.com/docs/api/rate-limits ; https://scryfall.com/docs/api/bulk-data","user_agent_suffix":"(scryfall-ingest)","rate_limit":{"rps":5,"burst":10},"last_run":{"_unavailable":true,"reason":"never_run"},"member_pricing":{"reviewed_uses":["member-display"],"cadence":"daily","evidence_url":"https://scryfall.com/docs/api#use-of-scryfall-data-and-images","methodology":"/methodology/member-pricing","collection":{"flag":null,"scheduled":false},"availability_note":"Reviewed uses are not activation or collection evidence. Runtime release permission is checked on each member read. cadence is the source's own publication rhythm (daily files, or live-page: a page's current value at the moment it is read), not a Cambridge collection schedule; Cambridge collects Cardmarket's One Piece files daily, CardRush product pages every two hours (one host per run) and TCGplayer price guides hourly (product lines in turn, each line's stalest sets first), as collection.scheduled shows; other imports are manual. This entry's last_run is legacy wholesale only; member observations carry their own source/retrieval clocks."}},{"id":"cardrush","name":"CardRush (JP)","description":"Japanese retail prices from CardRush product pages, under written permission recorded 2026-10-28. An identified page collector exists and reads them only while it is scheduled (DATA_COLLECT_CARDRUSH; /api/v1/sources says whether it is, in member_pricing.collection). What it collects is released for attributed member display and as an input to Cambridge-derived data only. The legacy archive, gathered before the permission, is frozen, withheld and not an input.","upstream":"https://www.cardrush-op.jp","catalog_section":"the-tributaries.md#23-cardrush-jp--page-collector-legacy-archive-frozen","access":"scrape","license":"internal-only","redistribute":false,"freshness":"page_collector","canonical_effort":"high","status":"partial","games":["op","pkm","dbf","dmw","vng","bsr"],"tos_notes":"Written permission held by Yu (email, not copied; recorded at docs/decisions/2026-10-28-tcg-data-permissions-and-paid-data-api.md#cardrush) covers collection from CardRush product pages, attributed member display and input to Cambridge-derived data. Without it, https://cardrush.media/data_policy prohibits automated crawling, scraping and price collection. robots.txt (www.cardrush.jp, cardrush.jp, www.cardrush-op.jp, and the pokemon, db, digimon, vanguard and bs hosts; 2026-10-28) disallows only GPTBot, Bytespider, TikTokSpider and meta-externalagent. Never member API, download or bulk. Rows gathered before 2026-10-28 are not inputs.","user_agent_suffix":"(cardrush-collector; permission cardrush/2026-10-28.1)","rate_limit":{"rps":0.5,"burst":1},"last_run":{"triggered_at":"2026-07-13 10:01:35.677432+00","finished_at":"2026-07-13 10:13:30.282+00","status":"done","spec_version":"1","rows_read":1504,"rows_normalized":1495,"rows_written":1495,"rows_quarantined":9,"errors":0,"age_hours":3083},"collector":{"hosts_available":true,"hosts":[{"host":"www.cardrush-op.jp","state":"ok","consecutive_refusals":0,"last_success_at":"2026-11-18T12:19:13.282Z","last_refusal_at":null,"last_refusal_reason":null,"not_before":null,"suspended_at":null,"suspended_reason":null},{"host":"www.cardrush-pokemon.jp","state":"suspended","consecutive_refusals":3,"last_success_at":null,"last_refusal_at":"2026-10-31T14:15:46.430Z","last_refusal_reason":"sitemap-challenge-challenge","not_before":"2026-10-31T20:15:46.430Z","suspended_at":"2026-10-31T14:15:46.430Z","suspended_reason":"sitemap-challenge-challenge"},{"host":"www.cardrush-db.jp","state":"ok","consecutive_refusals":0,"last_success_at":"2026-11-18T16:19:12.633Z","last_refusal_at":null,"last_refusal_reason":null,"not_before":null,"suspended_at":null,"suspended_reason":null},{"host":"www.cardrush-digimon.jp","state":"ok","consecutive_refusals":0,"last_success_at":"2026-11-18T18:19:13.833Z","last_refusal_at":null,"last_refusal_reason":null,"not_before":null,"suspended_at":null,"suspended_reason":null},{"host":"www.cardrush-vanguard.jp","state":"ok","consecutive_refusals":0,"last_success_at":"2026-11-18T20:19:12.477Z","last_refusal_at":null,"last_refusal_reason":null,"not_before":null,"suspended_at":null,"suspended_reason":null},{"host":"www.cardrush-bs.jp","state":"ok","consecutive_refusals":0,"last_success_at":"2026-11-18T10:19:12.541Z","last_refusal_at":null,"last_refusal_reason":null,"not_before":null,"suspended_at":null,"suspended_reason":null}]},"member_pricing":{"reviewed_uses":["member-display"],"cadence":"live-page","evidence_url":"repo:docs/decisions/2026-10-28-tcg-data-permissions-and-paid-data-api.md#cardrush","methodology":"/methodology/member-pricing","collection":{"flag":"DATA_COLLECT_CARDRUSH","scheduled":true},"availability_note":"Reviewed uses are not activation or collection evidence. Runtime release permission is checked on each member read. cadence is the source's own publication rhythm (daily files, or live-page: a page's current value at the moment it is read), not a Cambridge collection schedule; Cambridge collects Cardmarket's One Piece files daily, CardRush product pages every two hours (one host per run) and TCGplayer price guides hourly (product lines in turn, each line's stalest sets first), as collection.scheduled shows; other imports are manual. This entry's last_run is legacy wholesale only; member observations carry their own source/retrieval clocks."}},{"id":"pokemon-tcg-api","name":"Pokémon TCG API","description":"Pokémon TCG catalog adapter for sets, English printings, and image references. Upstream pricing subobjects are not a Cambridge partner feed. Paginated REST API at api.pokemontcg.io/v2.","upstream":"https://pokemontcg.io","catalog_section":"the-tributaries.md#32-pokémon-tcg-api-pokemontcgio","access":"public-api","license":"proprietary","redistribute":false,"freshness":"catalog","canonical_effort":"low","status":"partial","games":["pkm"],"tos_notes":"Public REST API; X-Api-Key is optional and raises limits. Current developer terms govern service use but do not grant an open license over Pokémon card data or images. Price and legality fields are informational only. https://docs.pokemontcg.io/ ; https://docs.pokemontcg.io/getting-started/authentication/ ; https://docs.pokemontcg.io/getting-started/rate-limits/ ; https://dev.pokemontcg.io/terms","user_agent_suffix":"(pokemon-tcg-api-ingest)","rate_limit":{"rps":1,"burst":5},"last_run":{"_unavailable":true,"reason":"never_run"}},{"id":"ygoprodeck","name":"YGOPRODeck","description":"Yu-Gi-Oh! — full card database including all printings, archetypes, prices via partner sourcing. Bulk endpoint cardinfo.php; no auth.","upstream":"https://db.ygoprodeck.com","catalog_section":"the-tributaries.md#33-ygoprodeck-yu-gi-oh","access":"public-api","license":"proprietary","redistribute":false,"freshness":"catalog","canonical_effort":"medium","status":"blocked","games":["ygo"],"tos_notes":"The API guide permits public consumption, asks clients to download/cache data, limits traffic to 20 requests/second, and forbids continual image hotlinking. It states no CC-BY or commercial data license; card text/images are identified as Konami/4K Media copyright. Cambridge requires written commercial permission before running this source. https://api.ygoprodeck.com/api-guide/","user_agent_suffix":"(ygoprodeck-ingest)","rate_limit":{"rps":1,"burst":3},"last_run":{"_unavailable":true,"reason":"never_run"}},{"id":"tcgplayer","name":"TCGplayer","description":"US market prices (TCGplayer's Market Price per condition and printing) from TCGplayer's public price guides. Written permission recorded 2026-10-28 covers collection from TCGplayer pages, including the data TCGplayer's public price-guide pages load (Yu confirmed 2026-10-29), attributed member display and input to Cambridge-derived data. An identified collector for the public price guides exists and reads them only while it is scheduled (DATA_COLLECT_TCGPLAYER; /api/v1/sources says whether it is, in member_pricing.collection). What it collects is released for attributed member display and as an input to Cambridge-derived data only. The OAuth API reader in this module stays locked.","upstream":"https://www.tcgplayer.com","catalog_section":"the-tributaries.md#21-tcgplayer-us-market-leader","access":"scrape","license":"internal-only","redistribute":false,"freshness":"page_collector","canonical_effort":"medium","status":"partial","games":["mtg","pkm","ygo","op","dbs","dbf","lgr","fab","dmw","vng","wei","bsr"],"tos_notes":"Written permission held by Yu (email, not copied; recorded at docs/decisions/2026-10-28-tcg-data-permissions-and-paid-data-api.md#tcgplayer) covers collection from TCGplayer pages, attributed member display and input to Cambridge-derived data. The price-guide pages are client-rendered, so the collector reads the data the public set price-guide page itself loads, one request per set; Yu confirmed on 2026-10-29 that the permission covers those requests. robots.txt (www.tcgplayer.com, 2026-10-28): User-agent *, Crawl-Delay 10, Allow /; seller listing paths are disallowed and never read; Crawl-Delay 10 holds across every TCGplayer host the collector reads. Product-level Market Price only; never sellers or listings. Without that consent the API terms prohibit combining TCGplayer prices with other prices and redistribution; new API access is closed and the API reader stays locked. https://help.tcgplayer.com/hc/en-us/articles/360061115874-TCGplayer-API-Terms-Conditions . Never member API, download or bulk.","user_agent_suffix":"(tcgplayer-collector; permission tcgplayer/2026-10-28.1)","rate_limit":{"rps":0.1,"burst":1},"last_run":{"_unavailable":true,"reason":"never_run"},"collector":{"hosts_available":true,"hosts":[{"host":"www.tcgplayer.com","state":"ok","consecutive_refusals":0,"last_success_at":"2026-11-18T20:49:48.730Z","last_refusal_at":null,"last_refusal_reason":null,"not_before":null,"suspended_at":null,"suspended_reason":null},{"host":"mpapi.tcgplayer.com","state":"ok","consecutive_refusals":0,"last_success_at":"2026-11-18T20:49:48.730Z","last_refusal_at":null,"last_refusal_reason":null,"not_before":null,"suspended_at":null,"suspended_reason":null},{"host":"infinite-api.tcgplayer.com","state":"ok","consecutive_refusals":0,"last_success_at":"2026-11-18T20:49:48.730Z","last_refusal_at":null,"last_refusal_reason":null,"not_before":null,"suspended_at":null,"suspended_reason":null}]},"member_pricing":{"reviewed_uses":["member-display"],"cadence":"live-page","evidence_url":"repo:docs/decisions/2026-10-28-tcg-data-permissions-and-paid-data-api.md#tcgplayer","methodology":"/methodology/member-pricing","collection":{"flag":"DATA_COLLECT_TCGPLAYER","scheduled":true},"availability_note":"Reviewed uses are not activation or collection evidence. Runtime release permission is checked on each member read. cadence is the source's own publication rhythm (daily files, or live-page: a page's current value at the moment it is read), not a Cambridge collection schedule; Cambridge collects Cardmarket's One Piece files daily, CardRush product pages every two hours (one host per run) and TCGplayer price guides hourly (product lines in turn, each line's stalest sets first), as collection.scheduled shows; other imports are manual. This entry's last_run is legacy wholesale only; member observations carry their own source/retrieval clocks."}},{"id":"tcgcollector","name":"TCGCollector","description":"Blocked TCGCollector acquisition adapter. Public indexing and Schema.org markup do not grant Cambridge commercial collection or mirroring rights; reopening requires written partner approval.","upstream":"https://www.tcgcollector.com","catalog_section":"the-tributaries.md#311-tcgcollector","access":"partner","license":"proprietary","redistribute":false,"freshness":"price_current","canonical_effort":"medium","status":"blocked","games":["pkm","pkp","mtg","op","ygo"],"tos_notes":"The terms at https://www.tcgcollector.com/legal/terms-of-service reserve API access for approved business partners and limit other materials to personal, non-commercial transitory use absent written agreement. Acquisition is hard-disabled because Cambridge has no recorded approval.","user_agent_suffix":"(tcgcollector-ingest)","rate_limit":{"rps":0.5,"burst":2},"last_run":{"_unavailable":true,"reason":"never_run"}},{"id":"cardmarket","name":"Cardmarket","description":"European market catalog and daily aggregate prices from Cardmarket's official files. The initial One Piece member-price batch was imported in production on 2026-09-08 UTC. A daily collector of the configured games (One Piece, as of 2026-10-28) exists and runs only while it is scheduled (DATA_COLLECT_CARDMARKET; /api/v1/sources says whether it is, in member_pricing.collection). Other game imports are not implied. Since 2026-10-28 these prices are also an input to Cambridge-derived data.","upstream":"https://www.cardmarket.com/en/Magic/Data","catalog_section":"the-tributaries.md#22-cardmarket-eu-market-leader","access":"public-file","license":"proprietary","redistribute":false,"freshness":"daily_file","canonical_effort":"medium","status":"partial","games":["mtg","pkm","ygo","op","lgr","fab","dmw"],"tos_notes":"Official Product Catalog / Price Guide statement says “use them however you see fit”; member-prices.ts records normalized member display/API/download permission with attribution. This is not CC0 or a site-wide license. Raw files stay private in this implementation. Yu's word of 2026-10-28 (docs/decisions/2026-10-28-tcg-data-permissions-and-paid-data-api.md#cardmarket) adds Cardmarket prices as an input to Cambridge-derived data (cardmarket-derivation/2026-10-28.1). robots.txt (2026-10-28): Content-Signal search=yes, ai-train=no, use=reference, with an EU DSM Art. 4 reservation; Cambridge-derived data from these prices is not CC0 and excludes AI training. API applications are closed, and existing credentials must not be shared. The API base for grandfathered users is apiv2.cardmarket.com. https://www.cardmarket.com/en/Magic/Data/Price-Guide ; https://www.cardmarket.com/en/Magic/Data/Product-List ; https://www.cardmarket.com/en/Insight/Articles/the-state-of-cardmarket-2024 ; https://help.cardmarket.com/en/cardmarket-api","user_agent_suffix":"(cardmarket-collector; permission member-prices/2026-09-07.1, cardmarket-derivation/2026-10-28.1)","rate_limit":{"rps":2,"burst":5},"last_run":{"_unavailable":true,"reason":"never_run"},"collector":{"hosts_available":true,"hosts":[{"host":"downloads.s3.cardmarket.com","state":"ok","consecutive_refusals":0,"last_success_at":"2026-11-18T05:30:14.772Z","last_refusal_at":null,"last_refusal_reason":null,"not_before":null,"suspended_at":null,"suspended_reason":null}]},"member_pricing":{"reviewed_uses":["member-display","member-api","member-download"],"cadence":"daily","evidence_url":"https://www.cardmarket.com/en/Insight/Articles/the-state-of-cardmarket-2024","methodology":"/methodology/member-pricing","collection":{"flag":"DATA_COLLECT_CARDMARKET","scheduled":true},"availability_note":"Reviewed uses are not activation or collection evidence. Runtime release permission is checked on each member read. cadence is the source's own publication rhythm (daily files, or live-page: a page's current value at the moment it is read), not a Cambridge collection schedule; Cambridge collects Cardmarket's One Piece files daily, CardRush product pages every two hours (one host per run) and TCGplayer price guides hourly (product lines in turn, each line's stalest sets first), as collection.scheduled shows; other imports are manual. This entry's last_run is legacy wholesale only; member observations carry their own source/retrieval clocks."}},{"id":"ebay","name":"eBay","description":"eBay marketplace ingest. v0 ships Browse API (current asks) on EBAY_GB; the same SourceModule branches to Marketplace Insights (90-day sold-comp history) once partner application is approved. Watch-list-driven: walks operator-curated SKU list per run, cross-checks parsed-title-SKU against expected SKU before write. Substrate-honest about sku-drift, low-confidence parses, and excluded conditions (damaged / counterfeit / proxy / sealed-bundle).","upstream":"https://api.ebay.com","catalog_section":"the-tributaries.md#25-ebay-full-marketplace-not-just-order-import","access":"oauth2","license":"partner-redistributable","redistribute":false,"freshness":"market_signal","canonical_effort":"very-high","status":"partial","games":[],"tos_notes":"eBay developer license: data licensed for partner-display use, not bulk redistribution. PWCC (eBay Vault) data carries additional restrictions. See https://developer.ebay.com/develop/apis/api-license-agreement. Marketplace Insights API is Limited Release — partner application + category whitelist required.","user_agent_suffix":"(ebay-comps-ingest)","rate_limit":{"rps":5,"burst":20},"last_run":{"_unavailable":true,"reason":"never_run"}},{"id":"bandai-en","name":"Bandai EN cardlists","description":"Internal parser for proprietary Bandai English cardlist fields. One Piece and DBS Fusion World fixture parsing is implemented; production ingest and all public publication are paused pending documented permission.","upstream":"https://en.onepiece-cardgame.com","catalog_section":"the-tributaries.md#bandai-en-official-english-cardlists","access":"scrape","license":"proprietary","redistribute":false,"freshness":"catalog","canonical_effort":"medium","status":"blocked","games":["op","dbf","dmw","una","bsr"],"tos_notes":"Bandai content is proprietary and redistribution is false. No written Cambridge permission is recorded for collection, storage, display, hotlinking, mirroring, or redistribution. Missing robots rules observed for One Piece on 2026-07-11 and DBS Fusion World on 2026-07-13 are not permission. The planned Cambridge image bucket does not exist. Production ingest and public readers are paused; parser tests use local fixtures. Policy: docs/EN-CARD-DATA.md.","rate_limit":{"rps":0.5,"burst":1},"last_run":{"_unavailable":true,"reason":"never_run"}},{"id":"bandai-jp","name":"Bandai JP official sample images","description":"Image addresses of official Japanese sample images from Bandai's One Piece and Fusion World card lists, for self-hosting with credit under Yu's decision of 2026-10-30. A manual tool only: no cron and no live reader.","upstream":"https://www.onepiece-cardgame.com","catalog_section":"the-tributaries.md#312-bandai-jp-official-sample-images","access":"scrape","license":"proprietary","redistribute":false,"freshness":"catalog","canonical_effort":"low","status":"blocked","games":["op","dbf"],"tos_notes":"Footers, read 2026-10-30: www.onepiece-cardgame.com 「このwebサイトに記載されているすべての画像・テキスト・データの無断転用、転載をお断りします。」; www.dbs-cardgame.com/fw/jp 「※当サイトに掲載されている画像、文章、データ等の無断使用・複製・転載を禁じます。」 Both refuse reproduction without permission, and no Bandai permission is recorded. Yu decided on 2026-10-30 to self-host these images with credit and takedown, under their 2026-07-13 rationale for English (docs/decisions/2026-10-30-japanese-official-images.md). robots.txt answered 404 on both hosts; the tool reads it every run, as User-Agent CambridgeTCG-official-media/1.0 naming that decision, at least 2 s apart per host. Reading the lists and the S3 upload each need Yu's word for that run. Display on Cambridge pages, the universal card API (image_same_language) and the JSON behind the market pages; never in exports, the member API or /api/v1/market/*.","rate_limit":{"rps":0.5,"burst":1},"last_run":{"_unavailable":true,"reason":"never_run"}},{"id":"vinted","name":"Vinted (UK)","description":"UK secondhand marketplace with a trading-cards vertical. No public or market API; sold listings are hidden from search and third-party 'sold' feeds are snapshot-diff inference, not transactions. Market-wide ingest is BLOCKED by ToS + UK GDPR. The only lawful path is consented first-party — a seller's own sales export (normalizer ready in ./normalize.ts, buyer PII structurally excluded).","upstream":"https://www.vinted.co.uk","catalog_section":"the-tributaries.md#211-vinted-uk","access":"blocked","license":"internal-only","redistribute":false,"freshness":"market_signal","canonical_effort":"very-high","status":"blocked","games":[],"tos_notes":"ToS (vinted.co.uk/terms_and_conditions, England & Wales law) §6 prohibits: using 'any kind of external software tools (bots, scraping programs, crawling programs, spiders)' unless authorised; 'data mine, screen scrape, crawl, disassemble, decompile or reverse engineer any part of the Site'; and 'adapt, copy, edit, distribute or commercialise any content on the Site without our prior written consent'. robots.txt (fetched 2026-07-11) permits generic catalog crawl but reserves rights under DSM Directive 2019/790 Art. 4 against dataset creation, and Disallows ClaudeBot / GPTBot / CCBot entirely; DataDome anti-bot is deployed. No sold-price endpoint exists anywhere; third-party 'sold data' is last-asking-price inference. UK GDPR: {username, price, date} is personal data; EDPB Guidelines 03/2026 Example 5 — scraping a prohibiting, anti-bot-protected platform fails the legitimate-interests test. Lawful path only: consented first-party seller export / authorised Vinted Pro Orders (pro-docs.svc.vinted.com, allowlist, own-orders-only); buyer-side PII dropped at the ingest boundary. Not legal advice — see docs/methodology/source-intake.md and get solicitor review before any launch.","user_agent_suffix":"(vinted-blocked-no-fetch)","last_run":{"_unavailable":true,"reason":"never_run"}}],"planned_slots":["cardtrader","limitless-tcg","edhrec","bandai-tcg","psa-registry","beckett-registry","shopify","stripe","ctcg-wholesale-rds","ctcg-storefront-rds"],"conventions":{"license_tiers":"cc0 / cc-by / cc-by-nc / cc-by-sa / mit / partner-redistributable / internal-only / proprietary. `redistribute: true` is only valid with cc0/cc-by/cc-by-sa/mit; the audit (pnpm audit:tributaries check 7) enforces this.","access_methods":"public-api / public-file / app-token / oauth2 / oauth1 / scrape / partner / paid-feed / blocked.","freshness_keys":"catalog (24h) / price_current (5min) / price_historical (immutable) / market_signal (1min) / status (30s) / methodology (24h) / identity (1h) / adopters (24h) / directory (10min) / live (5s) / rotating (5min) / page_collector (6h) / daily_file (26h). See packages/data-spec/src/freshness.ts.","source_license_propagation":"When a response is composed from multiple sources, _meta.sources lists them in contribution order and _meta.source_license (optional, when declared) carries each one's redistribution tier. Absence is substrate-honest; presence is the platform's declaration."}},"_meta":{"spec_version":"1","endpoint":"/api/v1/sources","retrieved_at":"2026-11-18T21:01:41.507Z","as_of":"2026-11-18T21:01:41.507Z","sources":["ctcg-derived","wholesale-rds.ingest_run","storefront-rds.price_collector_hosts"],"freshness_seconds":30,"license":"NOASSERTION","request_id":"req_0491df8e-29b","deprecation":null,"next_link":null,"self_reference":{"this_endpoint":"/api/v1/sources","contains_self":true},"source_license":["proprietary","internal-only","internal-only"],"does_not_include":["Ingest-run notes and trigger labels are operator-only and are not returned here.","Quarantine reasons, rows, payload keys, and payload bytes are not returned here."]}}