Formats and delivery
The proposed delivery specification uses newline-delimited JSON (JSONL) or Apache Parquet, partitioned by board and time period where agreed. Records are designed to use stable identifiers so that snapshots and incremental deliveries can be joined and deduplicated.
Delivery method, partitioning, update cadence, filtering and final field list are agreed during scoping. The examples below are illustrative; your export schema is confirmed before delivery.
What is standard, agreed, or illustrative
- — Standard where confirmed for your engagement: UTF-8 output, agreed field list, delivery manifest with file list and checksums.
- — Agreed per engagement: board and date scope, redaction profile, update cadence, delivery channel (object storage, scheduled export, delta feed, or dedicated endpoint).
- — Illustrative on this page: example JSON records, field tables and code snippets. These show the proposed schema, not a guarantee of any particular export without scoping.
Post record schema (proposed)
Illustrative example record. Final fields, normalisation rules and redaction profile are agreed during scoping.
{
"post_id": "5ch:news4vip:1712345678:0042",
"thread_id": "5ch:news4vip:1712345678",
"board_id": "news4vip",
"board_name_ja": "ニュー速VIP",
"board_category_ja": "雑談系",
"post_index": 42,
"posted_at": "2024-04-05T21:14:38+09:00",
"posted_at_precision": "second",
"name_field": "以下、5ちゃんねるからVIPがお送りします",
"poster_id": "aB3xY9Zk0",
"body_text": "そういうのはこっちのスレでやってくれ\n>>38 が言ってたやつな",
"body_normalized": "そういうのはこっちのスレでやってくれ >>38 が言ってたやつな",
"reply_to": [38],
"quoted_spans": [
{ "target_post_index": 38, "raw": ">>38" }
],
"char_count": 34,
"contains_ascii_art": false,
"redaction_profile": "standard",
"redactions_applied": 0,
"source_snapshot_id": "snap-2026-08-01",
"record_version": 1
}| Field | Type | Description |
|---|---|---|
| post_id | string | Globally stable post identifier |
| thread_id | string | Parent thread identifier |
| board_id | string | Board short code |
| board_name_ja | string | Board display name |
| board_category_ja | string | Board category grouping |
| post_index | int | 1-based position within thread |
| posted_at | string (RFC 3339) | Post timestamp, JST, timezone-explicit |
| posted_at_precision | enum | second | minute | day — source precision varies by era |
| name_field | string | Name field as submitted |
| poster_id | string | null | Source-provided poster ID where present |
| body_text | string | Post body, source form, newlines preserved |
| body_normalized | string | NFKC-normalised, whitespace-collapsed variant |
| reply_to | int[] | Resolved reply targets by post_index |
| quoted_spans | object[] | Raw anchor text and resolved target |
| char_count | int | Character count of body_text |
| contains_ascii_art | bool | Heuristic ASCII-art flag |
| redaction_profile | enum | raw | standard | strict — see licensing page |
| redactions_applied | int | Count of redactions in this record |
| source_snapshot_id | string | Snapshot this record was exported from |
| record_version | int | Schema version of this record |
Thread record schema (proposed)
Illustrative example record. Thread coverage and edge resolution depend on the agreed scope and source completeness.
{
"thread_id": "5ch:news4vip:1712345678",
"board_id": "news4vip",
"title": "日本語の口語表現について語るスレ",
"title_normalized": "日本語の口語表現について語るスレ",
"created_at": "2024-04-05T20:47:58+09:00",
"last_post_at": "2024-04-06T03:22:11+09:00",
"post_count": 412,
"unique_poster_id_count": 87,
"status": "archived",
"reply_edges": [[42, 38], [43, 42], [44, 38]],
"source_snapshot_id": "snap-2026-08-01",
"record_version": 1
}reply_edges lists explicit [from_post_index, to_post_index] pairs derived from >>N anchors in the source. This captures explicit reply links, not every conversational relationship. Missing context may occur where anchors are absent or ambiguous.
| Field | Type | Description |
|---|---|---|
| thread_id | string | Thread identifier |
| board_id | string | Board short code |
| title | string | Thread title |
| title_normalized | string | NFKC-normalised title |
| created_at | string (RFC 3339) | Thread creation timestamp |
| last_post_at | string (RFC 3339) | Last post timestamp |
| post_count | int | Total posts in thread |
| unique_poster_id_count | int | Unique poster IDs |
| status | enum | active | archived | capacity-closed |
| reply_edges | int[][] | [from_post_index, to_post_index] pairs |
| source_snapshot_id | string | Snapshot this record was exported from |
| record_version | int | Schema version |
Encoding and normalisation (proposed)
- — Output is UTF-8. Legacy source encodings are converted at parse time with a documented mapping; conversion failures are recorded rather than silently dropped.
- — body_text preserves the source form, including full-width characters, kaomoji, and intentional orthographic variation.
- — body_normalized provides an NFKC-normalised, whitespace-collapsed parallel field where agreed. Both may be delivered; consumers choose.
- — Reply anchors (>>N, ＞＞N, and historical variants) can be parsed into structured fields while the original text is retained in body_text.
- — Timestamps are RFC 3339 with explicit JST offset. Where source precision is coarser than one second, posted_at_precision records this rather than fabricating precision.
- — Identifiers are designed to be stable across snapshots within a defined export. A record's post_id does not change between deliveries of the same export definition.
Snapshots and incremental delivery (where agreed)
A snapshot is a point-in-time export of a defined scope (boards, date range, filters), identified by source_snapshot_id and accompanied by a manifest listing every file, its record count, and its SHA-256 checksum.
A delta is the set of records added, modified, or withdrawn between two snapshots, expressed as typed change records so that a downstream store can be brought forward without a full reload.
Deletion and correction events in the source can be propagated through delta feeds where the licence provides for it. Availability of incremental updates is confirmed during scoping.
# JSONL
import json
with open("posts.jsonl", encoding="utf-8") as f:
posts = [json.loads(line) for line in f]
# Parquet
import pyarrow.parquet as pq
table = pq.read_table("posts.parquet")
df = table.to_pandas()
# Hive partitions by board and month
part = pq.read_table("5ch_posts/board_id=news4vip/year=2026/month=08/")
# Reconstruct a thread with its reply graph
thread_id = posts[0]["thread_id"]
thread = [p for p in posts if p["thread_id"] == thread_id]
thread.sort(key=lambda p: p["post_index"])
edges = [(p["post_index"], t) for p in thread for t in p["reply_to"]]Delivery options
- Encrypted object storage
- Signed download URLs to encrypted shards with manifest and checksums — Standard option
- Scheduled snapshots
- Periodic full exports of a defined scope — Agreed per engagement
- Incremental delta feed
- Typed change records between snapshots — Agreed per engagement
- Dedicated access endpoint
- Programmatic access scoped to the licence — Agreed per engagement
- Private repository delivery
- Delivery to a customer-controlled private repository — Agreed per engagement
| Item | Notes |
|---|---|
| Encrypted object storage | Signed download URLs to encrypted shards with manifest and checksums — Standard option |
| Scheduled snapshots | Periodic full exports of a defined scope — Agreed per engagement |
| Incremental delta feed | Typed change records between snapshots — Agreed per engagement |
| Dedicated access endpoint | Programmatic access scoped to the licence — Agreed per engagement |
| Private repository delivery | Delivery to a customer-controlled private repository — Agreed per engagement |
Request a technical preview.
Tell us your boards, date range and intended uses. We will confirm scope and next steps for a preview or paid evaluation.