Sources & ingestion

To add sources to a knowledge base, go to Knowledge Bases → Sources and choose Upload, or choose Upload in a knowledge base's Sources section. The Add sources dialog has three tabs: Upload, Import and Add URL.

1. Choose the Knowledge base. When you start from a knowledge base, it's already set.
2. Check the Ingestion preset. It starts as the knowledge base's own preset, including the settings saved on the knowledge base. Another preset applies its own settings as they are, to these sources only.
3. Add files, select items to import, or type a URL.
4. Choose Start ingestion, Import selected or Add URL.

Each source gets its own ingestion job and joins the list as Queued. Adding sources needs the Upload sources permission, which every built-in role except Viewer has. Without it, Upload is disabled.

Grey ingests each source in the background, in four stages: Parse reads it, Chunk splits it, Embed turns the chunks into vectors and Graph extracts entities for the knowledge graph when the settings ask for it. A source's status shows where it is:

• Queued: waiting for its ingestion job to start.
• Processing: being ingested.
• Ready: its chunks can be searched.
• Failed: its last job didn't finish. See A source shows Failed.
• Pending: its job was canceled before it started, and it has no chunks yet.

When a failure is temporary, such as a provider timeout, Grey tries the job again on its own, up to three attempts in all. The source list and Ingestion Jobs don't update on their own: open the screen again to see the latest status.

On the Upload tab of Add sources, drop files on the box or click it to choose them. You can add several files at once.

Grey looks at each file as you add it, then shows its size, kind, page count or length, and an estimated ingestion cost from the preset's prices. The total appears under the list. Remove a file with the × beside it. Start ingestion uploads the files one at a time and queues each one; a file that fails to upload is reported and skipped.

Grey reads PDF, Word (.docx), Excel (.xlsx), PowerPoint (.pptx), OpenDocument (.odt, .ods and .odp), RTF, EPUB, HTML, Markdown, CSV, TSV, JSON and plain text files, as well as images, audio and video. Older Office files (.doc, .xls and .ppt) can't be read: save them as .docx, .xlsx or .pptx and upload them again.

The Import tab of Add sources lists what the same project holds in other Geiger apps, so you can ingest it without downloading and uploading it again:

• Assets: the project's stored documents, PDFs, images, audio and video in Geiger Assets. Other asset types aren't offered.
• Notes: the project's boards in Geiger Notes.

Select the items you want, then choose Import selected. Each one becomes a source whose origin is Assets or Notes, and Grey reads it from that app when the job runs. If you delete the board in Notes, re-ingesting its source fails.

On the Add URL tab of Add sources, type the page's address, like https://example.com/article, and choose Add URL. Grey fetches the page when the job runs and indexes its main content. The source is named after the address.

• Only http and https addresses work, and they can't include a username or password.
• Addresses on a private network are refused.
• A page can be up to 20 MB.
• Links to audio or video files don't work: upload the file, or import it from Assets.

Grey keeps what it fetched. To pick up changes to the page, re-ingest the source.

Knowledge Bases → Sources lists the sources in every knowledge base in the project. The figures across the top count all Sources, then those that are Ready, In progress (queued or processing) and in Needs review, whose review date is near or past.

Each row shows the source's name and origin (Upload, Assets, Notes or URL), its knowledge base, its size (pages, length or file size), status, freshness and when it was added. Filter by status, kind (Document, Audio, Video, Image, Web page or Note), origin and knowledge base, or search by source or knowledge base name.

Click a row to open the source's details. Each row's ⋯ menu has Open, Re-ingest, Mark verified, Owner & review date, Download and Delete; actions you don't have permission for don't appear. The list shows 25 sources a page; switch to 10, 50 or 100 at the bottom.

Click a source to open its details beside the list. The top shows its kind, origin, status and freshness.

• Audio and video from uploads or Assets play right there.
• Metadata: size, file size, tokens, chunks, file type, when it was uploaded and verified, its review date and owner.
• Ingestion jobs: every job run for the source, with its stage and status.
• Mentioned entities: the knowledge graph entities found in it.

The buttons at the bottom are Download (uploaded files only), Re-ingest, Mark verified, Owner & review date and Delete. Buttons you don't have permission for are disabled.

Freshness tracks whether someone has checked that a source is still current. Each source shows one of four badges:

• Verified: it was marked verified, and no review is due in the next 7 days.
• Review due: its review date is within the next 7 days.
• Stale: its review date has passed.
• Not verified: nobody has verified it, and no review is due in the next 7 days.

Marking a source verified on or after its review date clears Stale and Review due.

To record a check, choose Mark verified in the source's ⋯ menu or details. To set when it's next due, choose Owner & review date, pick a Review due date and choose Save; clear the date to remove it. Owner (user ID) is optional. Both actions need the Upload sources permission.

Freshness is for your team's own tracking: it doesn't change what pipelines search.

Re-ingest, in a source's ⋯ menu or details, queues a new ingestion job with the knowledge base's current preset and settings. The source keeps its existing chunks until the new ones are ready. Re-ingesting needs the Run ingestion jobs permission. To re-ingest every source in a knowledge base at once, use Re-ingest all in its ingestion settings.

Delete asks you to confirm first. The source's chunks and ingestion history are removed from retrieval, and a deleted source can't be restored from Grey. Deleting needs the Delete sources permission, which Owners, Admins and Managers have.

Knowledge Bases → Ingestion Jobs lists every ingestion job in the project, newest first. The figures across the top count jobs Queued and Running, jobs that failed in the last 7 days, and the estimated spend on jobs from the last 30 days.

Each row shows the job's source, knowledge base, preset tier, current stage, progress, status and cost: the actual cost once it's known, or the estimate marked with ~ until then. Filter by status (Queued, Running, Succeeded, Failed or Cancelled) and by knowledge base.

Each row's ⋯ menu has Retry (new job), which queues a fresh job with the same settings, and Cancel for a job that's queued or running. A running job stops at its next checkpoint. Both need the Run ingestion jobs permission. The list shows 25 jobs a page; switch to 10, 50 or 100 at the bottom.

A source fails when its ingestion job can't finish after its attempts. These are the usual causes, by kind:

• Images: Grey needs a vision-capable processing model in Settings → General, or a knowledge base whose embedding model can embed images.
• Audio and video: Grey needs a transcription connection (set in Settings → General, or your only one), and Transcribe turned on in the knowledge base's settings. Speech-free files fail too.
• PDFs: a scanned PDF with no text layer can't be read.
• Other files: older Office formats, and damaged or password-protected files, can't be read.
• Web pages: the page couldn't be reached, timed out, returned an error, or is over 20 MB.
• Any source: the knowledge base has no embedding connection, or the connection's API key stopped working.

Fix the cause, then choose Re-ingest on the source. The Project settings guide covers the processing and transcription models.