> For the complete documentation index, see [llms.txt](https://docs.akamas.io/insights/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.akamas.io/insights/connecting-your-data/datasources/on-prem-extraction.md). # On-Prem Extraction Akamas Vivo normally connects directly to your observability platform and extracts metrics from its API. That requires Akamas to reach your monitoring endpoint over the internet. When this is not possible, for instance in an air-gapped system, you can still gather your data by running this self-contained extraction stack on a single Linux host inside your own network. The platform queries your monitoring stack locally, processes the results, and produces a set of Parquet files that you download and send to Akamas. Those files hold the metric values read from your monitoring platform together with the names of the Kubernetes objects they belong to — clusters, nodes, namespaces, workloads, pods and containers. Akamas ingests them, and the extraction then appears in your tenant exactly like any other data import. {% hint style="info" %} This On-prem platform is a **data collection** component only, not a self-hosted Akamas Vivo. Analysis, recommendations and reporting continue to run in Akamas Vivo; only the extraction moves into your network. {% endhint %} The stack is distributed as a release bundle: a single tarball containing the container images, an installer and a command-line tool. One bundle serves one monitoring platform. To request a bundle, contact or your sales representative. ## Requirements | Requirement | Details | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Operating system | **Linux x86\_64**. macOS and Windows are not supported. | | Container runtime | Docker Engine with the Compose v2 plugin (`docker compose version`). | | Command-line tools | `curl`, `openssl`, `tar`. | | Memory | **At least 8 GB of RAM.** If the host is a virtual machine, allocate at least 8 GB **to the VM**, not only to the hypervisor host. | | Disk | At least 3× the expected size of the raw dataset. Ask Akamas for a sizing estimate for your environment. | | Network — outbound | Access from the host to your monitoring platform's API. For a SaaS platform that means outbound HTTPS to the internet; for an endpoint inside your own network, no internet access is required at all. | | Network — inbound | The installer publishes a single port on the host: **8080**, the extraction API used by the command-line tool. Nothing else is exposed. | {% hint style="warning" %} Port 8080 is published on **every** network interface of the host, over plain HTTP, and Docker's port publishing bypasses the host firewall. The extraction API requires the API key the installer generates into `.env`, but the health, metrics and API-schema endpoints answer without it. Restrict who can reach port 8080 and if it has to be reached from another machine, put a TLS reverse proxy in front of it so the API key does not travel in cleartext. {% endhint %} ### Sizing The 8 GB figure is an estimate, calibrated around five medium-sized clusters over a timeframe of around 1 month at a granularity of 5 minutes. You may need more or less depending on the size of your fleet, the timeframe and the granularity: for instance a 3-month extraction over those same five clusters needs a 16 GB host. {% hint style="warning" %} You can contact Akamas for a sizing check. Exceeding the host's capacity surfaces as a container being terminated part-way through a long run. {% endhint %} ## Installing the Stack 1. Unarchive the bundle. ```bash tar xzf insights-onprem---.tar.gz ``` 2. Run the installer from inside the extracted directory. ```bash cd insights-onprem--- ./install.sh ``` The installer takes no input: it loads the container images, generates every password and key the stack needs, and starts all services. Re-running it with the same bundle is always safe. {% hint style="danger" %} When the installer finishes, **back up the `.env` file and the `secrets/` directory** from the installation directory. They hold the generated credentials for the stack's own database and object store. Without them you cannot reattach to your data. Both hold credentials so give the backup the same protection you give your production secrets. {% endhint %} All subsequent commands are run from the installation directory using the bundled `./cli` tool. ## Configuring an Extraction An extraction is defined by a JSON file. The bundle ships a template for your provider under `examples/`; copy it and edit it in place. ```bash cp examples/extraction..json extraction.json ``` ### Common Fields | Field | What to put there | | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `providerType` | The monitoring platform this extraction reads from: `datadog`, `prometheus` or `dynatrace`. Already set correctly in the template you copied. | | `name` | Optional label for the extraction, shown by `./cli list`. Free text, up to 255 characters. Remove the key if you do not want a label. | | `timeRange.fromTime` / `timeRange.toTime` | Start and end of the period to extract. Either absolute UTC timestamps (`2026-06-01T00:00:00Z`) or relative values resolved every time the extraction is started: `now`, or `-` where the unit is `m`, `h` or `d`; for example `-1d` means 24 hours ago. | | `timeRange.resolution` | The resolution of the extracted data points. A finer resolution produces a more detailed analysis and a longer extraction. | | `extractionScope` | Which layers to extract. Valid values: `cluster`, `nodes`, `workloads`, `pods`, `containers`, `processes`. The shipped template includes all of them. `processes`, which collects process-level application runtime metrics, is by far the heaviest. | ### Provider-Specific Fields The remaining fields differ per monitoring platform. Credential scopes, token permissions and the metrics your monitoring platform must already be collecting are documented on the platform's own page: [Datadog](/insights/connecting-your-data/datasources/datadog.md), [Prometheus](/insights/connecting-your-data/datasources/prometheus.md), [Dynatrace](/insights/connecting-your-data/datasources/dynatrace.md). The same requirements apply to on-prem extraction. | Field | Datadog | Prometheus | Dynatrace | | ------------------ | ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `clusterIds` | The value of your cluster tag. | The value of your cluster label. The label *name* goes in `providerConfig`. | Your Kubernetes cluster name as known to Dynatrace. | | `providerEndpoint` | Your Datadog API endpoint for your site region. | The URL of your Prometheus, Thanos or Prometheus-compatible endpoint. | Your Dynatrace environment URL. | | `credentials` | `api_key` and `app_key`. | Omit the whole `credentials` object if your endpoint requires no authentication. If it does, provide `bearer_token`. | `api_token`, with the metric and entity read scopes. Optionally add `grail_token`, a Dynatrace Platform token, to also collect container CPU and memory *request* metrics. | | `providerConfig` | Leave empty. | `prometheusConfig.clusterLabelKey` — the label that identifies your cluster (default `cluster`) — and `useClusterLabel` set to `true`. | Leave empty, unless you supplied a `grail_token`: then also set `grailEndpoint` to your Grail query endpoint. | {% hint style="info" %} The credentials you write into this file are stored on your own host, in the stack's database, and are used only to query your monitoring platform. They are never sent to Akamas, and they are excluded from the support bundle you can send to Akamas' support. They are stored unencrypted, so the host itself is the security boundary: anyone with root or Docker access on it can read them. {% endhint %} ## Running an Extraction Start the extraction defined in your file: ```bash ./cli start extraction.json ``` The command prints the extraction id and then polls the extraction status until it completes or fails. Interrupting it with Ctrl-C is safe: the extraction keeps running on the host, and `./cli status --wait` resumes watching at any time. Expect an extraction to take anywhere from minutes to hours, depending on the number of monitored entities, the granularity and the length of the time window. ## Other CLI Commands | Command | What it does | | -------------------------- | --------------------------------------------------------------- | | `./cli list` | Lists every extraction on this host, with its label and status. | | `./cli status ` | Shows the current state of one extraction. | | `./cli status --wait` | Polls until the extraction completes or fails. | | `./cli pause ` | Pauses a running extraction. Progress is kept. | | `./cli resume ` | Resumes a paused extraction. | | `./cli cancel ` | Cancels an extraction permanently. | ## Sending the Data to Akamas When the extraction has completed, download its results: ```bash ./cli download ``` Results land under `downloads/` by default. **Send the whole directory to Akamas** once done. After a successful download, reclaim the disk used by the intermediate data: ```bash ./cli cleanup-raw ``` This deletes only the intermediate raw data, which can always be extracted again from your monitoring platform. The `downloads/` directory is left untouched. ## Tuning The values shipped in the bundle are a single budget sized for an 8 GB host, and in most cases you should not need to change them. Consider tuning only if extractions are consistently too slow for your window, or if a container is terminated for running out of memory. The settings live in the `.env` file in the installation directory. Apply a change by recreating the containers — `stop`/`start` reuses the old values: ```bash docker compose -p insights-onprem -f docker-compose.onprem.yml up -d ``` | Setting | Effect | Before you change it | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `EXTRACTION_CONCURRENCY` | How many extraction tasks run in parallel inside one extraction container. Raising it shortens the extraction phase and increases the query load on your monitoring platform. | Concurrency *is* the memory figure: those tasks share one container. Raising it requires raising `EXTRACTION_MEM_LIMIT` as well, which means a larger host. Lowering a memory limit without lowering concurrency does not prevent the container from running out of memory. | | `EXTRACTION_REPLICAS` | How many extraction containers run. Multiplies both the memory required and the load placed on your monitoring platform. | Set `DISPATCH_CONCURRENCY` to `EXTRACTION_REPLICAS × EXTRACTION_CONCURRENCY` at the same time, otherwise the extra containers sit idle. | | `PROCESSING_MEM_LIMIT` | Memory ceiling for the processing container, which merges the extracted data into the final result files. | This is a floor, not a target: `3072m` is the smallest value that completes a large merge. It only ever goes up, raise it if the processing container is terminated on a large extraction. | ## Upgrading Akamas ships a new bundle rather than upgrading the stack in place. 1. Download any extraction you still need: `./cli list`, then `./cli download `. 2. Uninstall the current stack (see below). This destroys everything still on the host, including results you have not downloaded. 3. Unarchive the new bundle into a **new** directory and run `./install.sh` there. Do not start the new stack before uninstalling the old one — both publish port 8080. The installer accepts only the bundle it originally installed, so a half-upgraded stack cannot happen by accident. ## Uninstall ```bash ./cli uninstall ``` {% hint style="danger" %} Uninstalling removes all containers and destroys all data volumes: the extraction database and any result files you have not downloaded. This cannot be reverted. {% endhint %} ## If Something Fails ```bash ./cli status # current state of the extraction ./cli logs # last log lines from all services ./cli logs manager # ... or from one service ``` If the problem persists, contact and attach the tarball produced by `./cli support-bundle `. It contains the container state, the service logs, the orchestration state of the extraction, and the versions of the installed images. Create the support bundle **before** uninstalling or upgrading. Removing a container deletes its logs.