Skip to main content
Version: 4.4

FS and S3/AMQP Locations

Status: Work In Progress This page is currently being drafted and should be reviewed.

This page documents the fs and s3_amqp location types used by the Databox indexer.

Terminology (code-aligned)​

  • Location type fs: local filesystem source.
  • Location type s3_amqp: S3 events consumed from AMQP queue.
  • Main commands:
    • index <location-name>
    • watch
    • list
  • Optional command:
    • index-all (batch run on all configured locations)

fs location​

Purpose​

Import files from a local directory into Databox and keep Databox synchronized with filesystem changes.

Required options​

  • workspaceSlug
  • dir

Optional options​

  • dirPrefix
  • sourceDir

Example​

{
  "name": "my_fs",
  "type": "fs",
  "options": {
    "dir": "%env(WATCH_DIR)%",
    "dirPrefix": "%env(WATCH_DIR_PREFIX)%",
    "sourceDir": "%env(WATCH_SOURCE_DIR)%",
    "workspaceSlug": "%env(DATABOX_WORKSPACE_SLUG)%"
  },
  "alternateUrls": [
    {
      "name": "indexer",
      "pathPattern": "indexer://${sourcePath}"
    }
  ]
}

Runtime behavior​

  • index <location-name>:
    • scans source entries
    • creates/updates assets and files in Databox
  • watch for fs:
    • listens for file events
    • add triggers create/update flow
    • unlink triggers delete flow

s3_amqp location​

Purpose​

Consume S3 object events from AMQP and apply create/delete synchronization in Databox.

Required options​

  • workspaceSlug
  • amqp.dsn
  • s3.endpoint
  • s3.bucketNames
  • s3.accessKey
  • s3.secretKey

Example​

{
  "name": "test-s3",
  "type": "s3_amqp",
  "options": {
    "amqp": {
      "dsn": "%env(AMQP_DSN)%"
    },
    "s3": {
      "endpoint": "%env(S3_ENDPOINT)%",
      "bucketNames": "%env(BUCKET_NAMES)%",
      "accessKey": "%env(S3_ACCESS_KEY)%",
      "secretKey": "%env(S3_SECRET_KEY)%"
    },
    "workspaceSlug": "%env(DATABOX_WORKSPACE_SLUG)%"
  }
}

Runtime behavior​

  • index <location-name>:
    • consumes iterator-based source flow and indexes assets
  • watch for s3_amqp:
    • listens to AMQP queue s3events
    • handles creation events:
      • s3:ObjectCreated:Put
      • s3:ObjectCreated:Post
      • s3:ObjectCreated:CompleteMultipartUpload
      • s3:ObjectCreated:Copy
    • handles deletion event:
      • s3:ObjectRemoved:Delete

Operational notes​

  • Keep workspaceSlug stable for a given location, otherwise idempotence can break.
  • Ensure bucket list and path filters are correct before enabling watch.
  • Do not store raw credentials in committed config files; use environment variables.