Skip to content
Back to writing

CloudWatch Nginx Logs to CloudFront Log format

When shipping Nginx access logs into AWS Firehose through a CloudWatch Logs subscription, you encounter an architectural mismatch: CloudWatch batches multiple log lines into a single event, but Firehose's transform API only allows you to return one record per recordId. This creates a challenge when you need to forward meaningful analytics to an HTTP endpoint that expects clean, single-event records in CloudFront log format without noise from crawlers or low-value requests.

This scenario commonly occurs when running applications on AWS Elastic Beanstalk, where Nginx logs are automatically streamed to CloudWatch Logs, and you need to pipe them to an analytics service expecting CloudFront-formatted, individual event records.

The Lambda transform described here selects one request from each CloudWatch log batch, preferring product and shop paths. It discards the other requests. That tradeoff may fit a selective event feed, but the output cannot represent complete traffic counts or customer journeys.

Real-World Use Case: Elastic Beanstalk Nginx Logs to Analytics Service

Consider a common scenario: you're running a web application on AWS Elastic Beanstalk with Nginx serving as the reverse proxy. Elastic Beanstalk automatically streams Nginx access logs to CloudWatch Logs, giving you centralized log aggregation. However, you need to forward these logs to a third-party analytics service that expects CloudFront log format and processes individual events, not batched log entries.

Here's the typical architecture:

  1. Elastic Beanstalk → Nginx generates access logs in standard combined format
  2. CloudWatch Logs → Automatically collects and batches Nginx log entries
  3. CloudWatch Logs Subscription → Streams batched log events to Kinesis Firehose
  4. Kinesis Firehose → Invokes a Lambda transform function to process each batch
  5. Lambda Transform → Converts Nginx logs to CloudFront format and selects the highest-priority event
  6. HTTP Endpoint → Receives transformed, single-event records in CloudFront format

The challenge arises because CloudWatch Logs batches multiple Nginx log lines together (often 5-10 lines per batch), but your analytics service expects one event per HTTP request. Additionally, the service expects CloudFront log format, not raw Nginx logs. This transform maps fields into a CloudFront-style record and selects one event from each batch.

The Problem: CloudWatch Batching vs. Firehose Constraints

CloudWatch Logs subscriptions batch multiple log entries together to optimize throughput and reduce API calls. A typical batch might contain:

  • A product page request (/products/sample-product-url)
  • A homepage request (/)
  • A bot request to /robots.txt
  • A crawler hit to /meta.json
  • Several miscellaneous API calls

However, when Firehose invokes your Lambda transform function, it expects exactly one transformed record per recordId. You can't return multiple records for a single batch, which means you need a strategy to select the most valuable log event from each batch.

The goal becomes: Given multiple logs in a single batch, select the best log event and flatten it into a single CloudFront-style analytics record that your downstream HTTP endpoint can process.

Our Prioritization Strategy

We assigned priorities to URL paths. A path indicates the requested page; it does not establish user intent. The strategy prioritizes requests in the following order:

  1. Product pages (/products/...) - Highest configured priority
  2. Shop pages (/shop...) - Second priority
  3. Homepage (/) - Third priority
  4. Everything else - Any path without one of the matches above

If a batch contains several product-page requests, only the first one selected is retained. A bot requesting a product URL receives the same priority as a person requesting it.

Parsing Nginx Log Lines

The first step in our transform is parsing raw Nginx access log entries into structured CloudFront-style records. A typical Nginx log line looks like this:

74.125.213.8 - - [06/Dec/2025:03:54:42 +0000] "GET /products/sample-product-url HTTP/1.1" 200 28517 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"

We parse these into structured records that match CloudFront log format for consistency with existing analytics tools:

function parseNginxLine(line) {
  // Nginx combined log format regex
  const re =
    /^(\S+)\s+\S+\s+\S+\s+\[([^\]]+)]\s+"(\S+)\s+([^" ]+)(?:\s+HTTP\/[0-9.]+)?"\s+(\d{3})\s+(\S+)\s+"([^"]*)"\s+"([^"]*)"(?:\s+"([^"]*)")?$/

  const m = line.match(re)
  if (!m) return null

  const clientIp = m[1]
  const dateTime = m[2]
  const method = m[3]
  const uri = m[4]
  const status = m[5]
  const bytesSent = m[6]
  const referer = m[7]
  const userAgent = m[8]

  // Parse date and time
  const dateTimeMatch = dateTime.match(
    /(\d{2})\/(\w{3})\/(\d{4}):(\d{2}):(\d{2}):(\d{2})\s+([+-]\d{4})/
  )
  if (!dateTimeMatch) return null

  const [, day, month, year, hour, minute, second, timezone] = dateTimeMatch
  const date = `${year}-${month}-${day}`
  const time = `${hour}:${minute}:${second}`

  // Parse URI into path and query
  const [path, query] = uri.split('?')
  const host = extractHostFromReferer(referer) || '-'

  return {
    date,
    time,
    'c-ip': clientIp,
    'cs-method': method,
    'x-host-header': host,
    'cs-uri-stem': path,
    'cs-uri-query': query || '-',
    'cs(User-Agent)': encodeURIComponent(userAgent),
    'cs(Referer)': referer || '-',
    'sc-status': status,
    'sc-bytes': bytesSent,
    'time-taken': '0.002', // Default value, can be extracted from logs if available
  }
}

The parser maps selected Nginx fields into a CloudFront-style object. It also supplies a fixed time-taken value, so that field is not a measured request duration. Check the mapping against the receiving service before using it.

Selecting the Best Event

The core of our solution is the priority function that assigns a numeric priority to each request path:

function getPathPriority(stem) {
  if (stem.startsWith('/products')) return 1
  if (stem.startsWith('/shop')) return 2
  if (stem === '/') return 3
  return 4 // everything else
}

Inside the Lambda handler, we iterate through all log events in the batch and select the one with the highest priority:

exports.handler = async (event) => {
  const output = event.records.map((record) => {
    const payload = Buffer.from(record.data, 'base64').toString('utf-8')
    const parsed = JSON.parse(payload)

    let bestEvent = null
    let bestPriority = Infinity

    for (const logEvent of parsed.logEvents) {
      const cfRecord = parseNginxLine(logEvent.message)
      if (!cfRecord) continue

      const priority = getPathPriority(cfRecord['cs-uri-stem'])

      if (priority < bestPriority) {
        bestPriority = priority
        bestEvent = cfRecord
        // Early exit optimization: if we find a product page, we can't do better
        if (priority === 1) break
      }
    }

    // If no valid events found, return the original record
    if (!bestEvent) {
      return {
        recordId: record.recordId,
        result: 'ProcessingFailed',
        data: {},
      }
    }

    // Encode the selected event
    const outJson = JSON.stringify(bestEvent)
    const encoded = Buffer.from(outJson).toString('base64')

    return {
      recordId: record.recordId,
      result: 'Ok',
      data: encoded,
    }
  })

  return { records: output }
}

The handler emits one selected event for each batch. The early exit keeps the first product-page request it encounters.

Firehose Transform Output Format

Firehose expects a specific response format from your Lambda transform function. Each record in the response must include:

  • recordId: The original record ID, unchanged
  • result: Either "Ok" for successful processing or "ProcessingFailed" for errors
  • data: Base64-encoded transformed data (only required when result is "Ok")

Our transform produces this format:

const outJson = JSON.stringify(bestEvent)
const encoded = Buffer.from(outJson).toString('base64')

return {
  recordId: record.recordId,
  result: 'Ok',
  data: encoded,
}

The downstream endpoint receives the selected event. Lower-priority paths can still be emitted when the batch contains no higher-priority match.

What selection preserves and loses

The transform reduces the number of events sent downstream. It retains the selected record and discards the other records in that batch.

That means it cannot preserve complete product-view counts, conversion funnels, or session history. URL priority also does not identify bots. Use the output only for a purpose that accepts these limits, and keep the original logs when later analysis needs the full record.

Extending the Solution

There are several ways to extend this approach for more sophisticated use cases:

E-commerce Funnel Prioritization: Prioritize by user journey stages—SKU view → Add-to-cart → Checkout—to capture the most valuable conversion signals.

Session Summarization: Instead of selecting a single event, group records by a verified session identifier before summarizing them. A delivery batch alone does not establish a session.

Crawler Detection: Implement more sophisticated bot detection by analyzing User-Agent strings and request patterns, automatically downranking known crawler signatures.

Multi-Event Output: For sinks that support it, emit NDJSON (newline-delimited JSON) format to send multiple events per batch when the receiving endpoint explicitly supports that payload.

Dynamic Priority Rules: Load priority rules from configuration or a database, allowing you to adjust prioritization without redeploying the Lambda function.

Implementation Considerations

When implementing this solution, consider the following:

Error Handling: Ensure your Lambda function gracefully handles malformed log lines, missing fields, and unexpected data formats. Return "ProcessingFailed" for records that can't be processed rather than throwing errors that could cause the entire batch to fail.

Performance: The early exit optimization (breaking when priority === 1) improves performance for batches that contain product page requests, which are typically the most valuable.

Monitoring: Add CloudWatch metrics to track how many events are filtered, which priority levels are selected most frequently, and any processing errors. This helps you understand the effectiveness of your prioritization strategy.

Testing: Test with real CloudWatch log batches to ensure your regex patterns correctly parse your specific Nginx log format. Different Nginx configurations may produce slightly different log formats.

Measure how many records each rule discards and check whether the retained events answer the downstream question. If the consumer needs complete traffic counts or session history, preserve the events instead of selecting one per batch.

Discuss on Twitter