Analytics

Pengin-Pi-3 records which IP addresses hit which paths. That is the whole purpose — a first-party traffic log kept on your own infrastructure, with no third-party script, no cookie, and no tag manager.

The design priority is that logging must never slow down or break a request. Every failure path is silent.

---

What is recorded

Four fields per page view:

| Field | Source | |---|---| | path | Request path, truncated to 500 characters | | ip_address | Real client IP, resolved through the proxy chain | | email_alias | The user's email, if authenticated; otherwise null | | timestamp | ISO 8601, server time |

The request method is not stored. Nor are headers, user agent, referrer, query strings, POST bodies, or response status.

Only GET requests that returned a status below 400 are logged at all. POSTs are never recorded, so form submissions leave no analytics trace.

---

Where the code is

| Path | Contents | |---|---| | util/analytics.py | get_client_ip(), is_trackable_path(), EXCLUDED_PATHS | | util/middleware/analytics.py | PageViewLoggerMiddleware | | main/settings.py | Middleware registration |

The middleware sits after AuthenticationMiddleware in the chain, which is what makes request.user available for the email_alias field. Moving it earlier would silently turn every hit anonymous.

---

Real client IP

get_client_ip() is the piece that makes the rest worth anything. Behind CloudFront, Traefik, and Nginx, REMOTE_ADDR is an internal container address — every visitor looks identical.

Headers are checked in trust order:

1. CloudFront-Viewer-Address, with the port stripped
2. The first element of X-Forwarded-For — the original client, since each proxy appends
3. REMOTE_ADDR as fallback

This function is shared, not analytics-only. Rate limiting (util/security/ratelimit.py) and the RedisLoggingMixin both call it, so correct IP resolution here is also what keeps one visitor behind a shared proxy from throttling everybody.

Nginx must be forwarding the headers. nginx.conf sets real_ip_header X-Forwarded-For, real_ip_recursive on, and trusts the RFC1918 ranges, then passes both X-Forwarded-For and CloudFront-Viewer-Address upstream. Altering that config breaks IP resolution everywhere it's used.

A note on trust: an earlier configuration trusted 0.0.0.0/0, which let any client spoof its own X-Forwarded-For. That was removed (2c89af5). Only ever trust proxy ranges you actually control.

---

Path filtering

is_trackable_path() drops requests against EXCLUDED_PATHS:

• /admin/, /static/, /media/
• /favicon.ico, /apple-touch-icon*.png
• /robots.txt, /sitemap.xml
• Anything ending .css, .js, .map, .woff, .woff2, .ttf, .ico, .png, .jpg, .jpeg, .svg, .webp

Without this, one page view produces dozens of rows for its assets and the log becomes unreadable.

Add your own exclusions to EXCLUDED_PATHS — it's a plain list of compiled regexes.

Note the 400 threshold: 404s and 403s are not recorded here. Visibility into scanners and probes comes from the Nginx access logs and the blocklist middleware instead, not from this table. See Security.

---

The Redis buffer

The middleware does not write to the database. It serializes the hit to JSON and pushes it onto a Redis list named pending_page_views:

client = cache.client.get_client()      # django-redis
client.rpush('pending_page_views', payload)

One rpush adds well under a millisecond to a request; a Postgres insert on every page view does not, and a traffic spike would put that load directly on the primary database.

A scheduled job then drains the list into Postgres in batches — buffer in memory, commit in bulk.

Without Redis

You asked about this specifically, and the answer is: analytics silently disables itself.

The whole push is wrapped in try: ... except Exception: pass. If Redis is unreachable, or the cache backend isn't django-redis (so cache.client doesn't exist), the exception is swallowed and the request continues normally.

Nothing logs, nothing warns, no page slows down. The trade is deliberate — analytics must never be able to break a request — but it means a Redis outage produces a silent gap in your data, not an error. Monitor Redis if the traffic log matters to you.

---

Status: the consumer half is withheld

The mirror ships the producer, not the consumer. The middleware fills the Redis buffer; nothing in main drains it.

The consumer half — the PageViewLog model, the flush and purge commands, and the admin traffic chart — lives on a withheld analytics branch on Tobu Pengin's internal Pengin Open Source GitLab. It is not on the public mirror. A version of it appeared briefly in 073bffe and was reverted in 1fbacb3; the branch is where that work continued.

You need access to that branch to have working analytics. Contact support@tobupengin.com. Everything below describes what it contains, so you know what you're asking for and what to expect once you have it.

Meanwhile

Until the branch lands, pending_page_views grows without bound in Redis — a slow memory leak rather than just missing data. Either comment the middleware out of MIDDLEWARE, or cap the list:

redis-cli LTRIM pending_page_views -10000 -1

Worth checking LLEN pending_page_views on any instance that has been running the middleware for a while.

What the branch provides

PageViewLog — path, ip_address, email_alias, timestamp, all four indexed, since every useful query filters on one of them. Ordered newest first.

flush_redis_views — pops up to 1,000 entries with LPOP, parses each, and commits with bulk_create(). FIFO: the middleware RPUSHes, the command LPOPs. Run on a schedule.

purge_page_views --days N — deletes rows past the retention window.

Admin integration — a traffic chart on the PageViewLog change list.

It is packaged as its own Django app rather than living in main, as a deliberate narrow exception to the single-app convention: analytics owns a model and a migration, and util/ is explicitly defined as not being for that. See Contributing.

Planned: analytics in the core admin view

The intent is to bring the viewer into main as part of the admin surface, alongside the staff console — traffic visible where the rest of site administration already happens, rather than as a separate app you have to install and wire up.

That is the reason the branch is still withheld rather than merged: the storage half is settled, the presentation half is being reworked to fit the core admin rather than carrying over the ported change-list chart as-is. The model and commands are unlikely to change much; where and how you look at the data is.

---

Scheduling the flush

Once you have the branch, flush and purge run from cron on the host. They are ordinary management commands with no daemon.

*/5 * * * * cd /opt/pengin-pi-3 && docker compose exec -T web \
    python manage.py flush_redis_views >> /var/log/pengin-analytics.log 2>&1

0 3 * * * cd /opt/pengin-pi-3 && docker compose exec -T web \
    python manage.py purge_page_views --days 90 >> /var/log/pengin-analytics.log 2>&1

Pick the interval against your traffic. The flush moves at most 1,000 entries per run, so a site doing more than 1,000 views in five minutes needs either a shorter interval or a larger batch — otherwise the buffer grows faster than it drains.

-T disables TTY allocation, which cron requires.

Once the viewer is in the core admin, the cron schedule stays as-is — moving the presentation layer doesn't change how the buffer is drained.

---

Retention and privacy

IP addresses are personal data under GDPR and several US state laws. Two things follow.

Set a retention window and run the purge. Ninety days is a common default. Keeping traffic logs indefinitely is a liability, not an asset.

email_alias ties a browsing path to a named person. That is a more sensitive record than an anonymous IP log, and it is the field most likely to need mentioning in a privacy policy.

The platform has no built-in Do Not Track handling, consent gate, or anonymization. If your jurisdiction requires any of those, they are yours to add.

---

Not the same as request logging

RedisLoggingMixin (util/mixins.py) is a separate mechanism, easy to confuse with this one:

| | Page view analytics | RedisLoggingMixin | |---|---|---| | Trigger | Middleware, every trackable GET | Only views extending SuperTemplateView | | Storage | pending_page_views list | request_log:<ms> keys | | Retention | Until flushed to Postgres | 1-hour TTL | | Records method | No | Yes | | Purpose | Durable traffic history | Short-lived request tracing |

Both call get_client_ip(). See Mixins.

---

Roadmap: offloading to AWS

The intent is to move buffering off the host, so analytics traffic stops competing with the application for memory and the flush stops being a cron job you have to remember.

Of the two services: SQS is the right fit. It is a durable pull-based queue with batch receive, dead-letter handling, and long polling — the same shape the Redis list serves today, with persistence and no local memory cost. A consumer pulls batches and bulk-inserts exactly as flush_redis_views does now.

SNS is pub/sub fan-out. It pushes to subscribers and doesn't buffer, so it solves a different problem — useful if you later want the same event delivered to several destinations at once, typically SNS fanning out into SQS queues rather than replacing them.

Offloading also removes the reason the payload is minimal. Once the host isn't holding the buffer, the record could reasonably carry user agent, referrer, response status, and timing without the memory cost that argues against them today.

Boto3 is already a dependency and SES already uses it, so the credential path exists. See Configuration.

---

Next

• Architecture — the request path and proxy chain
• Deployment — Nginx real-IP configuration
• Mixins — RedisLoggingMixin
• Configuration — Redis and AWS settings