SEO and Sitemaps

Pengin-Pi-3 generates robots.txt and sitemap.xml on demand from the database. Neither is a file on disk — both are views, rendered in memory from the URL resolver, the slug tree, and a set of admin-managed rules.

Writing them to disk caused synchronization anomalies and file-permission conflicts across multi-worker containers, which is why the whole subsystem is database-driven instead.

The sitemap output is tested against Google Search Console and Bing Webmaster Tools, and is maintained by the Pengin-Pi support team. Tobu Pengin, L.L.C. sponsors this work as part of its stake in the platform.

Endpoints:

/robots.txt        main/views/seo.py
/sitemap.xml       main/sitemaps.py

Both tolerate a trailing slash.

---

RobotsRule

One small model drives crawler policy for the whole site, editable from Django admin without a deploy.

| Field | Default | Meaning | |---|---|---| | user_agent | * | Which crawler the rule addresses | | path | — | Path or pattern, e.g. /admin/ or *edit* | | allow | False | True emits Allow, False emits Disallow |

Rules do two jobs at once. They are emitted into robots.txt, and the sitemap obeys them — any URL a rule disallows is stripped from sitemap.xml in real time rather than drifting out of sync with the policy you published.

That is the part worth internalizing: one rule, both effects. You never maintain a separate exclusion list for the sitemap.

With no rules defined at all, robots.txt falls back to disallowing /admin/, /login/, and /slug/.

Pattern matching

Two forms:

• Prefix — /products/ matches that path and everything beneath it
• Wildcard — a path containing * is matched with fnmatch. A pattern not starting with * or / gets wrapped as *pattern*, so edit and *edit* behave identically.

Precedence follows Google's own rules: the longest matching pattern wins, ties favour Allow. Specificity is the length of the rule exactly as authored, which keeps wildcard and prefix rules comparable.

So Disallow: *edit* plus Allow: /docs/how-to-edit/ works the way you'd expect — the longer, more specific rule takes the page back.

Live examples

From penginopensource.org/robots.txt:

User-agent: *
Disallow: *slug*
Disallow: *create*
Disallow: *edit*
Disallow: /docs/new-page/nested/
Disallow: /docs/new-page/

Sitemap: https://penginopensource.org/sitemap.xml

Three wildcard rules covering the editing surface site-wide — the slug editor, wiki page creation, wiki page editing — plus two prefix rules taking specific draft pages out of the index. The wildcards are the pattern to copy: *create* and *edit* catch every wiki and slug route at any depth without enumerating them.

From tobupengin.com/robots.txt, a fuller production deployment:

User-agent: *
Disallow: *forums*
Disallow: *tickets*
Disallow: *edit*
Disallow: *create*
Disallow: *admin*
Disallow: *slug*
Disallow: *send_email*
Disallow: *profile*
Allow: jobs
Disallow: *applications*
Disallow: abuseipdb-verification.html
Disallow: *calendar*
Disallow: *auth*
Disallow: /companies/sync/contracts/

Note Allow: jobs sitting among the disallows. Job postings should be indexed; the applications attached to them should not. And /companies/sync/contracts/ shows a precise prefix rule for one internal route rather than a broad wildcard that would catch too much.

A reasonable starting set for a new deployment: *admin*, *slug*, *create*, *edit*, *profile*, *auth*. Add wildcards for any gated section as you build it.

The sitemap link

robots.txt always appends an absolute Sitemap: line built from reverse('sitemap') and the request host, so the two never disagree about where the sitemap lives.

---

The sitemap

/sitemap.xml is a Django sitemap index over three sources, registered in main/urls.py:

| Source | Covers | changefreq | priority | |---|---|---|---| | StaticAppSitemap | Named URL patterns with no parameters | weekly | 0.8 | | SlugDatabaseSitemap | Every Slug | daily | 0.8 | | DynamicAppSitemap | Models opting in via SitemapEntry | daily | 0.7 |

All three filter through is_disallowed() against the current rules, and all three run the redirect probe described below.

---

Static routes

StaticAppSitemap discovers routes by walking Django's URL resolver rather than from a maintained list. You don't register a static page for the sitemap — it's found.

A route is included when it:

• has a name in its path() declaration — unnamed routes are invisible
• takes no URL parameters — anything with < in the pattern is skipped
• isn't under the admin namespace
• isn't in the hardcoded ignored_names set
• isn't login-gated
• isn't disallowed by a RobotsRule
• doesn't redirect

Namespaced includes are walked recursively, so a mounted app's named routes are picked up with their namespace prefix.

Practical consequence: adding path('pricing/', PricingView.as_view(), name='pricing') puts that page in the sitemap on the next request. Leaving the name= off keeps it out. That's the simplest lever you have.

ignored_names

A hardcoded set covering core auth and editing routes — login, logout, signup, profile, slug, slug_edit, slug_delete, reset_password, and friends. Matching is on the final segment after any namespace.

This is belt-and-braces alongside the login detection. For your own routes, use a RobotsRule rather than editing this set — the set is core, the rules are yours.

Login detection

_requires_login() catches two independent gating styles:

• A class-based view with LoginRequiredMixin anywhere in its MRO
• A view marked requires_auth = True — which main/auth/decorators.py's real gates (group_required, is_admin_required) set on the functions they wrap

The marker exists because MRO introspection cannot see a decorator applied via @method_decorator(..., name='dispatch'). is_admin_provider and user_group_provider are deliberately not marked — they inject context and never block, so they aren't gates.

If you write a new access-gating decorator, set requires_auth = True on the wrapped function. Forget it and your gated pages get advertised to crawlers, with nothing to tell you. See RBAC.

Redirect probing

Google Search Console flags any sitemap URL that redirects instead of resolving — "Sitemap contains a redirect." So every candidate URL, from all three sources, is probed: a bare anonymous GET through RequestFactory, and a 3xx response excludes it.

This is generic on purpose. A redirect can come from an unnamed path('', lambda r: redirect(...)) alias, or from a dispatch() that redirects unqualified users — arbitrary imperative logic no mixin or marker check can see.

It is a probe, not a full request: no session or messages middleware is attached. A view touching request.session before it would redirect raises, and the exception is caught and treated as "not a redirect." It fails open — a wrongly included redirect is a minor warning, a wrongly excluded real page is a bigger loss.

Worth knowing for cost: the probe executes the view. Sitemap generation on a large site runs every candidate view once.

---

Slug pages

SlugDatabaseSitemap includes every Slug, with no opt-in. Pages created in the editor and wiki pages alike are indexed the moment they exist, subject to the rules and the redirect probe.

lastmod comes from updated_at, falling back to date.

This is why *create* and *edit* wildcards matter on a site with a wiki: without them, every wiki page's create and edit route would be probed and potentially advertised.

To keep a specific page out, add a RobotsRule for its path. There is no per-slug noindex flag.

Each slug's meta_tags and meta_description fields feed the rendered page's head through layout.html's {% block tags %} and {% block meta_description %}. See Slugs and Pages.

---

Dynamic SEO: putting a model in the sitemap

This is the part that needs implementing per model. A model appears in the sitemap only if it explicitly subclasses SitemapEntry — there is no duck-typing on get_absolute_url().

That was deliberate. Earlier behaviour keyed off whether a model happened to define get_absolute_url(), which let unrelated models leak in and out of the index as methods were added and removed.

Implementation

from main.models.mixins import SitemapEntry, HistoryMixin, AbstractHistory


class Product(SitemapEntry, HistoryMixin, models.Model):
    sitemap_lastmod_field = 'updated_at'

    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    name = models.CharField(max_length=255)
    updated_at = models.DateTimeField(auto_now=True)

    def get_absolute_url(self):
        return reverse('products:product_detail', kwargs={'pk': self.id})

Three requirements:

1. Subclass SitemapEntry, first in the bases
2. Implement get_absolute_url() — the base raises NotImplementedError if you don't
3. Set sitemap_lastmod_field to a field name, usually updated_at. Optional; without it entries carry no lastmod

DynamicAppSitemap then walks apps.get_models(), finds every SitemapEntry subclass, and iterates its objects. Nothing else to register — no sitemap class to write, no entry in main/urls.py.

Objects whose get_absolute_url() raises are skipped silently.

Scale

items() iterates model.objects.all() for every registered model on every sitemap request, and redirect-probes each URL. That's fine at hundreds of objects and becomes a problem at tens of thousands. If you get there, write a dedicated Sitemap class with pagination and register it in urls.py alongside the three defaults rather than relying on SitemapEntry.

---

Structured data

JSON-LD is opt-in per page, not a context processor — structured data belongs only on pages that declare it.

util/seo.py builds a schema.org ProfessionalService dict from the Site model (company name, phone, postal address):

from util.seo import build_organization_schema

context['organization_schema'] = build_organization_schema(
    site, description="...", url="https://example.com/",
)

Then in the template:

{% block jsonld %}
  {% include "js/json-ld.html" with jsonld=organization_schema %}
{% endblock %}

The Site model is also exposed as site in every template by a context processor, which is what nav_bar.html, footer_bar.html, and copyright.html read instead of hardcoding a company name.

---

Verifying

curl https://yoursite.example/robots.txt
curl https://yoursite.example/sitemap.xml

Check a rule's effect directly:

python manage.py shell -c "
from main.sitemaps import is_disallowed
print(is_disallowed('/docs/new-page/'))
print(is_disallowed('/products/widget-a/'))
"

List what the static sitemap actually resolved:

python manage.py shell -c "
from main.sitemaps import StaticAppSitemap
print(StaticAppSitemap().items())
"

Submit sitemap.xml to Google Search Console and Bing Webmaster Tools. Both report per-URL problems, and "Sitemap contains a redirect" is the warning the probe exists to prevent.

---

Caveats

• No caching. Every request regenerates from the database and re-probes every URL. Consider a cache in front of both endpoints on a large site.
• Redirect probing executes views. A view with side effects on GET will have them triggered during sitemap generation.
• robots_txt groups by consecutive user_agent. Rules are emitted in default queryset order, so two rules for the same agent separated by a third agent's rule produce a repeated User-agent: block. Valid, but untidy — keep rules for one agent together.
• is_disallowed() ignores user_agent entirely. Sitemap filtering applies every rule regardless of which crawler it addresses. A rule targeting one bot still removes the URL from the sitemap for everyone.
• No per-object noindex. Exclusion is by path rule only.

---

Next

• Slugs and Pages — where page metadata lives
• Wiki Module — why *create* and *edit* rules matter
• RBAC — the requires_auth marker
• Architecture — the subsystem in context