Cleaning a Base URL in Python Before It Turns Into a Bug

A beginner’s guide to urlparse, urlunparse, query strings, and building API URLs without string hacks

A lot of API code breaks in small ways before it breaks in obvious ways. One of the most common examples is the base URL. A program expects something clean like https://api.example.com, but what it actually receives is a full URL with a path, a query string, and maybe a fragment attached to the end. At that point, every later request is built on top of something unstable.

The mistake beginners often make is treating a URL like an ordinary string. It looks like a string, so they slice it, split it, and rebuild it by hand. That works just often enough to feel reasonable. Then a path is already present, or a port appears, or a query string gets duplicated, and the request URL starts mutating into something the code never intended. Python’s urllib.parse module exists to keep that from happening.

The Problem Is Usually Not the Request Itself

A single request can look fine even when the URL handling behind it is weak. That is what makes this topic easy to ignore at first.

base_url = "https://api.example.com/v1/users?sort=asc"
full_url = f"{base_url}/posts"

print(full_url)

The class performing the action here is not really a class at all. It is just string concatenation. The problem is that a URL with extra parts is being treated as if it were a clean origin. The constraint appears the moment the original value already contains a path or a query string. Then each new request starts appending onto something that was never meant to be a base.

This is the same kind of pressure that shows up with a print() statement during debugging. A quick tool works for a moment, but it does not create structure. String-splicing URLs has the same temporary feel. It is not robust enough for code that has to keep building paths correctly.

What urlparse Actually Gives You

The function urlparse() breaks a URL into named parts. That is the key shift in thinking. Instead of guessing where the domain ends or where the query string begins, you let Python parse the URL according to URL rules.

from urllib.parse import urlparse


url = "https://api.example.com/v1/resource?param=value#section"
parsed = urlparse(url)

print(parsed.scheme)
print(parsed.netloc)
print(parsed.path)
print(parsed.query)
print(parsed.fragment)

The important components for this discussion are scheme and netloc. Those two together describe the origin: https://api.example.com. The path, query string, and fragment come after that.

That is the useful beginner lesson. A URL is not one opaque thing. It has structure, and urlparse() exposes that structure directly.

The Clean Base URL Is Usually Just Scheme and Netloc

If you want a stable base URL, keep the scheme and the network location, and discard the rest.

That is the cleanest rule for this kind of post. A base URL is usually the origin, not the full path someone happened to pass in. So if the input includes /v1/users, ?sort=asc, or #section, those pieces should not silently become part of every future request.

The companion function urlunparse() lets you rebuild a URL from its parsed pieces. If you pass empty strings for everything after scheme and netloc, the result is a clean origin.

from urllib.parse import urlparse, urlunparse


raw_url = "https://api.example.com/v1/resource?param=value#section"

parsed = urlparse(raw_url)
base_url = urlunparse((parsed.scheme, parsed.netloc, "", "", "", ""))

print(raw_url)
print(base_url)

That is the real job of this pattern. Not string surgery. Not guessing. Just parse, keep the parts you actually want, and rebuild the URL from those parts.

Why String Splitting Is the Wrong Habit

The temptation to use split("/") is understandable. It looks shorter. But URL structure has edge cases that plain splitting does not handle well. Authentication credentials, custom ports, and IPv6 addresses are all enough to make ad hoc string logic brittle.

examples = [
    "https://user:password@api.example.com/resource",
    "https://api.example.com:8080/resource",
    "https://[2001:db8::1]/resource",
]

The problem with string splitting is not that it never works. The problem is that it works only when the URL shape happens to match your assumptions. urlparse() exists so your code does not have to make those assumptions itself.

A Small Session Example Makes the Problem Visible

One practical place this matters is in a reusable API session. If the constructor accepts an arbitrary URL from a configuration file or user input, the class should normalize that input before storing it.

import requests
from urllib.parse import urlparse, urlunparse


class APISession:
    def __init__(self, url):
        parsed = urlparse(url)
        self.base_url = urlunparse(
            (parsed.scheme, parsed.netloc, "", "", "", "")
        )
        self.session = requests.Session()

    def get(self, path, **kwargs):
        full_url = f"{self.base_url}/{path.lstrip('/')}"
        return self.session.get(full_url, **kwargs)

    def post(self, path, **kwargs):
        full_url = f"{self.base_url}/{path.lstrip('/')}"
        return self.session.post(full_url, **kwargs)

The class performs the action by accepting a possibly messy URL and storing only the clean base. The problem is that later requests need a stable starting point. The constraint is that user input or configuration input may already include paths and query strings. The inverse is easy to describe and hard to defend: skip cleanup and trust that every caller always passes a perfect origin.

What Goes Wrong Without Cleanup

The bug becomes more obvious when the initial URL already includes a path. If that path is stored as the base and later endpoints are appended to it, the code begins to stack paths on top of paths.

client = APISession("https://api.example.com/v1/users?sort=asc")

response = client.get("/v1/users")

With cleanup, the request becomes https://api.example.com/v1/users. Without cleanup, the code may drift toward something like https://api.example.com/v1/users/v1/users. That is the kind of bug that looks small in a blog post and wastes time in a real project.

Sometimes You Do Want the Query String

Not every post about URL parsing should act as if query strings are only garbage to be removed. Sometimes you need to inspect them. That is where parse_qs() becomes useful.

from urllib.parse import parse_qs, urlparse


url = "https://api.example.com/search?q=python&page=2&limit=10"
parsed = urlparse(url)
params = parse_qs(parsed.query)

print(params)
print(params["page"][0])

The important detail here is that parse_qs() returns values as lists. That is intentional. A query parameter can appear more than once, so the parser does not assume each key maps to a single value.

That matters because it teaches the same lesson again: URL handling is structured data handling, not string guessing.

Building Query Strings Should Be Structured Too

The same idea applies in reverse. If query parameters need to be created, they should not be assembled by hand with string concatenation. They should be encoded with urlencode().

from urllib.parse import urlencode, urlparse, urlunparse


base = "https://api.example.com/search"
params = {
    "q": "python url parsing",
    "page": 2,
    "limit": 10,
}

query_string = urlencode(params)

parsed = urlparse(base)
full_url = urlunparse(
    (parsed.scheme, parsed.netloc, parsed.path, "", query_string, "")
)

print(query_string)
print(full_url)

This matters because spaces, ampersands, and reserved characters should be encoded correctly. Hand-built query strings often seem fine right up until a value contains something that needs escaping.

This Post Has a Narrower Job Than the Requests Posts

It helps to keep the article’s role clear. This is not the post about retries. It is not the post about rate limits. It is not even mainly the post about sessions. Its job is smaller and cleaner. It is about URL hygiene. How to parse. How to strip a URL down to a stable base. How to read a query string. How to build one safely.

That narrower focus is what keeps it from competing with the broader API articles. It is the post a reader lands on when the bug is specifically about URL shape.

What a Beginner Should Keep

The most useful beginner lesson is this: a URL should be treated as structured data, not as a raw string you happen to recognize visually. urlparse() exposes the structure. urlunparse() rebuilds the parts you want. parse_qs() reads query strings. urlencode() builds them safely.

Once that clicks, the idea of a clean base URL becomes straightforward. Keep the scheme. Keep the network location. Discard the rest unless the code has a specific reason to preserve it.

No Neat Bow

A malformed request URL is usually not the result of some dramatic failure. More often it comes from a quiet assumption that a URL can be handled like an ordinary string. Python gives better tools than that. Parse the URL. Keep the parts that matter. Rebuild it deliberately. For a beginner, that is enough understanding to prevent a surprising number of small, annoying bugs.

Raell Dottin's Technical Blog