I built the AmbitionBox salary scraper to collect that context into a consistent dataset. The Python pipeline discovers companies, extracts role-level salary records, writes batches and merges them into an analysis-ready CSV.

The parsing work

The most interesting part was the page’s embedded Nuxt data. It is not a plain JSON object: the payload uses a self-invoking JavaScript function, arguments and variable references to construct records.

The parser identifies the function boundaries, tracks nested delimiters and quoted strings, maps parameter names to argument values, and resolves the property assignments used by the job-profile records. It extracts the supported structure rather than executing the page’s JavaScript.

That distinction also defines a limitation: this is a parser for an observed payload format, not a general JavaScript interpreter. A change in the source representation can require an update.

A fallback with honest missing values

If the structured extraction returns no role records, the scraper falls back to the HTML table. That path can recover visible salary ranges and experience information, while leaving unavailable average-salary values empty.

An absent average should not become zero or an invented midpoint. Keeping it missing lets the later analysis distinguish between “this value was reported” and “this field was unavailable.”

Give the merged dataset a clear identity

The merge step uses company slug, role slug and location as the deduplication key. It keeps the last matching record in the concatenated input, converts numeric fields and sorts the output for inspection.

That is a concrete policy, not proof that a retained row is the newest observation. Adding an explicit collection timestamp would make that distinction easier to handle in future versions.

The workflow can collect available batch artifacts even when some jobs fail. A merged file therefore needs a completeness check before being treated as full coverage. I would add an expected-versus-received batch manifest and parser success counts as the next operational improvements.

What is complete

The repository contains the collection, parsing and merge pipeline. Its exploratory notebook and interactive dashboard are described as work in progress, so I do not present them as finished deliverables.

This is an educational data-engineering project. The dataset reflects the source’s salary reports; it is not verified payroll data or a representative survey of every employer. The portfolio value is the engineering: handling a nontrivial source format, preserving missing values and producing records that can be inspected and analysed.

Implementation details: payload parser, salary collection, merge logic, and batch workflow.

A better first move is to write down the smallest observable mismatch. “The API is wrong” is broad. “This request returns an empty list when the database contains one matching row” gives the investigation a boundary.

Keep one reproduction still

Save the request, relevant configuration and expected result. Remove unrelated inputs until the failure stops shrinking. If the problem is intermittent, record what changes between runs instead of silently treating one successful run as a fix.

Then choose one hypothesis. Perhaps the identifier changes type between request parsing and the database query. Check the value and type at those two boundaries. If both are correct, that hypothesis has earned its retirement.

1
2
3
4
5

logger.debug(
    "lookup input: request_id=%s customer_id_type=%s",
    request_id,
    type(customer_id).__name__,
)

This example deliberately records the type without dumping the entire request. Inspect only the fields needed to answer the question; a diagnostic log should not become a copy of every user’s data.

Find the first wrong state

Trace a short chain: input, parsed value, query parameters, query result, response. If the query result is already empty, changing response serialization is unlikely to help.

For a longer pipeline, inspect the middle first. A correct midpoint moves the search downstream; an incorrect midpoint moves it upstream. You do not need a trace of every line to eliminate half the possibilities.

Once the cause is understood, turn the reproduction into a regression check when the failure is worth protecting against. Remove temporary noise and retain the diagnostic context that would make the same incident understandable next time.

The useful output of debugging is an explanation: this input met this condition, which caused this wrong result. A code change should follow from that explanation.

One small API

1
2
3
4
5

from loadfile import load

df = load("data/local.csv")
df = load("gs://my-bucket/data.parquet")
df = load("archive.zip", filename="sales.csv")

The function is named load(). The package also exports load_data() as a backwards-compatible alias.

fsspec selects the storage backend from the path prefix. The package selects the reader from the file extension, or an explicit format= argument, and passes reader options through to pandas. Cloud backends require their corresponding optional dependencies and credentials.

1
2

df = load("export.tsv", format="csv", sep="\t")
df = load("large.csv", fast=True, usecols=["id", "value"])

CSV, Parquet, JSON, Excel and Feather share the same interface. fast=True opts into Arrow-backed reading; it changes the reading defaults rather than promising a fixed speedup for every file.

ZIP files without another helper

A ZIP containing one supported data file returns a DataFrame. Multiple supported members return a dictionary keyed by filename. You can select one member by name or pass a list to load a subset.

The implementation reads ZIP contents into memory, so archive size still matters. The aim is a convenient reusable loader, not an out-of-core processing engine.

Source: public API and loading implementation.

The useful part of a name is the context it saves the reader from reconstructing.

1
2
3
4

# These values have different meanings.
estimated_delivery_days = 4
scheduled_delivery_days = 2
delay_days = estimated_delivery_days - scheduled_delivery_days

Calling all three values variations of delivery would hide the relationship. Naming the unit and the kind of value makes the subtraction readable without opening another file.

Distinguish a fact from an estimate

This matters especially around model outputs and business calculations. savings can sound like money already saved. estimated_savings leaves room for the assumptions behind the number. model_confidence should not silently become accuracy when it reaches the interface.

The same care helps with state. A job can be accepted, queued, running or completed. A boolean named done often compresses several of those states and leaves failure behaviour unclear.

Match the scope

A short local name can be perfectly clear inside a three-line loop. A public function parameter needs more context because callers see it without the implementation beside it.

Renaming is also a chance to notice a confused abstraction. If a helper needs a name such as validate_and_save_and_maybe_notify, perhaps the problem is the number of responsibilities, not your vocabulary.

I would rather use a slightly longer name with a precise meaning than a polished domain term that nobody on the team uses. The goal is for the next reader to make the right prediction about the code before running it.

A useful event gives the investigation a place to start. For an external API call, that usually means the operation, a correlation identifier, elapsed time, outcome and an error category.

1
2
3
4
5
6
7
8
9

try:
    result = evaluate_shipment(payload)
except TimeoutError:
    logger.warning(
        "shipment evaluation timed out request_id=%s provider=%s",
        request_id,
        provider_name,
    )
    raise

The code is illustrative. In a real service, structured fields are easier to query than values embedded in a sentence, if the logging setup supports them.

Give levels a consistent meaning

If every successful request produces a warning, a warning stops being useful. Decide which events represent normal operation, recoverable trouble and a failed operation that needs attention. Keep that meaning consistent across the service.

An exception stack is useful where the exception is handled and the context is known. Logging the same exception at every layer can turn one failure into a wall of repeated traces.

Leave out what the investigation does not need

Full request bodies are an expensive default. They can include tokens, personal information and long documents. An identifier or a count may answer the debugging question without recording the underlying content.

Similarly, a success event should not claim more than happened. “Request accepted” and “job completed” are different events if a worker performs the job later.

Before adding a log line, imagine the query that would find it. If the line cannot help connect a symptom to an operation, it may belong in a temporary local diagnostic rather than the permanent event stream.

Reference: Python logging HOWTO.