Salary comparisons need more context than a role name and a number. Company, location, experience range and the number of reports all affect how a record should be read.
I built the AmbitionBox salary scraper to collect that context into a consistent dataset. The Python pipeline discovers companies, extracts role-level salary records, writes batches and merges them into an analysis-ready CSV.
The parsing work
The most interesting part was the page’s embedded Nuxt data. It is not a plain JSON object: the payload uses a self-invoking JavaScript function, arguments and variable references to construct records.
The parser identifies the function boundaries, tracks nested delimiters and quoted strings, maps parameter names to argument values, and resolves the property assignments used by the job-profile records. It extracts the supported structure rather than executing the page’s JavaScript.
That distinction also defines a limitation: this is a parser for an observed payload format, not a general JavaScript interpreter. A change in the source representation can require an update.
A fallback with honest missing values
If the structured extraction returns no role records, the scraper falls back to the HTML table. That path can recover visible salary ranges and experience information, while leaving unavailable average-salary values empty.
An absent average should not become zero or an invented midpoint. Keeping it missing lets the later analysis distinguish between “this value was reported” and “this field was unavailable.”
| Stage | Responsibility |
|---|---|
| Discovery | Build the company list used by subsequent batches. |
| Extraction | Parse role records, with optional location and role filters. |
| Batch execution | Split work through a GitHub Actions matrix and collect CSV artifacts. |
| Merge | Deduplicate records and normalise numeric and text columns with pandas. |
| Derived fields | Calculate salary in lakhs, experience midpoint and salary-range width. |
Give the merged dataset a clear identity
The merge step uses company slug, role slug and location as the deduplication key. It keeps the last matching record in the concatenated input, converts numeric fields and sorts the output for inspection.
That is a concrete policy, not proof that a retained row is the newest observation. Adding an explicit collection timestamp would make that distinction easier to handle in future versions.
The workflow can collect available batch artifacts even when some jobs fail. A merged file therefore needs a completeness check before being treated as full coverage. I would add an expected-versus-received batch manifest and parser success counts as the next operational improvements.
What is complete
The repository contains the collection, parsing and merge pipeline. Its exploratory notebook and interactive dashboard are described as work in progress, so I do not present them as finished deliverables.
This is an educational data-engineering project. The dataset reflects the source’s salary reports; it is not verified payroll data or a representative survey of every employer. The portfolio value is the engineering: handling a nontrivial source format, preserving missing values and producing records that can be inspected and analysed.
Implementation details: payload parser, salary collection, merge logic, and batch workflow.