Build Your Own CIAN Parser: Step-by-Step Guide to Scraping Listings, Prices, and Price History

Introduction: What You'll Get

The real estate market runs on numbers. Some people hunt for apartments priced below market, some size up competitors in new builds, and some put together reports for investors. What they all have in common: they need up-to-date data on listings and prices, and gathering it by hand is simply impossible. That's where a custom CIAN parser comes in handy.

In this guide, you'll build a working tool from scratch that does three things. First: it walks through listing pages for a given filter and collects a list of properties with price, address, square footage, and URL. Second: it visits each listing's detail page and pulls out details like floor, year built, and building type. Third, and most valuable: it saves data to a database and accumulates price history day by day, so you can see which listings dropped in price, which were delisted, and how the market is moving in a particular neighborhood.

Here's what the end result looks like: you have a folder with Python scripts, a SQLite database file, and a table you can open in Excel or Google Sheets. You run the script in the morning, and you get a fresh snapshot. After a week of runs, you already have dynamics.

Who This Guide Is For

• For marketers and analysts at real estate agencies who need price monitoring by neighborhood without buying expensive reports.
• For side-hustlers and business owners looking for niches and wanting to understand supply and demand in numbers.
• For beginner developers who want to learn scraping with a real, down-to-earth example.
• For investors and individual buyers who want to catch price drops before everyone else.

What You Need to Know Beforehand

Programming experience isn't required. We'll go through every line of code and explain why it's there. All you need is to be able to install programs, open a command line, and copy text. If you've ever opened developer tools in a browser, it'll be a breeze. If not, we'll show you where they are.

The only requirement: attention to detail. Scraping is sensitive to typos in class names and URLs. One extra letter and the script returns an empty list. Don't panic: every step has a checkpoint where you confirm everything is on track.

How Long It Takes

Setting up the environment takes about 30 minutes. You'll write your first working listing parser in an hour. Detail pages, proxies, and the database will take another hour and a half to two hours. That's three to four hours of focused time if you take it easy. Price history starts accumulating on its own from the second run onward.

Preparation: Tools and Environment

Before writing code, let's gather everything you need. Don't skip this section: half of beginners' problems come from a badly installed Python or missing libraries.

System Requirements

• A computer running Windows 10 or 11, macOS, or Linux. Any laptop from the last eight years will do.
• At least 4 GB of RAM. For the browser automation option, 8 GB is preferable.
• About 2 GB of free disk space for Python, libraries, and the database.
• A stable internet connection.

What to Install

1. Python 3.11 or newer. Download the installer from the official python.org website. On Windows, be sure to check Add Python to PATH at the bottom of the first screen. Without it, the python command won't work in the terminal. On macOS, Python is often preinstalled, but it's better to install a fresh version.
2. A code editor. We recommend Visual Studio Code: it's free, highlights syntax, and shows errors. Install the Python extension from the built-in extension marketplace (the icon with four squares on the left panel).
3. Chrome or Edge browser. We'll need developer tools to study page structure.
4. Python libraries. We'll install them via the terminal below.
5. Access to mobile proxies. You'll need them at step five. You need the server address, port, username, password, and a link to rotate the IP address. Your service provider gives you all of this in your account dashboard after purchase. If you don't have proxies yet, you can still complete the first steps without them.

Creating a Working Folder and Virtual Environment

1. Create a folder on your drive called cian_parser. Avoid Cyrillic characters and spaces in the path: they sometimes break tools.
2. Open a terminal. On Windows, press Win+R, type cmd, and hit Enter. On macOS, open Terminal via Spotlight.
3. Navigate to the folder using cd and its path, for example: cd C:\projects\cian_parser on Windows or cd ~/projects/cian_parser on macOS.
4. Create a virtual environment with python -m venv venv. This is an isolated copy of Python so project libraries don't conflict with system ones.
5. Activate the environment. On Windows: venv\Scripts\activate. On macOS and Linux: source venv/bin/activate. The terminal prompt will show (venv).
6. Install the libraries with a single command: pip install requests beautifulsoup4 lxml pandas openpyxl. Installation takes one to two minutes.

Check: type in the terminal python -c "import requests, bs4, pandas; print('ok')". If you see ok with no errors, the environment is ready. If you see ModuleNotFoundError, the environment isn't activated or the installation was interrupted. Activate venv again and repeat pip install.

Backups

In this project, the main value isn't the code — it's the accumulated database with price history. It can't be restored: past prices won't appear anywhere else. So make a pact with yourself from day one: copy your database file to the cloud or an external drive at least once a week. Later we'll add automatic backup to the script.

Basic Concepts: What You Need to Understand Before Starting

Let's go over the terms you'll encounter. If you're already familiar with scraping, skim this section, but pay attention to the legal part.

Key Terms in Plain Language

• Parsing (scraping): automatically fetching a website page with a program and extracting the data you need. Same as what you do with your eyes, but the script does it and a thousand times faster.
• HTML: the markup language every web page is made of. An apartment's price on CIAN lives inside an HTML tag with specific attributes, and our job is to find that tag.
• Selector: the address of an element inside HTML. For example, a span with the attribute data-mark equal to MainPrice. The parser uses the selector to know where to get the price from.
• HTTP request: a call to the site's server. The browser makes one when you open a page. The requests library does the same from code.
• Request headers: service information the browser sends with a request: browser type, language, data formats. The server uses these to decide what to return.
• Proxy: an intermediary server your requests pass through. Mobile proxies use IP addresses from cellular carriers and let you change the address on command.
• Pagination: splitting listings across pages. To collect all listings, the parser must go from the first page to the last.
• SQLite: a lightweight database in a single file. Requires no server setup, built into Python. Ideal for price history.

How Real Estate Listings Work

CIAN, Domclick, Yandex Real Estate, and other platforms work on a similar principle. There's a search page with filters: city, deal type, number of rooms, price range. Each filter becomes a parameter in the URL. For example, the deal_type parameter with the value sale means sale, and room1 set to 1 adds one-room apartments. Understanding these parameters gives you a powerful tool: instead of clicking around the site, you just craft the URL.

Inside the listing, each property appears as a card: title, price, address, a few photos, and a link to the detail page. The detail page contains full attributes and often duplicates all data in a hidden JSON block the site uses to render the UI. Parsing that block is far easier than HTML.

Legal and Ethical Boundaries

Warning: collect only publicly available listing information: price, area, address, building attributes. Do not collect or store phone numbers, names, or other personal data of sellers and agents: this is governed by data protection laws, and violations carry real consequences. Read the platform's terms of service before you start, and use data for your own analytics, not for resale or creating a copy of the site. Keep request rates reasonable: your parser shouldn't create a load that disrupts the service.

This approach isn't just legal — it's practical. A careful parser with pauses and IP rotation works for months, while an aggressive one gets rate-limited within an hour.

Step 1: Define Your Goal and Data Structure

Goal of this stage: clearly describe what exactly you're collecting and how it will be stored. Without this step, you'll write a parser that pulls everything, and then spend a week sorting through a data dump.

1. Formulate a business question. Examples: which one-room apartments in St. Petersburg dropped more than 5 percent in price over a month; how much does a square meter cost in new builds in a specific neighborhood; how fast do listings under a certain price move.
2. Define the listing filter. For this guide's example, we'll use: sale, secondary market, one- and two-room apartments, Moscow, price up to 15 million rubles. You'll plug in your own parameters.
3. Make a list of fields. For each listing we need: unique listing ID, URL, title, price, address, total area, floor and total floors, building type, year built, first seen date, last checked date. For price history: listing ID, date, price.
4. Open your code editor and create a file config.py in the project folder. Write in it the parameters you'll change most often:

BASE_URL = 'https://www.cian.ru/cat.php'
SEARCH_PARAMS = {'deal_type': 'sale', 'engine_version': 2, 'offer_type': 'flat', 'region': 1, 'room1': 1, 'room2': 1, 'maxprice': 15000000}
MAX_PAGES = 5
PAUSE_MIN = 4
PAUSE_MAX = 9
DB_PATH = 'realty.db'

Note: in the code example, line breaks are denoted by newline symbols; in an editor just write each variable on its own line. The region parameter equal to 1 corresponds to Moscow, 2 to St. Petersburg. You'll find other region codes by applying a filter on the site and checking the URL.

Tip: start with MAX_PAGES set to 2-3. Each listing page has about 28 properties, which is enough for debugging. Run the full collection once you're sure all fields are extracted correctly.

Check: you have a config.py file, and you've recorded in a notepad (or your head) a list of 12 fields and one specific business question. If your question sounds like I want all data for all of Russia, go back and narrow it down: full country-wide collection means hundreds of thousands of listings and a completely different infrastructure.

Potential Problems

You can't figure out which parameter controls the filter you need. Solution: open the site, set the filter manually, copy the URL from the address bar, and break it apart by the ampersand symbol. Each key-value pair is a parameter.

Step 2: Study the Listing Page Structure

Goal of this stage: find the HTML elements from which we'll pull price, title, address, and URL. This is the most investigative step, and it's exactly where beginners usually get lost, so let's go very slowly.

1. Open your browser and go to the CIAN listing page with your filters. Make sure you can see the list of apartments.
2. Hover over the price of any apartment, right-click, and choose Inspect (in Edge it's called Inspect too). The developer tools panel opens and highlights the price element.
3. Look at the highlighted line. At the time of writing, it's a span tag with the attribute data-mark equal to MainPrice. Write down this attribute: it'll be the selector for price.
4. Move up the element tree by clicking parent tags until you find a tag that wraps the entire listing card. Usually it's article with the attribute data-name equal to CardComponent. When you hover over it in the panel, the whole card with the photo and price gets highlighted on the page.
5. Inside the card, find the title (span with data-mark equal to OfferTitle), the address (several a links with data-name equal to GeoLabel that together make up the address), and the link to the listing (an a tag with href leading to a URL like cian.ru/sale/flat/number). Write down all four selectors.
6. Find the pagination block at the bottom of the page. Scroll the listings to the end, right-click the number of page two, and look at its URL. You'll see the parameter p equal to 2. So, to move through pages you just change this parameter.

Warning: attribute names like data-mark and data-name change periodically when platforms update their design. Don't copy the selectors from this text blindly: always verify them against the real page in the developer tools. Being able to find a selector yourself matters more than any ready-made list.

Check for Hidden JSON

Many platforms store listing data in ready form inside a script tag. That's more convenient than HTML: no need to glue the address together from pieces.

1. In the developer tools, press Ctrl+F (Cmd+F on macOS) and type the word offers or initialState.
2. If the search finds a script tag with a large block of text that looks like a dictionary with curly braces, the data is there in JSON. Note the variable name at the beginning of that block.
3. If nothing is found, no problem: the HTML approach from the next step always works.

Tip: open the Network tab in developer tools, refresh the page, and filter requests by Fetch/XHR type. Sometimes the site loads listings via a separate JSON request. If you see such a request with an offers field, parsing it is the easiest: you get clean data with no HTML.

Check: you've written down selectors for the card, price, title, address, and URL, and you know the pagination parameter name. When you click each selector in the panel, the corresponding element highlights on the page.

Potential Problems

The developer tools show HTML but there's no price in it. Cause: the site renders some data via script after loading. Solution: on the Network tab, check whether the price arrives in a separate request, or use the browser automation from the advanced section.

Step 3: Write Your First CIAN Parser for the Listing Page

Goal of this stage: get a script that downloads the listing page, extracts the list of properties, and prints them to the console. After this step, you'll have a working skeleton we'll build on.

1. Create a file parser.py in the project folder.
2. Import libraries and settings at the top of the file:

import time
import random
import requests
from bs4 import BeautifulSoup
from config import BASE_URL, SEARCH_PARAMS, MAX_PAGES, PAUSE_MIN, PAUSE_MAX

3. Define the request headers. The server should see a normal browser with Russian locale, otherwise you might get the wrong version of the page:

HEADERS = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0 Safari/537.36', 'Accept-Language': 'ru-RU,ru;q=0.9', 'Accept': 'text/html,application/xhtml+xml'}

4. Write a page-fetching function. It takes a page number, adds it to the parameters, and returns HTML. Always check the response code: 200 means success, anything else is a signal to stop and investigate:

def fetch_page(page, session):
params = dict(SEARCH_PARAMS)
params['p'] = page
resp = session.get(BASE_URL, params=params, headers=HEADERS, timeout=30)
if resp.status_code != 200:
print('Status', resp.status_code, 'on page', page)
return None
return resp.text

5. Write a parsing function. It finds all cards and pulls fields from each. Notice the if construct: if an element is missing, we don't crash but write None instead:

def parse_cards(html):
soup = BeautifulSoup(html, 'lxml')
cards = soup.select('article[data-name=CardComponent]')
result = []
for card in cards:
link_tag = card.select_one('a[href*=/sale/flat/]')
price_tag = card.select_one('span[data-mark=MainPrice]')
title_tag = card.select_one('span[data-mark=OfferTitle]')
geo_tags = card.select('a[data-name=GeoLabel]')
if not link_tag or not price_tag:
continue
url = link_tag.get('href')
offer_id = url.rstrip('/').split('/')[-1]
price_text = price_tag.get_text()
price = int(''.join(ch for ch in price_text if ch.isdigit()))
title = title_tag.get_text(strip=True) if title_tag else None
address = ', '.join(g.get_text(strip=True) for g in geo_tags)
result.append({'offer_id': offer_id, 'url': url, 'title': title, 'price': price, 'address': address})
return result

6. Build the main loop. It goes through pages, makes a random pause between requests, and adds results to a common list. The random pause matters: equal intervals look unnatural and create peak load:

def collect_listing():
session = requests.Session()
all_offers = []
for page in range(1, MAX_PAGES + 1):
html = fetch_page(page, session)
if html is None:
break
offers = parse_cards(html)
print('Page', page, 'listings:', len(offers))
if not offers:
break
all_offers.extend(offers)
time.sleep(random.uniform(PAUSE_MIN, PAUSE_MAX))
return all_offers
if __name__ == '__main__':
data = collect_listing()
for item in data[:5]:
print(item)
print('Total collected:', len(data))

7. Save the file and run it in the terminal with python parser.py. Make sure the venv environment is active.

Let's go over key points. Session preserves cookies between requests, so the site sees the consistent behavior of one visitor rather than a dozen scattered calls. The select_one function returns the first matching element or None, so we always check the result before calling get_text. We take the listing ID from the URL: it's the last fragment of the address, a number like 312456789. That's what will become the key for price history.

Tip: during debugging, save the HTML of the first page to a file using open('page1.html', 'w', encoding='utf-8').write(html). Then you can debug the parsing function on a local copy without sending extra requests to the site.

Check: in the console you see lines Page 1 listings: 28, Page 2 listings: 28 and so on, and five dictionaries with real prices and addresses at the bottom. Prices should be whole numbers with no spaces or ruble signs. If you see empty instead of numbers, go back to the selectors from step two.

Potential Problems

The script prints listings: 0 on the first page. Causes: the card selector changed or the server returned a placeholder page. Open the saved page1.html in a browser and see what you got. If it's a page asking you to confirm you're not a robot, increase pauses and move on to step five with proxies. If it's a normal listing, verify your selectors.

ValueError when converting the price. Cause: the price text has no digits, for example it says Price on request. Solution: wrap the conversion in try and write None for such cases.

Step 4: Scrape Listing Detail Pages

Goal of this stage: teach the parser to visit each listing's page and extract detailed attributes: area, floor, building type, year built. These fields are needed to calculate price per square meter and compare similar apartments.

1. Open any listing page from the feed in your browser. Press Ctrl+U to view the page source.
2. Press Ctrl+F and type totalArea. At the time of writing, listing data lives in a script tag inside a frontend config object, under a key like frontend-offer-card. You'll see fields totalArea, floorNumber, floorsCount, buildYear, materialType, and others.
3. If searching for totalArea gives nothing, look for attributes in the HTML: usually a block with name-value pairs like Total area and 38.5 m². Write down the selector of that block.
4. Add a function to parser.py that extracts JSON from the listing page. We find the needed script, cut out the object by curly braces, and parse it with the json module:

import json
import re
def parse_offer_page(html):
soup = BeautifulSoup(html, 'lxml')
details = {}
for script in soup.find_all('script'):
text = script.string or ''
if 'totalArea' in text and 'offerData' in text:
m = re.search(r'totalArea[^0-9]*([0-9.,]+)', text)
if m:
details['area'] = float(m.group(1).replace(',', '.'))
m = re.search(r'floorNumber[^0-9]*([0-9]+)', text)
if m:
details['floor'] = int(m.group(1))
m = re.search(r'floorsCount[^0-9]*([0-9]+)', text)
if m:
details['floors_total'] = int(m.group(1))
m = re.search(r'buildYear[^0-9]*([0-9]{4})', text)
if m:
details['build_year'] = int(m.group(1))
break
return details

5. Here we deliberately use regular expressions instead of full JSON parsing. The reason is simple: the script on the page contains not just JSON but code too, and isolating a clean object can be tricky. Regexes find a key and the first number after it, which is reliable enough for numeric fields.
6. Add a function that takes the list of listings from the feed and enriches each with detail page data. Pauses matter even more here, because the number of requests grows 28-fold:

def enrich_offers(offers, session):
for i, offer in enumerate(offers, 1):
try:
resp = session.get(offer['url'], headers=HEADERS, timeout=30)
if resp.status_code == 200:
offer.update(parse_offer_page(resp.text))
else:
print('Card', offer['offer_id'], 'status', resp.status_code)
except requests.RequestException as e:
print('Network error on', offer['offer_id'], e)
if i % 10 == 0:
print('Cards processed:', i)
time.sleep(random.uniform(PAUSE_MIN, PAUSE_MAX))
return offers

7. In the if __name__ block after collect_listing, add a call to enrich_offers(data, requests.Session()) and run the script again with MAX_PAGES set to 1 so you don't wait long.

How long this takes: 28 cards at an average pause of 6 seconds is about 3 minutes. A full collection of 5 listing pages with cards takes roughly 15 minutes. That's fine. A real estate parser shouldn't be fast, it should be stable.

Tip: don't re-parse cards on every run. Apartment attributes don't change: area and year built only need to be collected once. Only the price changes, and it's in the feed. In step six we'll make it so cards are only requested for new listings. That cuts request counts by an order of magnitude.

Check: the output dictionaries now have keys area, floor, floors_total, and build_year with plausible values: area between 15 and 200, floor no more than total floors, year between 1900 and 2026. If some listings lack fields, that's normal: not all sellers fill in the year built.

Potential Problems

The regex finds room area instead of total area. Cause: JSON has similar keys like livingArea or kitchenArea. Solution: refine the pattern by adding a quote or colon before the key so it doesn't match part of another word.

Step 5: Set Up Mobile Proxies and Make Collection Robust

Goal of this stage: distribute requests through mobile proxies with IP rotation, add retries and proper response handling. After this step, the parser will be able to run regularly and for long stretches without creating excessive load from a single address.

Why a Real Estate Parser Needs Mobile Proxies

Any large platform limits request rate from a single IP address. That's overload protection, and it kicks in on any automation. A home address starts getting 429 responses or challenge pages after a few hundred requests. Mobile proxies solve the problem differently: you get an IP from a cellular carrier's pool, and it changes on command or on a timer. Your requests spread across addresses, load per address stays low, and the parser runs smoothly. For regular price monitoring this is fundamental: you need not one-off data but daily snapshots over months.

Setup

1. Open your mobile proxy service dashboard and find the proxy you bought. Copy four values: host, port, username, password. Also copy the IP rotation link: usually a URL with a key that, when called, gives the proxy a new address.
2. Add proxy parameters to config.py. Never write passwords into code you put anywhere: keep them in a separate file or environment variables:

PROXY_HOST = 'your_host'
PROXY_PORT = 'your_port'
PROXY_USER = 'your_username'
PROXY_PASS = 'your_password'
ROTATE_URL = 'ip_rotation_link'
ROTATE_EVERY = 25

3. Add a function to parser.py that creates a session with the proxy. The requests library accepts a dictionary with addresses for http and https:

from config import PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS, ROTATE_URL, ROTATE_EVERY
def make_session():
session = requests.Session()
proxy_url = 'http://' + PROXY_USER + ':' + PROXY_PASS + '@' + PROXY_HOST + ':' + PROXY_PORT
session.proxies = {'http': proxy_url, 'https': proxy_url}
return session
def rotate_ip():
try:
r = requests.get(ROTATE_URL, timeout=20)
print('IP rotation:', r.status_code)
time.sleep(5)
except requests.RequestException as e:
print('Failed to rotate IP:', e)

4. Check that the proxy works. Create a temporary file check_proxy.py with code that requests an IP lookup service through the session and prints the response:

from parser import make_session
s = make_session()
print(s.get('https://api.ipify.org', timeout=20).text)

5. Run it. You should see an IP address different from your home one. Call rotate_ip and run the check again: the address should change.
6. Now add a request function with retries. It handles three situations: successful response, 429 or 403 status (need to wait and rotate), network error (retry):

def safe_get(session, url, params=None, retries=3):
for attempt in range(1, retries + 1):
try:
resp = session.get(url, params=params, headers=HEADERS, timeout=30)
if resp.status_code == 200:
return resp
if resp.status_code in (429, 403):
print('Status', resp.status_code, 'attempt', attempt, 'rotating IP and waiting')
rotate_ip()
time.sleep(30 * attempt)
continue
print('Unexpected status', resp.status_code)
return None
except requests.RequestException as e:
print('Network error', e, 'attempt', attempt)
time.sleep(10 * attempt)
return None

7. Replace session.get calls in fetch_page and enrich_offers with safe_get. Add a counter to enrich_offers: every ROTATE_EVERY requests, call rotate_ip. This is planned rotation that keeps any single address from accumulating too many calls.

Warning: if you get a 429 or a challenge page, don't try to break through with frequent retries. That will only make things worse for the current address. The right response: stop, increase pauses, rotate IP, and continue at a calm pace. A parser that respects site limits lives longer and collects more.

Tip: use one proxy channel per parsing stream. The temptation to run ten threads through one address is strong, but that's a direct route to restrictions. If you need speed, buy several channels and spread different regions or filters across them.

Check: check_proxy.py shows a mobile carrier address, and after rotation the address changes. The parser goes through two listing pages with cards without a single 429. The log shows IP rotation: 200 every 25 cards.

Potential Problems

ProxyError or 407 error. Cause: wrong username or password, or the wrong port. Solution: check the credentials in your dashboard and make sure you're using the HTTP proxy port, not SOCKS. If it's SOCKS5, install the pysocks library and use the prefix socks5h instead of http in proxy_url.

After rotation the address doesn't change. Cause: the carrier assigned the same address, or the rotation hasn't taken effect yet. Solution: increase the pause after rotate_ip to 10 seconds and check whether your plan limits IP rotation frequency.

Step 6: Save Data and Build Price History

Goal of this stage: switch the parser from printing to writing into a SQLite database so that every run adds a new point to price history instead of overwriting the old one. This is the heart of the whole project.

Designing Tables

We need two tables. First, offers, stores the listing: one row per offer with attributes and dates. Second, prices, stores price on a date: multiple rows per offer. The split is needed to avoid duplicating area and address with every price record.

1. Create a file storage.py and define table creation:

import sqlite3
from datetime import date
from config import DB_PATH
def get_conn():
conn = sqlite3.connect(DB_PATH)
conn.execute('CREATE TABLE IF NOT EXISTS offers (offer_id TEXT PRIMARY KEY, url TEXT, title TEXT, address TEXT, area REAL, floor INTEGER, floors_total INTEGER, build_year INTEGER, first_seen TEXT, last_seen TEXT, is_active INTEGER DEFAULT 1)')
conn.execute('CREATE TABLE IF NOT EXISTS prices (offer_id TEXT, checked_on TEXT, price INTEGER, PRIMARY KEY (offer_id, checked_on))')
conn.commit()
return conn

2. Add a function that returns the set of already known offer_ids. It's needed to request cards only for new listings:

def known_ids(conn):
rows = conn.execute('SELECT offer_id FROM offers').fetchall()
return set(r[0] for r in rows)

3. Write a save function. For a new listing, insert a row into offers. For any listing, update last_seen and write the price for today. The INSERT OR REPLACE construct in prices means: if today's price was already recorded, update it, otherwise insert:

def save_offers(conn, offers):
today = date.today().isoformat()
for o in offers:
conn.execute('INSERT OR IGNORE INTO offers (offer_id, url, title, address, area, floor, floors_total, build_year, first_seen, last_seen) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', (o['offer_id'], o['url'], o.get('title'), o.get('address'), o.get('area'), o.get('floor'), o.get('floors_total'), o.get('build_year'), today, today))
conn.execute('UPDATE offers SET last_seen = ?, is_active = 1 WHERE offer_id = ?', (today, o['offer_id']))
if o.get('price'):
conn.execute('INSERT OR REPLACE INTO prices (offer_id, checked_on, price) VALUES (?, ?, ?)', (o['offer_id'], today, o['price']))
conn.commit()

4. Add a function that marks delisted listings. If an offer hasn't appeared in the feed for more than three days, we consider it inactive. This gives you data on how fast properties sell:

def mark_inactive(conn, days=3):
conn.execute('UPDATE offers SET is_active = 0 WHERE julianday(?) - julianday(last_seen) > ?', (date.today().isoformat(), days))
conn.commit()

5. Rewrite the main block of parser.py so it collects the feed, identifies new listings, enriches only those, and saves everything:

from storage import get_conn, known_ids, save_offers, mark_inactive
if __name__ == '__main__':
conn = get_conn()
session = make_session()
listing = collect_listing(session)
old = known_ids(conn)
new_offers = [o for o in listing if o['offer_id'] not in old]
print('New listings:', len(new_offers), 'of', len(listing))
enrich_offers(new_offers, session)
save_offers(conn, listing)
mark_inactive(conn)
print('Saved. Total in database:', conn.execute('SELECT COUNT(*) FROM offers').fetchone()[0])

6. Don't forget to change collect_listing so it accepts session as a parameter instead of creating its own. Run the script. A realty.db file will appear in the project folder.

Viewing Price History

Create a file report.py that exports price changes to Excel. The query below finds listings whose latest price differs from the first one:

import sqlite3
import pandas as pd
from config import DB_PATH
conn = sqlite3.connect(DB_PATH)
query = 'SELECT o.offer_id, o.address, o.area, o.url, MIN(p.checked_on) AS first_date, MAX(p.checked_on) AS last_date, (SELECT price FROM prices WHERE offer_id = o.offer_id ORDER BY checked_on ASC LIMIT 1) AS first_price, (SELECT price FROM prices WHERE offer_id = o.offer_id ORDER BY checked_on DESC LIMIT 1) AS last_price FROM offers o JOIN prices p ON p.offer_id = o.offer_id GROUP BY o.offer_id'
df = pd.read_sql(query, conn)
df['change_pct'] = ((df['last_price'] - df['first_price']) / df['first_price'] * 100).round(1)
df['price_per_m2'] = (df['last_price'] / df['area']).round(0)
df = df.sort_values('change_pct')
df.to_excel('report.xlsx', index=False)
print(df.head(10))

After the first run, the change_pct column will be zero: there's no history yet. From the second day, the first changes appear. Within two weeks you'll see the big picture: how many listings dropped in price, by how much on average, and which neighborhoods move faster.

Tip: add database copying at the end of parser.py: shutil.copy(DB_PATH, 'backup_' + date.today().isoformat() + '.db'). One line of code protects weeks of accumulated data. Once a month, delete old copies, keeping the last five.

Check: realty.db exists, report.xlsx opens in Excel, and it has columns with address, area, prices, and price per square meter. Run parser.py again a few minutes later: the New listings line should show 0 or a small number, and cards shouldn't be requested again. This confirms deduplication works.

Potential Problems

database is locked error. Cause: the database is open in another program, like a SQLite viewer, or two script instances are running simultaneously. Solution: close extra programs and don't run the script in parallel with itself.

In report.xlsx all listings show the same date. Cause: the script has only run for one day. This is expected, just wait for the next runs.

Step 7: Automate Runs and Extend to Other Platforms

Goal of this stage: make the parser run itself every morning and prepare the code for connecting other real estate platforms. Price history is only valuable with regularity, so automation is mandatory.

Scheduling Runs

1. On Windows, open Task Scheduler via the Start menu search. Click Create Basic Task. Enter a name, e.g., Real Estate Parser.
2. Choose the trigger Daily, set a time, e.g., 07:30. Early morning is convenient: site load is minimal, and you have fresh data by the start of the workday.
3. In the action, choose Start a program. In the Program field, specify the full path to python.exe inside the venv folder, e.g., C:\projects\cian_parser\venv\Scripts\python.exe. In the Arguments field, enter parser.py. In the Start in field, specify the project folder.
4. Save the task and click Run in the right panel to test it. The database in the project folder should update.
5. On macOS and Linux, use cron. Type crontab -e in the terminal and add a line: 30 7 * * * cd /path/to/cian_parser && ./venv/bin/python parser.py >> run.log 2>&1. The log of all runs will accumulate in run.log.

Preparing for Other Platforms

Domclick, Yandex Real Estate, Metr Kvadratny, and regional portals work similarly, but their selectors and filter parameters are different. To avoid rewriting the parser for each one, put the differences into separate modules.

1. Create a sites folder in the project. Inside it, create a file cian.py and move the fetch_page and parse_cards functions there along with search parameters. Keep a consistent interface: the parse_cards function takes HTML and returns a list of dictionaries with the same keys offer_id, url, title, price, address.
2. For a new platform, create a file, e.g., domclick.py, and repeat the research from step two: open the feed, find the card, price, link, and pagination parameter. Write your own versions of fetch_page and parse_cards.
3. Prepend the platform prefix to offer_id, e.g., cian_312456789 and domclick_98765. Otherwise IDs from different sites may collide and scramble the history.
4. In parser.py, import modules from sites and run collection for each in a loop. Database tables are shared: one schema for all sources, and a source column will tell you where a listing came from.

An important note about Domclick and Yandex Real Estate: these platforms actively use internal JSON APIs visible in the Network tab. Their feeds are often easier to parse than HTML, but response structures change more often. Check selectors and fields once a month.

Tip: the same property listed on multiple platforms can have different prices. Comparing such pairs yields interesting analytics and helps you find sellers who dropped the price somewhere but forgot to update it elsewhere. You can match listings by address, area, and floor.

Check: the scheduled task ran manually without errors, and the run.log or Task Scheduler history shows a successful mark. The folder structure contains a sites folder with at least one module, the parser runs from the project root, and works like before.

Potential Problems

The scheduler says the task completed, but the database didn't update. Cause: the script ran from a different working directory and created a new empty database somewhere else. Solution: fill in the Start in field in the task, or use an absolute path to the database in config.py.

Result Check: Finished Parser Checklist

Go through the list and tick off each item. If everything is done, you have a full-fledged CIAN parser with price history.

• The parser.py script runs with a command from an activated environment with no import errors.
• The feed is collected from multiple pages, and the number of listings per page matches what you see in the browser.
• Prices save as whole numbers, addresses are readable, links open.
• Cards are requested only for new listings, and the log shows a New listings line with a shrinking number on repeat runs.
• Requests go through a mobile proxy, check_proxy.py shows a carrier address, and rotation changes it.
• On 429 status, the parser pauses and rotates IP instead of crashing.
• The realty.db file grows, and the prices table gets new rows every day.
• report.xlsx is generated and opens, and within a few days it has non-zero price changes.
• Auto-start is set up, and the log records every run.
• Database backup is created automatically.

How to Test End-to-End

1. Delete or rename realty.db to start fresh.
2. Set MAX_PAGES to 2 and run parser.py. Time it: it should take about 6-8 minutes including cards.
3. Run report.py and make sure the report has about 56 rows.
4. Open realty.db with any SQLite viewer, or run the query SELECT COUNT(*) FROM prices in Python. The number should match the number of listings.
5. Run parser.py again. Runtime should drop to about a minute because cards aren't requested. The number of rows in prices won't change because it's the same date.
6. Manually change the price of one listing in the database to a different number, set the date to yesterday, and run the parser. In report.xlsx that listing should show a price change. This confirms the history logic works without waiting for real changes.

Success Metrics

Share of listings with area filled in above 90 percent. Share of requests with 200 status above 97 percent. Zero unhandled exceptions per run. Full run duration is predictable and doesn't grow from run to run. If metrics are lower, go back to the typical errors section.

Common Mistakes and Fixes

We've put together the problems nearly everyone hits when writing a real estate listing parser for the first time. Format: problem, cause, solution.

The Parser Returns 0 Listings, Though It Worked Yesterday

Cause: the platform updated its layout and changed the data-mark or data-name attributes. Solution: open the feed in a browser, repeat step two, and update the selectors. Make a habit of keeping selectors in one place at the top of the module so edits take a minute. Add a check to the parser: if the first page has 0 listings, send yourself a message notification.

Prices Save Off by a Factor of a Thousand

Cause: some listings show prices in thousands or with a per-month note, or the text grabbed the price per square meter. Solution: make sure you're grabbing MainPrice, not a neighboring element with the per-meter price. Add a sanity check: a sale price for an apartment in Moscow under a million rubles is almost certainly a parsing error — log such cases.

429 Status Arrives by the Third Page

Cause: pauses are too short, or the proxy isn't connected yet, so all requests go from one home IP. Solution: increase PAUSE_MIN and PAUSE_MAX to 6 and 12, connect a mobile proxy, enable planned rotation every 20-25 requests. Check that you haven't launched multiple script instances simultaneously.

UnicodeEncodeError When Printing to Windows Console

Cause: the standard Windows console doesn't always render properly. Solution: run chcp 65001 in the terminal before starting, or add the line sys.stdout.reconfigure(encoding='utf-8') at the top of the script. You can also write logs to a file instead of the console.

Address Comes Out Incomplete or Duplicated

Cause: the card address is made of several GeoLabel links, some of which duplicate the city and district. Solution: remove duplicates while preserving order, or take the address from the listing detail page where it's a single string. For neighborhood analytics, add a separate district field, cutting it from the address by a known list of neighborhoods.

The Parser Works Manually but Not in the Scheduler

Cause: the scheduler uses a different Python interpreter without installed libraries, or a different working directory. Solution: specify the absolute path to python.exe inside venv and fill in the working folder field. Redirect output to a log file so you can see errors.

The Database Weighs Gigabytes After a Month

Cause: you're saving full page HTML or all JSON fields into the database. Solution: store only the needed fields. If you want to save raw pages for re-parsing, put them in compressed files on disk, not in SQLite. Once a quarter, run VACUUM to compact the database.

Duplicate Listings Under Different IDs

Cause: the seller delisted an offer and relisted it, getting a new number. Solution: add a secondary matching key built from address, area, and floor. Listings with the same key but different offer_id can be linked in a separate table, and price history can be calculated by the linked group. This is advanced analytics, but it's exactly what reveals real price drops hidden behind a relisting.

Advanced Features for Pros

The base parser is ready. If you want more, here are the areas that give the biggest payoff.

Browser Automation with Playwright

Some pages load data via scripts after load, and requests gets an empty skeleton. In such cases, use Playwright: it controls a real browser. Install it with pip install playwright and playwright install chromium. Pass the proxy when launching the browser via a proxy parameter with a dictionary of server, username, password. Wait for cards to appear via page.wait_for_selector and pass page.content() to the already-written parse_cards function. Note that the browser consumes ten times more resources, so use it narrowly, only for problematic pages.

Parallel Collection Across Multiple Proxy Channels

If you need to collect multiple regions, buy a separate mobile proxy for each region and run a separate process per channel. Don't use multithreading inside one channel: the point of load distribution is lost. An easy way: pass the region parameter to the script as a command-line argument, and let the scheduler run several tasks with different arguments and a small offset in time.

Price Drop Notifications

Add a comparison at the end of the parser between today's price and the previous one for each listing. If the drop exceeds a threshold, say 3 percent, compose a message with address, old and new price, and link, and send it to yourself via a messenger bot. This turns the parser from an analytics tool into an action tool: you'll know about good deals within an hour of the change.

Analytics and Visualization

With pandas you can group data by neighborhood and calculate median price per square meter, share of listings with price drops, average time on market (difference between first_seen and last_seen for inactive listings). The matplotlib library will plot a monthly trend in three lines of code. Upload the report to Google Sheets, and colleagues without coding skills get a live dashboard.

Storing in PostgreSQL

Once you have more than a hundred thousand listings, SQLite will slow down on analytical queries. Migrating to PostgreSQL is simple: the table schema stays the same, only the connection string and library change (psycopg2 instead of sqlite3). Only do this when truly needed: for a single city, SQLite is enough for years.

Parser Health Monitoring

Write to a separate runs table the start time, end time, number of listings collected, number of errors, and number of IP rotations. If listings drop sharply or errors exceed 5 percent, send a notification. Such monitoring lets you spot layout changes the day they happen, not two weeks later by an empty report.

FAQ: Common Questions About Building a Real Estate Parser

Is It Legal to Scrape CIAN and Other Platforms?

Collecting publicly available listing information for personal analytics is generally acceptable, but each platform's terms of service describe the conditions, and you should read them. It's absolutely off-limits to collect sellers' personal data, publish a copied database as your own, or create load that disrupts the service. If you plan commercial use of the data, consult a lawyer.

Why Mobile Proxies Rather Than Server Ones?

Mobile carrier addresses are shared by thousands of real subscribers and change constantly. Platforms are more lenient toward them than toward data center addresses, which almost no real buyers come from. Plus, the ability to rotate IP via a link gives you controlled rotation without buying hundreds of addresses.

How Often to Run the Parser for Price History?

Once a day is optimal. Real estate prices change rarely, and collecting more than once a day makes little sense while load grows. If you need to catch drops quickly, run twice a day — morning and evening — but only on a narrow filter.

How Many Listings Can You Collect per Day Through One Proxy?

With pauses of 4-9 seconds and planned rotation, roughly 500-700 requests per hour without problems, or 8-12 thousand per day running around the clock. For monitoring one city, 2-3 thousand requests per day is usually enough, because cards are only requested for new listings.

What If Selectors Have Changed and I Can't Find Them?

Go back to step two and work from the price: right-click on the price, Inspect, and climb up the tree to the card. Look for attributes with the word data and meaningful names: they're more stable than classes with random characters. Also check the Network tab: maybe data now comes via a separate JSON request, and parsing it will be even easier.

Can I Skip Proxies for a Small Project?

For a one-off two- or three-page collection, yes. For daily monitoring with hundreds of requests, a home address will quickly start getting restrictions, and data will become incomplete. Price history with gaps loses value, so for regular work you need proxies.

How Do I Parse Rentals Instead of Sales?

Change the deal_type parameter to rent and add the rental type to search parameters: long-term or daily. Listing URLs will contain rent instead of sale, so update the link selector in parse_cards. The rest of the logic, including price history, works unchanged.

Should I Store Listing Photos?

For price analytics, no. Photos take up lots of space and aren't needed for calculations. If you're building a catalog for internal use, save only image URLs, not the files themselves.

How Do I Know If a Listing Sold Rather Than Was Delisted?

Platforms don't report the reason for delisting. An indirect sign: the listing disappeared from the feed and didn't reappear within a month. Listings that vanish and return after a few days with a different price are more likely relisted. Link them by address and area, as described in the errors section.

What If I Need Data for Ten Cities?

Make a list of regions in config.py and run collection in a loop with a separate proxy channel for every two or three cities. Stagger run times so you don't collect everything at once. The database stays shared; add a region field to the offers table.

Conclusion

Let's look at what you've done. You set up an environment with Python and libraries, figured out how real estate platform feeds are built, and learned to find selectors yourself. You wrote a CIAN parser that collects the feed and listing pages. You connected mobile proxies with rotation and retries, which made collection robust and predictable. You designed a database with price history, set up reports, and scheduled auto-runs. This is a real working tool, not a tutorial example.

What to do next: let the parser run for two weeks without changes. During that time, history accumulates and you'll see weak spots: where the share of filled fields drops, which listings duplicate, where the report needs new columns. Only after that, add features. Then connect a second platform using the modular structure from step seven: you'll be surprised how much faster the second time goes.

Where to grow: price drop notifications will turn the tool into a deal source. Neighborhood and building-type analytics will make you a market expert with numbers in hand. Linking relisted properties will reveal real discounts invisible on the site. And a careful attitude toward the platform — pauses, address rotation via mobile proxies, and collecting only needed data — will let the tool run for months without failures.

Real estate scraping isn't about speed, it's about regularity and data quality. You've laid the right foundation. Happy scraping, and may your database grow every morning.