Automated Application Catalogue, Kabeen

Problem

Each application entry in the catalogue (vendor, product, website, contact, description, category) was researched and written by hand: about 15 minutes per entry, which could not scale to a catalogue of 100,000+ applications. The same product also appeared under many names (Microsoft, Office, Excel), so duplicates and inconsistencies built up.

What I built

• Scraping and ETL: collection of vendor and technology data from the web, then normalisation into one schema.
• LLM classification and enrichment with GPT-4o and GPT-4o mini: categories, descriptions and missing fields filled in automatically.
• Entity resolution: products grouped under their vendor (for example Microsoft → Office, Excel) by combining fuzzy matching, TF-IDF and embeddings stored in pgvector.
• Neo4j knowledge graph linking each vendor to its technologies.
• Data quality and anomaly detection engine in three levels: rules, then statistical checks, then semantic checks; doubtful entries go to human validation.
• Full product: PostgreSQL database, FastAPI backend and React interface.

Results

Technology stack

PythonGPT-4oGPT-4o miniScrapingETLTF-IDFFuzzy matchingpgvectorNeo4jPostgreSQLFastAPIReact