{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CORESE client: query\n",
    "\n",
    "CORESE is a Semantic Web Factory (triple store & SPARQL endpoint) implementing RDF, RDFS, SPARQL 1.1 Query & Update. This notebook is intended to be used as a test client for the local CORESE server.\n",
    "\n",
    "For more information see: https://project.inria.fr/corese"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup and run CORESE server"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download CORESE server \n",
    "\n",
    "Download the server from https://project.inria.fr/corese/download/\n",
    "\n",
    "Only do it once. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modify the profile.ttl file\n",
    "\n",
    "1. Define a profile.ttl document as shown in the downloaded profile.ttl file.\n",
    "The string \"myserver\" is the name of the service that you implement, you can choose another name. \n",
    "Put profile.ttl in a directory (say absolute path DIR) in your file system.\n",
    "\n",
    "2. Specify the file(s) that you want to load\n",
    "in your service by setting, in profile.ttl,  the value of the sw:path property. In the example, replace  <http://www.w3.org/1999/02/22-rdf-syntax-ns> by the URL of your document. It can be a path in the file system.  The path can be relative to the path of profile.ttl. URL and path must be within \"<\" and \">\".\n",
    "If you load several documents, write the list of URL/path separated by comma \",\" :\n",
    "sw:path <u1>, <u2>, ... \n",
    "The profile.ttl document is  RDF in Turtle syntax.\n",
    "    \n",
    "One time operation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start the server\n",
    "\n",
    "Run the following command from the directory where the server is downloaded\n",
    "\n",
    "_java -jar -Dfile.encoding=UTF8 -Xmx20G corese-server-4.1.2.jar -e -lp -debug -pp profile.ttl_\n",
    "\n",
    "or if running on Windows change the path and run the code below.\n",
    "\n",
    "NOTE: please notice that the server name/version in the command line should corresponds to your server executable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import subprocess\n",
    "\n",
    "corese_path = os.path.normpath('C:/Users/abobashe/Documents/MonaLIA/CORESE')\n",
    "command_line = 'start /w cmd /k java -jar -Dfile.encoding=UTF8 -Xmx20G corese-server-4.1.2.jar -e -lp -debug -pp profile.ttl'\n",
    "\n",
    "corese_server = subprocess.Popen(command_line, shell=True, cwd=corese_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install SPARQLWrapper package\n",
    "\n",
    "This package helps to convert service output to a Pandas DataFrame. https://rdflib.dev/sparqlwrapper/\n",
    "\n",
    "NOTE: If you are running Anaconda distribution and prefer the conda installation find the command here https://anaconda.org/conda-forge/sparqlwrapper\n",
    "\n",
    "Only run it once or periodically to check for the updates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install SPARQLWrapper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SPARQLWrapper ver. 1.8.5\n"
     ]
    }
   ],
   "source": [
    "import SPARQLWrapper\n",
    "import json\n",
    "print('SPARQLWrapper ver.', SPARQLWrapper.__version__)\n",
    "\n",
    "from SPARQLWrapper import SPARQLWrapper, JSON, XML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Pandas package\n",
    "\n",
    "To use Pandas DataFrame to contain the query results.\n",
    "\n",
    "NOTE: if you are running Anaconda distribution the preferred way to install pandas is:\n",
    "\n",
    "_conda install pandas_\n",
    "\n",
    "Only run it once or periodically to check for the updates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pandas ver. 0.23.4\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "print('Pandas ver.', pd.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sparql_service_to_dataframe(service, query):\n",
    "    \"\"\"\n",
    "    Helper function to convert SPARQL results into a Pandas DataFrame.\n",
    "    \n",
    "    Credit to Ted Lawless https://lawlesst.github.io/notebook/sparql-dataframe.html\n",
    "    \"\"\"\n",
    "    sparql = SPARQLWrapper(service)\n",
    "    sparql.setQuery(query)\n",
    "    sparql.setReturnFormat(JSON)\n",
    "    result = sparql.query()\n",
    "\n",
    "    processed_results = json.load(result.response)\n",
    "    cols = processed_results['head']['vars']\n",
    "\n",
    "    out = []\n",
    "    for row in processed_results['results']['bindings']:\n",
    "        item = []\n",
    "        for c in cols:\n",
    "            item.append(row.get(c, {}).get('value'))\n",
    "        out.append(item)\n",
    "\n",
    "    return pd.DataFrame(out, columns=cols)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run queries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "wds_Corese = 'http://corese.inria.fr/sparql'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = '''\n",
    "\n",
    "select *\n",
    "{\n",
    "  ?x ?p ?y\n",
    "}\n",
    "limit 10\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10, 3)\n"
     ]
    }
   ],
   "source": [
    "df = sparql_service_to_dataframe(wds_Corese, query)\n",
    "print(df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "require": [
     "base/js/events",
     "datatables.net",
     "d3",
     "chartjs",
     "dt-config",
     "dt-components",
     "dt-graph-objects",
     "dt-toolbar",
     "dt-tooltips",
     "jupyter-datatables"
    ],
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>p</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>http://www.inria.fr/2015/humans#Man</td>\n",
       "      <td>http://ns.inria.fr/sparql-template/icon</td>\n",
       "      <td>http://corese.inria.fr/img/male.png</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>http://www.inria.fr/2015/humans#Person</td>\n",
       "      <td>http://ns.inria.fr/sparql-template/icon</td>\n",
       "      <td>http://corese.inria.fr/img/person.png</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>http://www.inria.fr/2015/humans#Lecturer</td>\n",
       "      <td>http://ns.inria.fr/sparql-template/icon</td>\n",
       "      <td>http://corese.inria.fr/img/professor.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>http://www.inria.fr/2015/humans#Researcher</td>\n",
       "      <td>http://ns.inria.fr/sparql-template/icon</td>\n",
       "      <td>http://corese.inria.fr/img/professor.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>http://www.inria.fr/2015/humans#Woman</td>\n",
       "      <td>http://ns.inria.fr/sparql-template/icon</td>\n",
       "      <td>http://corese.inria.fr/img/female.png</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            x  \\\n",
       "0  http://www.inria.fr/2015/humans#Man          \n",
       "1  http://www.inria.fr/2015/humans#Person       \n",
       "2  http://www.inria.fr/2015/humans#Lecturer     \n",
       "3  http://www.inria.fr/2015/humans#Researcher   \n",
       "4  http://www.inria.fr/2015/humans#Woman        \n",
       "\n",
       "                                         p  \\\n",
       "0  http://ns.inria.fr/sparql-template/icon   \n",
       "1  http://ns.inria.fr/sparql-template/icon   \n",
       "2  http://ns.inria.fr/sparql-template/icon   \n",
       "3  http://ns.inria.fr/sparql-template/icon   \n",
       "4  http://ns.inria.fr/sparql-template/icon   \n",
       "\n",
       "                                          y  \n",
       "0  http://corese.inria.fr/img/male.png       \n",
       "1  http://corese.inria.fr/img/person.png     \n",
       "2  http://corese.inria.fr/img/professor.jpg  \n",
       "3  http://corese.inria.fr/img/professor.jpg  \n",
       "4  http://corese.inria.fr/img/female.png     "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option('display.max_colwidth', -1) # if your Pandas version is < 1.0 then use -1 as second parameter, None otherwise\n",
    "pd.set_option('display.precision', 3)\n",
    "pd.set_option('display.max_rows', 999)\n",
    "\n",
    "df.head(10)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "require": {
   "paths": {
    "buttons.colvis": "https://cdn.datatables.net/buttons/1.5.6/js/buttons.colVis.min",
    "buttons.flash": "https://cdn.datatables.net/buttons/1.5.6/js/buttons.flash.min",
    "buttons.html5": "https://cdn.datatables.net/buttons/1.5.6/js/buttons.html5.min",
    "buttons.print": "https://cdn.datatables.net/buttons/1.5.6/js/buttons.print.min",
    "chartjs": "https://cdnjs.cloudflare.com/ajax/libs/Chart.js/2.8.0/Chart",
    "d3": "https://d3js.org/d3.v5.min",
    "d3-array": "https://d3js.org/d3-array.v2.min",
    "datatables.net": "https://cdn.datatables.net/1.10.18/js/jquery.dataTables",
    "datatables.net-buttons": "https://cdn.datatables.net/buttons/1.5.6/js/dataTables.buttons.min",
    "datatables.responsive": "https://cdn.datatables.net/responsive/2.2.2/js/dataTables.responsive.min",
    "datatables.scroller": "https://cdn.datatables.net/scroller/2.0.0/js/dataTables.scroller.min",
    "datatables.select": "https://cdn.datatables.net/select/1.3.0/js/dataTables.select.min",
    "jszip": "https://cdnjs.cloudflare.com/ajax/libs/jszip/2.5.0/jszip.min",
    "moment": "https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.8.0/moment",
    "pdfmake": "https://cdnjs.cloudflare.com/ajax/libs/pdfmake/0.1.36/pdfmake.min",
    "vfsfonts": "https://cdnjs.cloudflare.com/ajax/libs/pdfmake/0.1.36/vfs_fonts"
   },
   "shim": {
    "buttons.colvis": {
     "deps": [
      "jszip",
      "datatables.net-buttons"
     ]
    },
    "buttons.flash": {
     "deps": [
      "jszip",
      "datatables.net-buttons"
     ]
    },
    "buttons.html5": {
     "deps": [
      "jszip",
      "datatables.net-buttons"
     ]
    },
    "buttons.print": {
     "deps": [
      "jszip",
      "datatables.net-buttons"
     ]
    },
    "chartjs": {
     "deps": [
      "moment"
     ]
    },
    "datatables.net": {
     "exports": "$.fn.dataTable"
    },
    "datatables.net-buttons": {
     "deps": [
      "datatables.net"
     ]
    },
    "pdfmake": {
     "deps": [
      "datatables.net"
     ]
    },
    "vfsfonts": {
     "deps": [
      "datatables.net"
     ]
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}