Previous logfile

OCDP import 2017-08-11T10:10:57+0200

Download crawl

from http://harth.org/andreas/2017/ocdp/2017-07-25/eurostat.nt.gz ...

from http://harth.org/andreas/2017/ocdp/2017-07-25/undata.nt.gz ...

Import crawl

Connected to OpenLink Virtuoso
Driver: 07.20.3214 OpenLink Virtuoso ODBC Driver
OpenLink Interactive SQL (Virtuoso), version 0.9849b.
Type HELP; for help and EXIT; to exit.
SQL> 
Done. -- 24 msec.
SQL> 
Done. -- 302537 msec.
SQL> Deleting previous data
SQL> 
Done. -- 30152 msec.
SQL> 
Done. -- 118256 msec.
SQL> Load new data
SQL> 
Done. -- 85 msec.
SQL> 
Done. -- 191068 msec.
SQL> Check for possible import errors
SQL> ll_error
VARCHAR
_______________________________________________________________________________

NULL

1 Rows. -- 1 msec.
SQL> Counting imported triples
SQL> numTriples
INTEGER
_______________________________________________________________________________

22460314

1 Rows. -- 1005 msec.
SQL>

Create canonical graph

Prepare graph

SPARQL Update query loadqueries/pre-canonicalise4.rq:

callret-0
Insert into <http://citydata.wu.ac.at/qb>, 43 (or less) triples -- done
0.11

SPARQL Update query loadqueries/pre-canonicalise6.rq:

yearDimension1 cityDimension1 indicatorDimension1 callret-3
http://purl.org/dc/terms/date http://ontologycentral.com/2009/01/eurostat/ns#cities http://ontologycentral.com/2009/01/eurostat/ns#indic_ur 20
http://purl.org/dc/terms/date http://ontologycentral.com/2009/01/eurostat/ns#geo http://ontologycentral.com/2009/01/eurostat/ns#indic_ur 20
1.51

Materialise

SPARQL Update query loadqueries/canonicalise3.rq:

callret-0
Insert into <http://citydata.wu.ac.at/qb-materialised-global-cube>, 7998187 (or less) triples -- done
671.44

Number of observations: 991916 at 2017-08-11T10:37:04+0200

Learn models and impute missing values

Learn step asynchronously

Download data from http://citydata.ai.wu.ac.at/MV-Predictions/data/A3_2004-2016_ncp80_seed_100_pred_80 [CURRENTLY DISABLED] 

Connected to OpenLink Virtuoso
Driver: 07.20.3214 OpenLink Virtuoso ODBC Driver
OpenLink Interactive SQL (Virtuoso), version 0.9849b.
Type HELP; for help and EXIT; to exit.
SQL> 
Done. -- 1 msec.
SQL> 
Done. -- 23738 msec.
SQL> Load new data
SQL> 
Done. -- 18 msec.
SQL> 
Done. -- 116138 msec.
SQL> Check for possible import errors
SQL> ll_state    ll_error
INTEGER     VARCHAR
_______________________________________________________________________________

2           NULL
2           NULL

2 Rows. -- 23 msec.
SQL> Counting imported triples
SQL> numTriples
INTEGER
_______________________________________________________________________________

13188198

1 Rows. -- 740 msec.
SQL>

Prepare graph

SPARQL Update query loadqueries/pre-canonicalise4.rq:

callret-0
Insert into <http://citydata.ai.wu.ac.at/MV-Predictions/>, 1 (or less) triples -- done
0.16

SPARQL Update query loadqueries/pre-canonicalise6.rq:

yearDimension1 cityDimension1 indicatorDimension1 callret-3
1.50

Materialise global cube

SPARQL Update query loadqueries/canonicalise4.rq:

callret-0
Insert into <http://citydata.wu.ac.at/qb-materialised-global-cube>, 9589808 (or less) triples -- done
631.27

Consolidate entities

ls: cannot access loadqueries/post*: No such file or directory

Move all consolidation into the correct graph

SPARQL Update query loadqueries/add-temp-to-materialised.rq:

callret-0
ADD <http://citydata.wu.ac.at/tmp> to <http://citydata.wu.ac.at/qb-materialised-global-cube> -- done Clear graph <http://citydata.wu.ac.at/tmp> -- done Commit -- done
1.62

Number of observations: 2190642 at 2017-08-11T10:49:59+0200

Apply QB equations

Normalise QB equations

~/svn-citydata/code/qbequations ~/virtuoso/import
29.93
rapper: Parsing URI file:///srv/ai/aic/home/bischof/svn-citydata/code/qbequations/eurostat-rules.ttl with parser guess
rapper: Serializing with serializer rdfxml-abbrev
rapper: Guessed parser name 'turtle'
rapper: Parsing returned 6438 triples
~/virtuoso/import

Load QB rules into triple store:
callret-0
Load <http://citydata.wu.ac.at/ocdp/eurostat-rules> into graph <http://citydata.wu.ac.at/qb> -- done
0.35

Generate SPARQL INSERT queries

/usr/local/lib/python2.7/dist-packages/SPARQLWrapper/Wrapper.py:438: UserWarning: keepalive support not available, so the execution of this method has no effect
  warnings.warn("keepalive support not available, so the execution of this method has no effect")
147 queries generated
44 rules ignored because too many variables
15.38

Evaluate SPARQL insert queries (timeout 30 seconds)

Number of observations: 2190642

QueryTime# triplesError
http://…c5e60278333.rq: 1.96 0 Virtuoso 42000 Error The estimated execution time 42868 (sec) exceeds the limit of 4000 (sec).
http://…5a55ac4b378.rq: 0.78 608
http://…4e7b5eaf345.rq: 49.64 232416
http://…c42199d9554.rq: 11.99 263899
http://…10abe909f13.rq: 13.95 291660
http://…f1268795d42.rq: 11.42 200772
http://…833b7c0e64e.rq: 12.03 222444
http://…02bffc8787f.rq: 26.92 126096
http://…4feb01819b1.rq: 65.70 251008
http://…006d7703b0f.rq: 48.83 320980
http://…4cd68e03482.rq: 10.86 214094
http://…9107a3cd023.rq: 62.80 229344
http://…c4e75c58892.rq: 36.35 167968
http://…528b2b28ac2.rq: 51.58 203776
http://…2e2a007da4c.rq: 68.25 250944
http://…3769c5dff40.rq: 1.99 0 Virtuoso 42000 Error The estimated execution time 77819 (sec) exceeds the limit of 4000 (sec).
http://…676644948f7.rq: 13.91 249850
http://…c0c3ea9177c.rq: 1.95 0 Virtuoso 42000 Error The estimated execution time 67598 (sec) exceeds the limit of 4000 (sec).
http://…4809889ff8c.rq: 26.40 126096
http://…e75f0f6bff8.rq: 7.74 139328
http://…7da8907fcb1.rq: 2.08 0 Virtuoso 42000 Error The estimated execution time 76279 (sec) exceeds the limit of 4000 (sec).
http://…547b2a09732.rq: 7.67 139312
http://…41511bb54c2.rq: 7.25 127528
http://…3f2213858c6.rq: 17.23 329575
http://…27995480803.rq: 11.67 215982
http://…46ee98f7c58.rq: 12.98 247977
http://…9650f14ffc7.rq: 13.88 267318
http://…a5fec6fb191.rq: 56.23 232384
http://…b4690cc26c7.rq: 11.70 223006
http://…cba38da0801.rq: 0.65 624
http://…73451b3b140.rq: 10.67 207108
http://…3b0e773f707.rq: 7.02 124573
http://…f34aa351518.rq: 10.88 181929
http://…1b1c69e5726.rq: 43.29 192960
http://…bd705912c82.rq: 9.67 193344
http://…0a240c4d7e8.rq: 44.46 211536
http://…d93f49d2d7c.rq: 46.69 192352
http://…0298630c249.rq: 12.55 235692
http://…80649628fe1.rq: 67.40 250560
http://…c2094a54793.rq: 46.15 184192
http://…94d61096cb6.rq: 1.98 0 Virtuoso 42000 Error The estimated execution time 43140 (sec) exceeds the limit of 4000 (sec).
http://…6ac45cd09c8.rq: 47.25 192112
http://…a9c2425a053.rq: 46.45 211184
http://…5fccfc868e4.rq: 13.64 262012
http://…5d8cbe3b643.rq: 48.94 193199
http://…54d157dc720.rq: 59.38 226416
http://…e853ff8bdad.rq: 8.01 118032
http://…2944b2b28f3.rq: 19.59 474669
http://…28e3844f452.rq: 14.31 280353
http://…b9bc6f6e319.rq: 2.11 0 Virtuoso 42000 Error The estimated execution time 76279 (sec) exceeds the limit of 4000 (sec).
http://…5976bc67d3f.rq: 0.68 720
http://…b094fb684c7.rq: 22.34 441972
http://…87a7b591d2f.rq: 13.81 248890
http://…c0731ac1cc4.rq: 10.44 207108
http://…e4ac8b6d0b0.rq: 14.64 276428
http://…578f4fd6151.rq: 21.69 377950
http://…398b559ed46.rq: 18.81 377540
http://…32a55826eec.rq: 6.59 138510
http://…4bf45990f1c.rq: 9.54 190302
http://…ad004b71d72.rq: 15.97 329719
http://…702bf3d4dae.rq: 2.05 0 Virtuoso 42000 Error The estimated execution time 43128 (sec) exceeds the limit of 4000 (sec).
http://…056bd2b98d9.rq: 15.76 293565
http://…53cb870a116.rq: 10.93 192109
http://…71cb1a87feb.rq: 54.55 211696
http://…fbf75ebdbf8.rq: 14.50 270470
http://…eb4569a82b9.rq: 8.01 118032
http://…7921c6faf6b.rq: 11.25 216057
http://…86cd866d89a.rq: 45.60 211376
http://…6e2f92b4fed.rq: 47.02 232416
http://…dfe5642f9b7.rq: 15.33 268353
http://…7412018ffc2.rq: 7.05 127261
http://…8e4cb38b51f.rq: 19.57 440633
http://…5175c7e7c50.rq: 39.40 166368
http://…d8477e9e27c.rq: 43.12 211728
http://…7e7aa1c1741.rq: 20.28 79424
http://…a07751e9d28.rq: 26.94 78352
http://…a2634030e69.rq: 65.22 249680
http://…260436d823a.rq: 2.01 0 Virtuoso 42000 Error The estimated execution time 43127 (sec) exceeds the limit of 4000 (sec).
http://…34f452e791c.rq: 13.94 275004
http://…90154dec6fd.rq: 67.97 250656
http://…be9db24fb9c.rq: 55.09 211983
http://…a59a9f58c09.rq: 41.66 187024
http://…fd66c127240.rq: 7.37 127496
http://…4165db5e17d.rq: 35.82 167680
http://…88e2d1903fb.rq: 17.01 317913
http://…826179cb61e.rq: 13.12 242696
http://…919227525e6.rq: 13.27 243656
http://…c68addaeead.rq: 10.51 179246
http://…48a42ce3f44.rq: 7.09 117509
http://…5ce081f5ad6.rq: 12.31 193899
http://…1453a55ca88.rq: 57.63 233040
http://…b22b020d767.rq: 7.41 123688
http://…f10d21b5f2a.rq: 19.17 376356
http://…6d2a8752e0a.rq: 1.87 0 Virtuoso 42000 Error The estimated execution time 7697 (sec) exceeds the limit of 4000 (sec).
http://…f00ed8fcf9c.rq: 14.43 266038
http://…37938fadb8e.rq: 55.40 233039
http://…6617ae6a71c.rq: 36.48 168112
http://…db93affa275.rq: 52.28 220096
http://…318ed5e698c.rq: 71.21 251040
http://…c6d9f87989b.rq: 7.39 123688
http://…ea8f08dd7e9.rq: 34.54 101424
http://…01a19dc31c6.rq: 6.86 125597
http://…3b63d1d6cf0.rq: 63.15 228224
http://…6810bcdf7e5.rq: 1.85 0 Virtuoso 42000 Error The estimated execution time 6900 (sec) exceeds the limit of 4000 (sec).
http://…dcec0709a4d.rq: 14.02 281793
http://…20afba545d0.rq: 11.74 225182
http://…f306d38525b.rq: 6.99 126221
http://…b9c88e1f3e5.rq: 48.23 202544
http://…779c8de912e.rq: 70.63 250448
http://…432ae80e329.rq: 60.40 250384
http://…ce2f6aa0aa6.rq: 20.30 441812
http://…729b7371f77.rq: 8.71 136846
http://…580d172ba1b.rq: 13.14 255654
http://…1c838a677a7.rq: 11.33 217913
http://…2c629af5bc1.rq: 49.50 221632
http://…ac4dea86513.rq: 0.44 208
http://…373919ee76c.rq: 16.14 297326
http://…851e851ec1a.rq: 12.35 244139
http://…a33565a2bc4.rq: 1.81 0 Virtuoso 42000 Error The estimated execution time 10504 (sec) exceeds the limit of 4000 (sec).
http://…11e1768d2c1.rq: 39.01 165088
http://…36314bdd0c0.rq: 12.71 234332
http://…2b4adbb40c0.rq: 8.41 193136
http://…19a57c9bd6f.rq: 1.79 0 Virtuoso 42000 Error The estimated execution time 11135 (sec) exceeds the limit of 4000 (sec).
http://…ddbb0e97102.rq: 11.94 262571
http://…68921ddfc0d.rq: 25.82 126096
http://…e527f16dd73.rq: 13.97 270244
http://…19e37555bf0.rq: 65.59 251104
http://…d8648a9e72e.rq: 12.41 222505
http://…16df27acc71.rq: 6.63 127528
http://…fe12947f0e3.rq: 38.80 184783
http://…ae077189b17.rq: 9.79 187408
http://…5e10a4f694c.rq: 14.72 303828
http://…d121e557425.rq: 26.33 292077
http://…688c43cd517.rq: 17.42 328048
http://…bc2dde9e4af.rq: 12.33 249289
http://…8155910b8da.rq: 7.35 127496
http://…32bf80a4040.rq: 17.36 382074
http://…ee68b5188df.rq: 12.63 231760
http://…385ed778ee3.rq: 8.83 192688
http://…7150fdce541.rq: 15.98 322389
http://…d47acb3152e.rq: 11.04 198952
http://…a06e0e079bb.rq: 0.42 496
http://…7dbeb306787.rq: 11.40 203232
http://…5741823cbcb.rq: 12.33 261484
http://…9ca053a47bd.rq: 10.92 200074
http://…e198604c6be.rq: 9.13 167824
http://…8e1b477fe87.rq: 11.54 238085

Import data

Connected to OpenLink Virtuoso
Driver: 07.20.3214 OpenLink Virtuoso ODBC Driver
OpenLink Interactive SQL (Virtuoso), version 0.9849b.
Type HELP; for help and EXIT; to exit.
SQL> 
Done. -- 10 msec.
SQL> Load new data
SQL> 
Done. -- 76 msec.
SQL> 
Done. -- 351613 msec.
SQL> Check for possible import errors
SQL> ll_error
VARCHAR
_______________________________________________________________________________

NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
NULL

135 Rows. -- 11 msec.
SQL> Counting imported triples
SQL> numTriples
INTEGER
_______________________________________________________________________________

46793663

1 Rows. -- 1797 msec.
SQL>

Number of observations: 4009242 at 2017-08-11T11:55:00+0200

NTriples containing all results of the QB equations: qb-equations-data.nt.

Finished

Data source# observations
Crawl991916
Predicted1198726
QB equations1818600

Import finished at 2017-08-11T11:55:00+0200.